Note: This is test shard 2 of 8.
[==========] Running 8 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from RebalanceIgnoredTserversTest
[ RUN ] RebalanceIgnoredTserversTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 01:52:26.716259 11433 test_util.cc:268] Using random seed: -622356425
I20240502 01:52:26.727093 11433 ts_itest-base.cc:112] Starting cluster with:
I20240502 01:52:26.727171 11433 ts_itest-base.cc:113] --------------
I20240502 01:52:26.727245 11433 ts_itest-base.cc:114] 5 tablet servers
I20240502 01:52:26.727299 11433 ts_itest-base.cc:115] 3 replicas per TS
I20240502 01:52:26.727360 11433 ts_itest-base.cc:116] --------------
2024-05-02T01:52:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T01:52:26Z Disabled control of system clock
I20240502 01:52:26.749128 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.42.126:41021
--webserver_interface=127.11.42.126
--webserver_port=0
--builtin_ntp_servers=127.11.42.84:42949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.42.126:41021
--enable_range_replica_placement=false with env {}
W20240502 01:52:27.053449 11445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:27.053695 11445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:27.053788 11445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:27.060802 11445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 01:52:27.060894 11445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:27.060943 11445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 01:52:27.060999 11445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 01:52:27.069433 11445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:42949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.11.42.126:41021
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.42.126:41021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.11.42.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:27.070547 11445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:27.072108 11445 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:27.081969 11450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:27.082263 11453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:27.082538 11451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:27.082729 11445 server_base.cc:1034] running on GCE node
I20240502 01:52:27.083397 11445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:27.084653 11445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:27.085846 11445 hybrid_clock.cc:648] HybridClock initialized: now 1714614747085780 us; error 42 us; skew 500 ppm
I20240502 01:52:27.086117 11445 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:27.093482 11445 webserver.cc:460] Webserver started at http://127.11.42.126:42669/ using document root <none> and password file <none>
I20240502 01:52:27.094135 11445 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:27.094244 11445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:27.094506 11445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:27.097070 11445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "f4710fbf674945e6a3b43e1b2b8c74e1"
format_stamp: "Formatted at 2024-05-02 01:52:27 on dist-test-slave-xvk5"
I20240502 01:52:27.097776 11445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "f4710fbf674945e6a3b43e1b2b8c74e1"
format_stamp: "Formatted at 2024-05-02 01:52:27 on dist-test-slave-xvk5"
I20240502 01:52:27.102653 11445 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20240502 01:52:27.105789 11459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:27.106995 11445 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:52:27.107131 11445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/wal
uuid: "f4710fbf674945e6a3b43e1b2b8c74e1"
format_stamp: "Formatted at 2024-05-02 01:52:27 on dist-test-slave-xvk5"
I20240502 01:52:27.107290 11445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:27.122314 11445 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:27.123076 11445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:27.123324 11445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:27.140672 11445 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.126:41021
I20240502 01:52:27.140703 11510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.126:41021 every 8 connection(s)
I20240502 01:52:27.141948 11445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 01:52:27.145393 11511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:27.151162 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 11445
I20240502 01:52:27.151387 11433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 01:52:27.153931 11511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4710fbf674945e6a3b43e1b2b8c74e1: Bootstrap starting.
I20240502 01:52:27.157070 11511 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4710fbf674945e6a3b43e1b2b8c74e1: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:27.158124 11511 log.cc:824] T 00000000000000000000000000000000 P f4710fbf674945e6a3b43e1b2b8c74e1: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:27.160482 11511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4710fbf674945e6a3b43e1b2b8c74e1: No bootstrap required, opened a new log
I20240502 01:52:27.167378 11511 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4710fbf674945e6a3b43e1b2b8c74e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4710fbf674945e6a3b43e1b2b8c74e1" member_type: VOTER last_known_addr { host: "127.11.42.126" port: 41021 } }
I20240502 01:52:27.167832 11511 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4710fbf674945e6a3b43e1b2b8c74e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:27.167950 11511 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4710fbf674945e6a3b43e1b2b8c74e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4710fbf674945e6a3b43e1b2b8c74e1, State: Initialized, Role: FOLLOWER
I20240502 01:52:27.168512 11511 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4710fbf674945e6a3b43e1b2b8c74e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4710fbf674945e6a3b43e1b2b8c74e1" member_type: VOTER last_known_addr { host: "127.11.42.126" port: 41021 } }
I20240502 01:52:27.168694 11511 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4710fbf674945e6a3b43e1b2b8c74e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:27.168824 11511 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4710fbf674945e6a3b43e1b2b8c74e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:27.168968 11511 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4710fbf674945e6a3b43e1b2b8c74e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:27.171795 11511 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4710fbf674945e6a3b43e1b2b8c74e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4710fbf674945e6a3b43e1b2b8c74e1" member_type: VOTER last_known_addr { host: "127.11.42.126" port: 41021 } }
I20240502 01:52:27.172216 11511 leader_election.cc:302] T 00000000000000000000000000000000 P f4710fbf674945e6a3b43e1b2b8c74e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4710fbf674945e6a3b43e1b2b8c74e1; no voters:
I20240502 01:52:27.172866 11511 leader_election.cc:288] T 00000000000000000000000000000000 P f4710fbf674945e6a3b43e1b2b8c74e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:27.172987 11516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4710fbf674945e6a3b43e1b2b8c74e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:27.173681 11516 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4710fbf674945e6a3b43e1b2b8c74e1 [term 1 LEADER]: Becoming Leader. State: Replica: f4710fbf674945e6a3b43e1b2b8c74e1, State: Running, Role: LEADER
I20240502 01:52:27.174258 11516 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4710fbf674945e6a3b43e1b2b8c74e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4710fbf674945e6a3b43e1b2b8c74e1" member_type: VOTER last_known_addr { host: "127.11.42.126" port: 41021 } }
I20240502 01:52:27.175689 11511 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4710fbf674945e6a3b43e1b2b8c74e1 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:52:27.177031 11517 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4710fbf674945e6a3b43e1b2b8c74e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4710fbf674945e6a3b43e1b2b8c74e1. Latest consensus state: current_term: 1 leader_uuid: "f4710fbf674945e6a3b43e1b2b8c74e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4710fbf674945e6a3b43e1b2b8c74e1" member_type: VOTER last_known_addr { host: "127.11.42.126" port: 41021 } } }
I20240502 01:52:27.177394 11517 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4710fbf674945e6a3b43e1b2b8c74e1 [sys.catalog]: This master's current role is: LEADER
I20240502 01:52:27.177055 11518 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4710fbf674945e6a3b43e1b2b8c74e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4710fbf674945e6a3b43e1b2b8c74e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4710fbf674945e6a3b43e1b2b8c74e1" member_type: VOTER last_known_addr { host: "127.11.42.126" port: 41021 } } }
I20240502 01:52:27.177726 11518 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4710fbf674945e6a3b43e1b2b8c74e1 [sys.catalog]: This master's current role is: LEADER
I20240502 01:52:27.178862 11523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:52:27.189971 11523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:52:27.198314 11523 catalog_manager.cc:1347] Generated new cluster ID: fb23605e6f0e4ddfb5b9e505cfc19c30
I20240502 01:52:27.198442 11523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:52:27.214788 11523 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:52:27.215858 11523 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:52:27.226639 11523 catalog_manager.cc:5857] T 00000000000000000000000000000000 P f4710fbf674945e6a3b43e1b2b8c74e1: Generated new TSK 0
I20240502 01:52:27.227595 11523 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:52:27.237747 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.65:0
--local_ip_for_outbound_sockets=127.11.42.65
--webserver_interface=127.11.42.65
--webserver_port=0
--tserver_master_addrs=127.11.42.126:41021
--builtin_ntp_servers=127.11.42.84:42949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20240502 01:52:27.551196 11535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:27.551460 11535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:27.551589 11535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:27.558703 11535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:27.558877 11535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.65
I20240502 01:52:27.567270 11535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:42949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.11.42.65
--webserver_port=0
--tserver_master_addrs=127.11.42.126:41021
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.65
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:27.568400 11535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:27.570009 11535 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:27.581339 11541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:27.581497 11540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:27.581560 11543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:27.581991 11535 server_base.cc:1034] running on GCE node
I20240502 01:52:27.582491 11535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:27.583276 11535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:27.584501 11535 hybrid_clock.cc:648] HybridClock initialized: now 1714614747584437 us; error 38 us; skew 500 ppm
I20240502 01:52:27.584784 11535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:27.592202 11535 webserver.cc:460] Webserver started at http://127.11.42.65:35755/ using document root <none> and password file <none>
I20240502 01:52:27.592860 11535 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:27.592980 11535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:27.593305 11535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:27.595691 11535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "6c8fe71c8f5e4aa3a0fd5113a9b24a61"
format_stamp: "Formatted at 2024-05-02 01:52:27 on dist-test-slave-xvk5"
I20240502 01:52:27.597326 11535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "6c8fe71c8f5e4aa3a0fd5113a9b24a61"
format_stamp: "Formatted at 2024-05-02 01:52:27 on dist-test-slave-xvk5"
I20240502 01:52:27.602092 11535 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20240502 01:52:27.605273 11549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:27.606487 11535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240502 01:52:27.606621 11535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "6c8fe71c8f5e4aa3a0fd5113a9b24a61"
format_stamp: "Formatted at 2024-05-02 01:52:27 on dist-test-slave-xvk5"
I20240502 01:52:27.606770 11535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:27.629282 11535 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:27.630323 11535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:27.630625 11535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:27.631634 11535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:27.633337 11535 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:27.633440 11535 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:27.633571 11535 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:27.633630 11535 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:27.665576 11535 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.65:44379
I20240502 01:52:27.665616 11662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.65:44379 every 8 connection(s)
I20240502 01:52:27.666800 11535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 01:52:27.673533 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 11535
I20240502 01:52:27.673727 11433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 01:52:27.677093 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.66:0
--local_ip_for_outbound_sockets=127.11.42.66
--webserver_interface=127.11.42.66
--webserver_port=0
--tserver_master_addrs=127.11.42.126:41021
--builtin_ntp_servers=127.11.42.84:42949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240502 01:52:27.680018 11663 heartbeater.cc:347] Connected to a master server at 127.11.42.126:41021
I20240502 01:52:27.680311 11663 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:27.681205 11663 heartbeater.cc:511] Master 127.11.42.126:41021 requested a full tablet report, sending...
I20240502 01:52:27.683517 11476 ts_manager.cc:194] Registered new tserver with Master: 6c8fe71c8f5e4aa3a0fd5113a9b24a61 (127.11.42.65:44379)
I20240502 01:52:27.685448 11476 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.65:32883
W20240502 01:52:27.983649 11666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:27.983896 11666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:27.983985 11666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:27.990980 11666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:27.991123 11666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.66
I20240502 01:52:27.999439 11666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:42949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.11.42.66
--webserver_port=0
--tserver_master_addrs=127.11.42.126:41021
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.66
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:28.000581 11666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:28.002146 11666 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:28.012919 11671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:28.013301 11672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:28.013594 11674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:28.013931 11666 server_base.cc:1034] running on GCE node
I20240502 01:52:28.014462 11666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:28.015273 11666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:28.016480 11666 hybrid_clock.cc:648] HybridClock initialized: now 1714614748016414 us; error 42 us; skew 500 ppm
I20240502 01:52:28.016728 11666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:28.018755 11666 webserver.cc:460] Webserver started at http://127.11.42.66:36953/ using document root <none> and password file <none>
I20240502 01:52:28.019387 11666 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:28.019490 11666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:28.019737 11666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:28.022125 11666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "7ba4fdcc9de34d428e3ebc34c74f0436"
format_stamp: "Formatted at 2024-05-02 01:52:28 on dist-test-slave-xvk5"
I20240502 01:52:28.022890 11666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "7ba4fdcc9de34d428e3ebc34c74f0436"
format_stamp: "Formatted at 2024-05-02 01:52:28 on dist-test-slave-xvk5"
I20240502 01:52:28.027590 11666 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 01:52:28.030730 11680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:28.031877 11666 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 01:52:28.032020 11666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "7ba4fdcc9de34d428e3ebc34c74f0436"
format_stamp: "Formatted at 2024-05-02 01:52:28 on dist-test-slave-xvk5"
I20240502 01:52:28.032173 11666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:28.062202 11666 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:28.062970 11666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:28.063226 11666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:28.064082 11666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:28.065510 11666 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:28.065582 11666 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:28.065703 11666 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:28.065752 11666 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:28.094681 11666 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.66:43637
I20240502 01:52:28.094779 11793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.66:43637 every 8 connection(s)
I20240502 01:52:28.095890 11666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 01:52:28.103473 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 11666
I20240502 01:52:28.103626 11433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 01:52:28.107187 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.67:0
--local_ip_for_outbound_sockets=127.11.42.67
--webserver_interface=127.11.42.67
--webserver_port=0
--tserver_master_addrs=127.11.42.126:41021
--builtin_ntp_servers=127.11.42.84:42949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240502 01:52:28.107882 11794 heartbeater.cc:347] Connected to a master server at 127.11.42.126:41021
I20240502 01:52:28.108193 11794 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:28.109160 11794 heartbeater.cc:511] Master 127.11.42.126:41021 requested a full tablet report, sending...
I20240502 01:52:28.110575 11476 ts_manager.cc:194] Registered new tserver with Master: 7ba4fdcc9de34d428e3ebc34c74f0436 (127.11.42.66:43637)
I20240502 01:52:28.111440 11476 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.66:56031
W20240502 01:52:28.425791 11797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:28.426054 11797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:28.426153 11797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:28.433458 11797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:28.433610 11797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.67
I20240502 01:52:28.442112 11797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:42949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.11.42.67
--webserver_port=0
--tserver_master_addrs=127.11.42.126:41021
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.67
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:28.443235 11797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:28.444795 11797 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:28.455832 11802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:28.456118 11803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:28.456389 11805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:28.456614 11797 server_base.cc:1034] running on GCE node
I20240502 01:52:28.457177 11797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:28.457979 11797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:28.459187 11797 hybrid_clock.cc:648] HybridClock initialized: now 1714614748459123 us; error 39 us; skew 500 ppm
I20240502 01:52:28.459461 11797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:28.461525 11797 webserver.cc:460] Webserver started at http://127.11.42.67:38091/ using document root <none> and password file <none>
I20240502 01:52:28.462152 11797 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:28.462286 11797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:28.462564 11797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:28.464923 11797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "06a2ae4b0a294968a9c2ea6cdeaa62a3"
format_stamp: "Formatted at 2024-05-02 01:52:28 on dist-test-slave-xvk5"
I20240502 01:52:28.465662 11797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "06a2ae4b0a294968a9c2ea6cdeaa62a3"
format_stamp: "Formatted at 2024-05-02 01:52:28 on dist-test-slave-xvk5"
I20240502 01:52:28.470177 11797 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 01:52:28.473165 11811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:28.474298 11797 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:52:28.474443 11797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "06a2ae4b0a294968a9c2ea6cdeaa62a3"
format_stamp: "Formatted at 2024-05-02 01:52:28 on dist-test-slave-xvk5"
I20240502 01:52:28.474597 11797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:28.488713 11797 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:28.489439 11797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:28.489692 11797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:28.490516 11797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:28.491914 11797 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:28.491987 11797 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:28.492102 11797 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:28.492154 11797 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:28.522399 11797 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.67:36373
I20240502 01:52:28.522456 11923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.67:36373 every 8 connection(s)
I20240502 01:52:28.523947 11797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 01:52:28.532510 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 11797
I20240502 01:52:28.532713 11433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 01:52:28.535939 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.68:0
--local_ip_for_outbound_sockets=127.11.42.68
--webserver_interface=127.11.42.68
--webserver_port=0
--tserver_master_addrs=127.11.42.126:41021
--builtin_ntp_servers=127.11.42.84:42949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240502 01:52:28.540777 11924 heartbeater.cc:347] Connected to a master server at 127.11.42.126:41021
I20240502 01:52:28.541065 11924 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:28.541810 11924 heartbeater.cc:511] Master 127.11.42.126:41021 requested a full tablet report, sending...
I20240502 01:52:28.543115 11476 ts_manager.cc:194] Registered new tserver with Master: 06a2ae4b0a294968a9c2ea6cdeaa62a3 (127.11.42.67:36373)
I20240502 01:52:28.543866 11476 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.67:35281
I20240502 01:52:28.688189 11663 heartbeater.cc:503] Master 127.11.42.126:41021 was elected leader, sending a full tablet report...
W20240502 01:52:28.856423 11927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:28.856689 11927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:28.856832 11927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:28.863854 11927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:28.864112 11927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.68
I20240502 01:52:28.872637 11927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:42949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.11.42.68
--webserver_port=0
--tserver_master_addrs=127.11.42.126:41021
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.68
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:28.873826 11927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:28.875434 11927 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:28.886617 11932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:28.886862 11933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:28.887116 11935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:28.887341 11927 server_base.cc:1034] running on GCE node
I20240502 01:52:28.887845 11927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:28.888643 11927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:28.889840 11927 hybrid_clock.cc:648] HybridClock initialized: now 1714614748889781 us; error 28 us; skew 500 ppm
I20240502 01:52:28.890105 11927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:28.892167 11927 webserver.cc:460] Webserver started at http://127.11.42.68:40487/ using document root <none> and password file <none>
I20240502 01:52:28.892833 11927 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:28.892956 11927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:28.893302 11927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:28.895733 11927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "88568756bbbf4e759910daea9ddaf433"
format_stamp: "Formatted at 2024-05-02 01:52:28 on dist-test-slave-xvk5"
I20240502 01:52:28.896431 11927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "88568756bbbf4e759910daea9ddaf433"
format_stamp: "Formatted at 2024-05-02 01:52:28 on dist-test-slave-xvk5"
I20240502 01:52:28.901240 11927 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20240502 01:52:28.904343 11941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:28.905596 11927 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:52:28.905757 11927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "88568756bbbf4e759910daea9ddaf433"
format_stamp: "Formatted at 2024-05-02 01:52:28 on dist-test-slave-xvk5"
I20240502 01:52:28.905920 11927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:28.930347 11927 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:28.931071 11927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:28.931339 11927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:28.932194 11927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:28.933602 11927 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:28.933677 11927 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:28.933794 11927 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:28.933849 11927 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:28.962852 11927 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.68:36219
I20240502 01:52:28.962909 12053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.68:36219 every 8 connection(s)
I20240502 01:52:28.964057 11927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 01:52:28.970216 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 11927
I20240502 01:52:28.970378 11433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 01:52:28.973671 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.69:0
--local_ip_for_outbound_sockets=127.11.42.69
--webserver_interface=127.11.42.69
--webserver_port=0
--tserver_master_addrs=127.11.42.126:41021
--builtin_ntp_servers=127.11.42.84:42949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240502 01:52:28.976162 12054 heartbeater.cc:347] Connected to a master server at 127.11.42.126:41021
I20240502 01:52:28.976477 12054 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:28.977298 12054 heartbeater.cc:511] Master 127.11.42.126:41021 requested a full tablet report, sending...
I20240502 01:52:28.978708 11476 ts_manager.cc:194] Registered new tserver with Master: 88568756bbbf4e759910daea9ddaf433 (127.11.42.68:36219)
I20240502 01:52:28.979530 11476 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.68:53881
I20240502 01:52:29.113701 11794 heartbeater.cc:503] Master 127.11.42.126:41021 was elected leader, sending a full tablet report...
W20240502 01:52:29.281630 12057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:29.281900 12057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:29.282027 12057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:29.289135 12057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:29.289311 12057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.69
I20240502 01:52:29.297812 12057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:42949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.11.42.69
--webserver_port=0
--tserver_master_addrs=127.11.42.126:41021
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.69
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:29.299013 12057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:29.300657 12057 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:29.312060 12062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:29.312285 12063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:29.312590 12065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:29.312847 12057 server_base.cc:1034] running on GCE node
I20240502 01:52:29.313410 12057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:29.314263 12057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:29.315469 12057 hybrid_clock.cc:648] HybridClock initialized: now 1714614749315399 us; error 44 us; skew 500 ppm
I20240502 01:52:29.315716 12057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:29.317754 12057 webserver.cc:460] Webserver started at http://127.11.42.69:42077/ using document root <none> and password file <none>
I20240502 01:52:29.318387 12057 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:29.318490 12057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:29.318733 12057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:29.321126 12057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "0d62c47f31894023a643c717f39abcea"
format_stamp: "Formatted at 2024-05-02 01:52:29 on dist-test-slave-xvk5"
I20240502 01:52:29.321823 12057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "0d62c47f31894023a643c717f39abcea"
format_stamp: "Formatted at 2024-05-02 01:52:29 on dist-test-slave-xvk5"
I20240502 01:52:29.326542 12057 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 01:52:29.329684 12071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:29.330861 12057 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20240502 01:52:29.330986 12057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "0d62c47f31894023a643c717f39abcea"
format_stamp: "Formatted at 2024-05-02 01:52:29 on dist-test-slave-xvk5"
I20240502 01:52:29.331130 12057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:29.346326 12057 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:29.346963 12057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:29.347189 12057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:29.347978 12057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:29.349397 12057 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:29.349467 12057 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:29.349572 12057 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:29.349617 12057 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:29.377503 12057 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.69:39271
I20240502 01:52:29.377552 12183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.69:39271 every 8 connection(s)
I20240502 01:52:29.378682 12057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:52:29.389005 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 12057
I20240502 01:52:29.389225 11433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 01:52:29.390216 12184 heartbeater.cc:347] Connected to a master server at 127.11.42.126:41021
I20240502 01:52:29.390489 12184 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:29.391381 12184 heartbeater.cc:511] Master 127.11.42.126:41021 requested a full tablet report, sending...
I20240502 01:52:29.392756 11475 ts_manager.cc:194] Registered new tserver with Master: 0d62c47f31894023a643c717f39abcea (127.11.42.69:39271)
I20240502 01:52:29.393733 11475 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.69:51995
I20240502 01:52:29.405267 11433 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 01:52:29.424921 11475 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37658:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 01:52:29.456962 12119 tablet_service.cc:1467] Processing CreateTablet for tablet 3cab96886c8842f399b7f5b66873e33b (DEFAULT_TABLE table=TestTable [id=863b2bda8b814df2971d7000451003af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:29.458575 12119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cab96886c8842f399b7f5b66873e33b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:29.458601 11989 tablet_service.cc:1467] Processing CreateTablet for tablet 3cab96886c8842f399b7f5b66873e33b (DEFAULT_TABLE table=TestTable [id=863b2bda8b814df2971d7000451003af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:29.458885 11598 tablet_service.cc:1467] Processing CreateTablet for tablet 3cab96886c8842f399b7f5b66873e33b (DEFAULT_TABLE table=TestTable [id=863b2bda8b814df2971d7000451003af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:29.459772 11989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cab96886c8842f399b7f5b66873e33b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:29.460240 11598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cab96886c8842f399b7f5b66873e33b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:29.469813 12202 tablet_bootstrap.cc:492] T 3cab96886c8842f399b7f5b66873e33b P 88568756bbbf4e759910daea9ddaf433: Bootstrap starting.
I20240502 01:52:29.471647 12203 tablet_bootstrap.cc:492] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea: Bootstrap starting.
I20240502 01:52:29.472925 12204 tablet_bootstrap.cc:492] T 3cab96886c8842f399b7f5b66873e33b P 6c8fe71c8f5e4aa3a0fd5113a9b24a61: Bootstrap starting.
I20240502 01:52:29.473205 12202 tablet_bootstrap.cc:654] T 3cab96886c8842f399b7f5b66873e33b P 88568756bbbf4e759910daea9ddaf433: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:29.474454 12202 log.cc:824] T 3cab96886c8842f399b7f5b66873e33b P 88568756bbbf4e759910daea9ddaf433: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:29.474862 12203 tablet_bootstrap.cc:654] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:29.475649 12204 tablet_bootstrap.cc:654] T 3cab96886c8842f399b7f5b66873e33b P 6c8fe71c8f5e4aa3a0fd5113a9b24a61: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:29.476197 12203 log.cc:824] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:29.476862 12204 log.cc:824] T 3cab96886c8842f399b7f5b66873e33b P 6c8fe71c8f5e4aa3a0fd5113a9b24a61: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:29.477816 12202 tablet_bootstrap.cc:492] T 3cab96886c8842f399b7f5b66873e33b P 88568756bbbf4e759910daea9ddaf433: No bootstrap required, opened a new log
I20240502 01:52:29.478121 12202 ts_tablet_manager.cc:1367] T 3cab96886c8842f399b7f5b66873e33b P 88568756bbbf4e759910daea9ddaf433: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20240502 01:52:29.478585 12203 tablet_bootstrap.cc:492] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea: No bootstrap required, opened a new log
I20240502 01:52:29.478842 12203 ts_tablet_manager.cc:1367] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20240502 01:52:29.479527 12204 tablet_bootstrap.cc:492] T 3cab96886c8842f399b7f5b66873e33b P 6c8fe71c8f5e4aa3a0fd5113a9b24a61: No bootstrap required, opened a new log
I20240502 01:52:29.479763 12204 ts_tablet_manager.cc:1367] T 3cab96886c8842f399b7f5b66873e33b P 6c8fe71c8f5e4aa3a0fd5113a9b24a61: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240502 01:52:29.486454 12203 raft_consensus.cc:348] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88568756bbbf4e759910daea9ddaf433" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 36219 } } peers { permanent_uuid: "6c8fe71c8f5e4aa3a0fd5113a9b24a61" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 44379 } } peers { permanent_uuid: "0d62c47f31894023a643c717f39abcea" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 39271 } }
I20240502 01:52:29.486668 12202 raft_consensus.cc:348] T 3cab96886c8842f399b7f5b66873e33b P 88568756bbbf4e759910daea9ddaf433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88568756bbbf4e759910daea9ddaf433" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 36219 } } peers { permanent_uuid: "6c8fe71c8f5e4aa3a0fd5113a9b24a61" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 44379 } } peers { permanent_uuid: "0d62c47f31894023a643c717f39abcea" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 39271 } }
I20240502 01:52:29.486970 12203 raft_consensus.cc:374] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:29.487076 12202 raft_consensus.cc:374] T 3cab96886c8842f399b7f5b66873e33b P 88568756bbbf4e759910daea9ddaf433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:29.487159 12203 raft_consensus.cc:729] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d62c47f31894023a643c717f39abcea, State: Initialized, Role: FOLLOWER
I20240502 01:52:29.487246 12202 raft_consensus.cc:729] T 3cab96886c8842f399b7f5b66873e33b P 88568756bbbf4e759910daea9ddaf433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88568756bbbf4e759910daea9ddaf433, State: Initialized, Role: FOLLOWER
I20240502 01:52:29.487897 12202 consensus_queue.cc:260] T 3cab96886c8842f399b7f5b66873e33b P 88568756bbbf4e759910daea9ddaf433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88568756bbbf4e759910daea9ddaf433" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 36219 } } peers { permanent_uuid: "6c8fe71c8f5e4aa3a0fd5113a9b24a61" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 44379 } } peers { permanent_uuid: "0d62c47f31894023a643c717f39abcea" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 39271 } }
I20240502 01:52:29.488117 12203 consensus_queue.cc:260] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88568756bbbf4e759910daea9ddaf433" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 36219 } } peers { permanent_uuid: "6c8fe71c8f5e4aa3a0fd5113a9b24a61" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 44379 } } peers { permanent_uuid: "0d62c47f31894023a643c717f39abcea" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 39271 } }
I20240502 01:52:29.488606 12204 raft_consensus.cc:348] T 3cab96886c8842f399b7f5b66873e33b P 6c8fe71c8f5e4aa3a0fd5113a9b24a61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88568756bbbf4e759910daea9ddaf433" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 36219 } } peers { permanent_uuid: "6c8fe71c8f5e4aa3a0fd5113a9b24a61" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 44379 } } peers { permanent_uuid: "0d62c47f31894023a643c717f39abcea" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 39271 } }
I20240502 01:52:29.489156 12204 raft_consensus.cc:374] T 3cab96886c8842f399b7f5b66873e33b P 6c8fe71c8f5e4aa3a0fd5113a9b24a61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:29.489334 12204 raft_consensus.cc:729] T 3cab96886c8842f399b7f5b66873e33b P 6c8fe71c8f5e4aa3a0fd5113a9b24a61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c8fe71c8f5e4aa3a0fd5113a9b24a61, State: Initialized, Role: FOLLOWER
I20240502 01:52:29.489984 12204 consensus_queue.cc:260] T 3cab96886c8842f399b7f5b66873e33b P 6c8fe71c8f5e4aa3a0fd5113a9b24a61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88568756bbbf4e759910daea9ddaf433" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 36219 } } peers { permanent_uuid: "6c8fe71c8f5e4aa3a0fd5113a9b24a61" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 44379 } } peers { permanent_uuid: "0d62c47f31894023a643c717f39abcea" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 39271 } }
I20240502 01:52:29.491621 12204 ts_tablet_manager.cc:1398] T 3cab96886c8842f399b7f5b66873e33b P 6c8fe71c8f5e4aa3a0fd5113a9b24a61: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20240502 01:52:29.492554 12054 heartbeater.cc:503] Master 127.11.42.126:41021 was elected leader, sending a full tablet report...
I20240502 01:52:29.493137 12202 ts_tablet_manager.cc:1398] T 3cab96886c8842f399b7f5b66873e33b P 88568756bbbf4e759910daea9ddaf433: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
I20240502 01:52:29.494405 12184 heartbeater.cc:503] Master 127.11.42.126:41021 was elected leader, sending a full tablet report...
I20240502 01:52:29.494930 12203 ts_tablet_manager.cc:1398] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea: Time spent starting tablet: real 0.016s user 0.010s sys 0.006s
I20240502 01:52:29.546278 11924 heartbeater.cc:503] Master 127.11.42.126:41021 was elected leader, sending a full tablet report...
W20240502 01:52:29.630211 12185 tablet.cc:2369] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:52:29.647688 12209 raft_consensus.cc:482] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:29.647928 12209 raft_consensus.cc:504] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88568756bbbf4e759910daea9ddaf433" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 36219 } } peers { permanent_uuid: "6c8fe71c8f5e4aa3a0fd5113a9b24a61" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 44379 } } peers { permanent_uuid: "0d62c47f31894023a643c717f39abcea" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 39271 } }
I20240502 01:52:29.649693 12209 leader_election.cc:288] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88568756bbbf4e759910daea9ddaf433 (127.11.42.68:36219), 6c8fe71c8f5e4aa3a0fd5113a9b24a61 (127.11.42.65:44379)
I20240502 01:52:29.656766 12009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cab96886c8842f399b7f5b66873e33b" candidate_uuid: "0d62c47f31894023a643c717f39abcea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88568756bbbf4e759910daea9ddaf433" is_pre_election: true
I20240502 01:52:29.657306 12009 raft_consensus.cc:2454] T 3cab96886c8842f399b7f5b66873e33b P 88568756bbbf4e759910daea9ddaf433 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d62c47f31894023a643c717f39abcea in term 0.
I20240502 01:52:29.657364 11618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cab96886c8842f399b7f5b66873e33b" candidate_uuid: "0d62c47f31894023a643c717f39abcea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c8fe71c8f5e4aa3a0fd5113a9b24a61" is_pre_election: true
I20240502 01:52:29.657974 11618 raft_consensus.cc:2454] T 3cab96886c8842f399b7f5b66873e33b P 6c8fe71c8f5e4aa3a0fd5113a9b24a61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d62c47f31894023a643c717f39abcea in term 0.
I20240502 01:52:29.657984 12072 leader_election.cc:302] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d62c47f31894023a643c717f39abcea, 88568756bbbf4e759910daea9ddaf433; no voters:
I20240502 01:52:29.658351 12209 raft_consensus.cc:2781] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:29.658514 12209 raft_consensus.cc:482] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:29.658630 12209 raft_consensus.cc:3037] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:29.661077 12209 raft_consensus.cc:504] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88568756bbbf4e759910daea9ddaf433" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 36219 } } peers { permanent_uuid: "6c8fe71c8f5e4aa3a0fd5113a9b24a61" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 44379 } } peers { permanent_uuid: "0d62c47f31894023a643c717f39abcea" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 39271 } }
I20240502 01:52:29.661767 12209 leader_election.cc:288] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea [CANDIDATE]: Term 1 election: Requested vote from peers 88568756bbbf4e759910daea9ddaf433 (127.11.42.68:36219), 6c8fe71c8f5e4aa3a0fd5113a9b24a61 (127.11.42.65:44379)
I20240502 01:52:29.662211 12009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cab96886c8842f399b7f5b66873e33b" candidate_uuid: "0d62c47f31894023a643c717f39abcea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88568756bbbf4e759910daea9ddaf433"
I20240502 01:52:29.662340 11618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cab96886c8842f399b7f5b66873e33b" candidate_uuid: "0d62c47f31894023a643c717f39abcea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c8fe71c8f5e4aa3a0fd5113a9b24a61"
I20240502 01:52:29.662410 12009 raft_consensus.cc:3037] T 3cab96886c8842f399b7f5b66873e33b P 88568756bbbf4e759910daea9ddaf433 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:29.662550 11618 raft_consensus.cc:3037] T 3cab96886c8842f399b7f5b66873e33b P 6c8fe71c8f5e4aa3a0fd5113a9b24a61 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:29.664845 12009 raft_consensus.cc:2454] T 3cab96886c8842f399b7f5b66873e33b P 88568756bbbf4e759910daea9ddaf433 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d62c47f31894023a643c717f39abcea in term 1.
I20240502 01:52:29.664930 11618 raft_consensus.cc:2454] T 3cab96886c8842f399b7f5b66873e33b P 6c8fe71c8f5e4aa3a0fd5113a9b24a61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d62c47f31894023a643c717f39abcea in term 1.
I20240502 01:52:29.665217 12072 leader_election.cc:302] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d62c47f31894023a643c717f39abcea, 88568756bbbf4e759910daea9ddaf433; no voters:
I20240502 01:52:29.665556 12209 raft_consensus.cc:2781] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:29.666018 12209 raft_consensus.cc:686] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea [term 1 LEADER]: Becoming Leader. State: Replica: 0d62c47f31894023a643c717f39abcea, State: Running, Role: LEADER
I20240502 01:52:29.666625 12209 consensus_queue.cc:237] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88568756bbbf4e759910daea9ddaf433" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 36219 } } peers { permanent_uuid: "6c8fe71c8f5e4aa3a0fd5113a9b24a61" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 44379 } } peers { permanent_uuid: "0d62c47f31894023a643c717f39abcea" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 39271 } }
W20240502 01:52:29.669451 11664 tablet.cc:2369] T 3cab96886c8842f399b7f5b66873e33b P 6c8fe71c8f5e4aa3a0fd5113a9b24a61: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:52:29.671293 11476 catalog_manager.cc:5484] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d62c47f31894023a643c717f39abcea (127.11.42.69). New cstate: current_term: 1 leader_uuid: "0d62c47f31894023a643c717f39abcea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88568756bbbf4e759910daea9ddaf433" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 36219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c8fe71c8f5e4aa3a0fd5113a9b24a61" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 44379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d62c47f31894023a643c717f39abcea" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 39271 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:29.688594 11433 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 01:52:29.691200 11433 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 6c8fe71c8f5e4aa3a0fd5113a9b24a61 to finish bootstrapping
I20240502 01:52:29.699788 11433 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 7ba4fdcc9de34d428e3ebc34c74f0436 to finish bootstrapping
I20240502 01:52:29.707655 11433 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 06a2ae4b0a294968a9c2ea6cdeaa62a3 to finish bootstrapping
I20240502 01:52:29.715637 11433 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 88568756bbbf4e759910daea9ddaf433 to finish bootstrapping
W20240502 01:52:29.716075 12055 tablet.cc:2369] T 3cab96886c8842f399b7f5b66873e33b P 88568756bbbf4e759910daea9ddaf433: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:52:29.722201 11433 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 0d62c47f31894023a643c717f39abcea to finish bootstrapping
I20240502 01:52:30.155755 12119 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:52:30.158457 11728 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:30.172537 11598 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:30.172546 11859 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:30.176754 11989 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:30.208005 12248 consensus_queue.cc:1035] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c8fe71c8f5e4aa3a0fd5113a9b24a61" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 44379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:30.233148 12251 consensus_queue.cc:1035] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea [LEADER]: Connected to new peer: Peer: permanent_uuid: "88568756bbbf4e759910daea9ddaf433" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 36219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:30.766749 11598 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:30.767087 12119 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:52:30.771584 11859 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:30.772985 11728 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:30.790585 11989 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:31.364784 11859 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:31.366554 11598 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:31.371168 11728 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:31.376739 11989 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:31.391636 12119 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:52:31.533703 12119 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:52:31.540304 11859 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:31.543192 11598 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:31.557330 11728 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:31.574601 11989 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:31.724117 11859 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:31.729508 11598 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:31.730782 11728 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:31.731185 12119 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:52:31.734442 11989 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:32.237699 11476 ts_manager.cc:292] Set tserver state for 6c8fe71c8f5e4aa3a0fd5113a9b24a61 to MAINTENANCE_MODE
I20240502 01:52:32.744431 11598 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:32.749650 11728 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:32.750113 11859 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:32.754997 12119 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:52:32.775671 11989 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:32.917410 12119 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:52:32.920917 11728 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:32.922912 11859 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:32.924443 11598 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:32.932068 11989 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:33.094105 12119 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:52:33.098039 11598 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:33.098587 11859 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:33.110177 11728 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:33.124565 11989 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:33.180616 12139 consensus_queue.cc:237] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "88568756bbbf4e759910daea9ddaf433" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 36219 } } peers { permanent_uuid: "6c8fe71c8f5e4aa3a0fd5113a9b24a61" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 44379 } attrs { replace: true } } peers { permanent_uuid: "0d62c47f31894023a643c717f39abcea" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 39271 } }
I20240502 01:52:33.182770 11618 raft_consensus.cc:1261] T 3cab96886c8842f399b7f5b66873e33b P 6c8fe71c8f5e4aa3a0fd5113a9b24a61 [term 1 FOLLOWER]: Refusing update from remote peer 0d62c47f31894023a643c717f39abcea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:52:33.182849 12009 raft_consensus.cc:1261] T 3cab96886c8842f399b7f5b66873e33b P 88568756bbbf4e759910daea9ddaf433 [term 1 FOLLOWER]: Refusing update from remote peer 0d62c47f31894023a643c717f39abcea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:52:33.183389 12410 consensus_queue.cc:1035] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea [LEADER]: Connected to new peer: Peer: permanent_uuid: "88568756bbbf4e759910daea9ddaf433" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 36219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:52:33.183712 12526 consensus_queue.cc:1035] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c8fe71c8f5e4aa3a0fd5113a9b24a61" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 44379 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:52:33.187059 12410 raft_consensus.cc:2932] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "88568756bbbf4e759910daea9ddaf433" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 36219 } } peers { permanent_uuid: "6c8fe71c8f5e4aa3a0fd5113a9b24a61" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 44379 } attrs { replace: true } } peers { permanent_uuid: "0d62c47f31894023a643c717f39abcea" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 39271 } } }
I20240502 01:52:33.187363 12009 raft_consensus.cc:2932] T 3cab96886c8842f399b7f5b66873e33b P 88568756bbbf4e759910daea9ddaf433 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "88568756bbbf4e759910daea9ddaf433" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 36219 } } peers { permanent_uuid: "6c8fe71c8f5e4aa3a0fd5113a9b24a61" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 44379 } attrs { replace: true } } peers { permanent_uuid: "0d62c47f31894023a643c717f39abcea" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 39271 } } }
I20240502 01:52:33.187891 11618 raft_consensus.cc:2932] T 3cab96886c8842f399b7f5b66873e33b P 6c8fe71c8f5e4aa3a0fd5113a9b24a61 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "88568756bbbf4e759910daea9ddaf433" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 36219 } } peers { permanent_uuid: "6c8fe71c8f5e4aa3a0fd5113a9b24a61" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 44379 } attrs { replace: true } } peers { permanent_uuid: "0d62c47f31894023a643c717f39abcea" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 39271 } } }
I20240502 01:52:33.191936 11476 catalog_manager.cc:5484] T 3cab96886c8842f399b7f5b66873e33b P 88568756bbbf4e759910daea9ddaf433 reported cstate change: config changed from index -1 to 2. New cstate: current_term: 1 leader_uuid: "0d62c47f31894023a643c717f39abcea" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "88568756bbbf4e759910daea9ddaf433" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 36219 } } peers { permanent_uuid: "6c8fe71c8f5e4aa3a0fd5113a9b24a61" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 44379 } attrs { replace: true } } peers { permanent_uuid: "0d62c47f31894023a643c717f39abcea" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 39271 } } }
I20240502 01:52:33.202281 12139 consensus_queue.cc:237] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "88568756bbbf4e759910daea9ddaf433" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 36219 } } peers { permanent_uuid: "6c8fe71c8f5e4aa3a0fd5113a9b24a61" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 44379 } attrs { replace: true } } peers { permanent_uuid: "0d62c47f31894023a643c717f39abcea" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 39271 } } peers { permanent_uuid: "06a2ae4b0a294968a9c2ea6cdeaa62a3" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36373 } attrs { promote: true } }
I20240502 01:52:33.204517 12009 raft_consensus.cc:1261] T 3cab96886c8842f399b7f5b66873e33b P 88568756bbbf4e759910daea9ddaf433 [term 1 FOLLOWER]: Refusing update from remote peer 0d62c47f31894023a643c717f39abcea: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:52:33.205189 11618 raft_consensus.cc:1261] T 3cab96886c8842f399b7f5b66873e33b P 6c8fe71c8f5e4aa3a0fd5113a9b24a61 [term 1 FOLLOWER]: Refusing update from remote peer 0d62c47f31894023a643c717f39abcea: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:52:33.205121 12529 consensus_queue.cc:1035] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea [LEADER]: Connected to new peer: Peer: permanent_uuid: "88568756bbbf4e759910daea9ddaf433" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 36219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:52:33.205757 12529 consensus_queue.cc:1035] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c8fe71c8f5e4aa3a0fd5113a9b24a61" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 44379 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:52:33.207665 12526 raft_consensus.cc:2932] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 06a2ae4b0a294968a9c2ea6cdeaa62a3 (127.11.42.67) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "88568756bbbf4e759910daea9ddaf433" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 36219 } } peers { permanent_uuid: "6c8fe71c8f5e4aa3a0fd5113a9b24a61" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 44379 } attrs { replace: true } } peers { permanent_uuid: "0d62c47f31894023a643c717f39abcea" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 39271 } } peers { permanent_uuid: "06a2ae4b0a294968a9c2ea6cdeaa62a3" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36373 } attrs { promote: true } } }
I20240502 01:52:33.209555 12009 raft_consensus.cc:2932] T 3cab96886c8842f399b7f5b66873e33b P 88568756bbbf4e759910daea9ddaf433 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 06a2ae4b0a294968a9c2ea6cdeaa62a3 (127.11.42.67) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "88568756bbbf4e759910daea9ddaf433" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 36219 } } peers { permanent_uuid: "6c8fe71c8f5e4aa3a0fd5113a9b24a61" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 44379 } attrs { replace: true } } peers { permanent_uuid: "0d62c47f31894023a643c717f39abcea" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 39271 } } peers { permanent_uuid: "06a2ae4b0a294968a9c2ea6cdeaa62a3" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36373 } attrs { promote: true } } }
I20240502 01:52:33.210108 11618 raft_consensus.cc:2932] T 3cab96886c8842f399b7f5b66873e33b P 6c8fe71c8f5e4aa3a0fd5113a9b24a61 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 06a2ae4b0a294968a9c2ea6cdeaa62a3 (127.11.42.67) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "88568756bbbf4e759910daea9ddaf433" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 36219 } } peers { permanent_uuid: "6c8fe71c8f5e4aa3a0fd5113a9b24a61" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 44379 } attrs { replace: true } } peers { permanent_uuid: "0d62c47f31894023a643c717f39abcea" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 39271 } } peers { permanent_uuid: "06a2ae4b0a294968a9c2ea6cdeaa62a3" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36373 } attrs { promote: true } } }
I20240502 01:52:33.211490 11462 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3cab96886c8842f399b7f5b66873e33b with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 01:52:33.212433 11475 catalog_manager.cc:5484] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea reported cstate change: config changed from index 2 to 3, NON_VOTER 06a2ae4b0a294968a9c2ea6cdeaa62a3 (127.11.42.67) added. New cstate: current_term: 1 leader_uuid: "0d62c47f31894023a643c717f39abcea" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "88568756bbbf4e759910daea9ddaf433" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 36219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c8fe71c8f5e4aa3a0fd5113a9b24a61" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 44379 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d62c47f31894023a643c717f39abcea" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 39271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06a2ae4b0a294968a9c2ea6cdeaa62a3" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36373 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:52:33.214049 12072 consensus_peers.cc:485] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea -> Peer 06a2ae4b0a294968a9c2ea6cdeaa62a3 (127.11.42.67:36373): Couldn't send request to peer 06a2ae4b0a294968a9c2ea6cdeaa62a3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3cab96886c8842f399b7f5b66873e33b. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:52:33.613513 12546 ts_tablet_manager.cc:902] T 3cab96886c8842f399b7f5b66873e33b P 06a2ae4b0a294968a9c2ea6cdeaa62a3: Initiating tablet copy from peer 0d62c47f31894023a643c717f39abcea (127.11.42.69:39271)
I20240502 01:52:33.615239 12546 tablet_copy_client.cc:308] T 3cab96886c8842f399b7f5b66873e33b P 06a2ae4b0a294968a9c2ea6cdeaa62a3: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.69:39271
I20240502 01:52:33.622241 12159 tablet_copy_service.cc:140] P 0d62c47f31894023a643c717f39abcea: Received BeginTabletCopySession request for tablet 3cab96886c8842f399b7f5b66873e33b from peer 06a2ae4b0a294968a9c2ea6cdeaa62a3 ({username='slave'} at 127.11.42.67:52815)
I20240502 01:52:33.622491 12159 tablet_copy_service.cc:161] P 0d62c47f31894023a643c717f39abcea: Beginning new tablet copy session on tablet 3cab96886c8842f399b7f5b66873e33b from peer 06a2ae4b0a294968a9c2ea6cdeaa62a3 at {username='slave'} at 127.11.42.67:52815: session id = 06a2ae4b0a294968a9c2ea6cdeaa62a3-3cab96886c8842f399b7f5b66873e33b
I20240502 01:52:33.626003 12159 tablet_copy_source_session.cc:205] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:52:33.630415 12546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cab96886c8842f399b7f5b66873e33b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.639137 12546 tablet_copy_client.cc:783] T 3cab96886c8842f399b7f5b66873e33b P 06a2ae4b0a294968a9c2ea6cdeaa62a3: tablet copy: Starting download of 0 data blocks...
I20240502 01:52:33.639432 12546 tablet_copy_client.cc:647] T 3cab96886c8842f399b7f5b66873e33b P 06a2ae4b0a294968a9c2ea6cdeaa62a3: tablet copy: Starting download of 1 WAL segments...
I20240502 01:52:33.641388 12546 tablet_copy_client.cc:523] T 3cab96886c8842f399b7f5b66873e33b P 06a2ae4b0a294968a9c2ea6cdeaa62a3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:52:33.643999 12546 tablet_bootstrap.cc:492] T 3cab96886c8842f399b7f5b66873e33b P 06a2ae4b0a294968a9c2ea6cdeaa62a3: Bootstrap starting.
I20240502 01:52:33.650070 12546 log.cc:824] T 3cab96886c8842f399b7f5b66873e33b P 06a2ae4b0a294968a9c2ea6cdeaa62a3: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:33.653357 12546 tablet_bootstrap.cc:492] T 3cab96886c8842f399b7f5b66873e33b P 06a2ae4b0a294968a9c2ea6cdeaa62a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:52:33.653679 12546 tablet_bootstrap.cc:492] T 3cab96886c8842f399b7f5b66873e33b P 06a2ae4b0a294968a9c2ea6cdeaa62a3: Bootstrap complete.
I20240502 01:52:33.654036 12546 ts_tablet_manager.cc:1367] T 3cab96886c8842f399b7f5b66873e33b P 06a2ae4b0a294968a9c2ea6cdeaa62a3: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.009s
I20240502 01:52:33.659533 12546 raft_consensus.cc:348] T 3cab96886c8842f399b7f5b66873e33b P 06a2ae4b0a294968a9c2ea6cdeaa62a3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "88568756bbbf4e759910daea9ddaf433" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 36219 } } peers { permanent_uuid: "6c8fe71c8f5e4aa3a0fd5113a9b24a61" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 44379 } attrs { replace: true } } peers { permanent_uuid: "0d62c47f31894023a643c717f39abcea" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 39271 } } peers { permanent_uuid: "06a2ae4b0a294968a9c2ea6cdeaa62a3" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36373 } attrs { promote: true } }
I20240502 01:52:33.659902 12546 raft_consensus.cc:729] T 3cab96886c8842f399b7f5b66873e33b P 06a2ae4b0a294968a9c2ea6cdeaa62a3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 06a2ae4b0a294968a9c2ea6cdeaa62a3, State: Initialized, Role: LEARNER
I20240502 01:52:33.660279 12546 consensus_queue.cc:260] T 3cab96886c8842f399b7f5b66873e33b P 06a2ae4b0a294968a9c2ea6cdeaa62a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "88568756bbbf4e759910daea9ddaf433" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 36219 } } peers { permanent_uuid: "6c8fe71c8f5e4aa3a0fd5113a9b24a61" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 44379 } attrs { replace: true } } peers { permanent_uuid: "0d62c47f31894023a643c717f39abcea" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 39271 } } peers { permanent_uuid: "06a2ae4b0a294968a9c2ea6cdeaa62a3" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36373 } attrs { promote: true } }
I20240502 01:52:33.661751 12546 ts_tablet_manager.cc:1398] T 3cab96886c8842f399b7f5b66873e33b P 06a2ae4b0a294968a9c2ea6cdeaa62a3: Time spent starting tablet: real 0.008s user 0.003s sys 0.005s
I20240502 01:52:33.662639 12159 tablet_copy_service.cc:342] P 0d62c47f31894023a643c717f39abcea: Request end of tablet copy session 06a2ae4b0a294968a9c2ea6cdeaa62a3-3cab96886c8842f399b7f5b66873e33b received from {username='slave'} at 127.11.42.67:52815
I20240502 01:52:33.662781 12159 tablet_copy_service.cc:433] P 0d62c47f31894023a643c717f39abcea: ending tablet copy session 06a2ae4b0a294968a9c2ea6cdeaa62a3-3cab96886c8842f399b7f5b66873e33b on tablet 3cab96886c8842f399b7f5b66873e33b with peer 06a2ae4b0a294968a9c2ea6cdeaa62a3
I20240502 01:52:34.044937 11879 raft_consensus.cc:1203] T 3cab96886c8842f399b7f5b66873e33b P 06a2ae4b0a294968a9c2ea6cdeaa62a3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
I20240502 01:52:34.436407 12570 raft_consensus.cc:1050] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea: attempting to promote NON_VOTER 06a2ae4b0a294968a9c2ea6cdeaa62a3 to VOTER
I20240502 01:52:34.437232 12570 consensus_queue.cc:237] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88568756bbbf4e759910daea9ddaf433" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 36219 } } peers { permanent_uuid: "6c8fe71c8f5e4aa3a0fd5113a9b24a61" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 44379 } attrs { replace: true } } peers { permanent_uuid: "0d62c47f31894023a643c717f39abcea" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 39271 } } peers { permanent_uuid: "06a2ae4b0a294968a9c2ea6cdeaa62a3" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36373 } attrs { promote: false } }
I20240502 01:52:34.439724 12009 raft_consensus.cc:1261] T 3cab96886c8842f399b7f5b66873e33b P 88568756bbbf4e759910daea9ddaf433 [term 1 FOLLOWER]: Refusing update from remote peer 0d62c47f31894023a643c717f39abcea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:52:34.439711 11879 raft_consensus.cc:1261] T 3cab96886c8842f399b7f5b66873e33b P 06a2ae4b0a294968a9c2ea6cdeaa62a3 [term 1 LEARNER]: Refusing update from remote peer 0d62c47f31894023a643c717f39abcea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:52:34.439774 11618 raft_consensus.cc:1261] T 3cab96886c8842f399b7f5b66873e33b P 6c8fe71c8f5e4aa3a0fd5113a9b24a61 [term 1 FOLLOWER]: Refusing update from remote peer 0d62c47f31894023a643c717f39abcea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:52:34.440361 12570 consensus_queue.cc:1035] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c8fe71c8f5e4aa3a0fd5113a9b24a61" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 44379 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:52:34.440692 12578 consensus_queue.cc:1035] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea [LEADER]: Connected to new peer: Peer: permanent_uuid: "06a2ae4b0a294968a9c2ea6cdeaa62a3" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36373 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:52:34.441031 12571 consensus_queue.cc:1035] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea [LEADER]: Connected to new peer: Peer: permanent_uuid: "88568756bbbf4e759910daea9ddaf433" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 36219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:52:34.443930 12570 raft_consensus.cc:2932] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 06a2ae4b0a294968a9c2ea6cdeaa62a3 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88568756bbbf4e759910daea9ddaf433" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 36219 } } peers { permanent_uuid: "6c8fe71c8f5e4aa3a0fd5113a9b24a61" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 44379 } attrs { replace: true } } peers { permanent_uuid: "0d62c47f31894023a643c717f39abcea" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 39271 } } peers { permanent_uuid: "06a2ae4b0a294968a9c2ea6cdeaa62a3" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36373 } attrs { promote: false } } }
I20240502 01:52:34.445695 11879 raft_consensus.cc:2932] T 3cab96886c8842f399b7f5b66873e33b P 06a2ae4b0a294968a9c2ea6cdeaa62a3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 06a2ae4b0a294968a9c2ea6cdeaa62a3 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88568756bbbf4e759910daea9ddaf433" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 36219 } } peers { permanent_uuid: "6c8fe71c8f5e4aa3a0fd5113a9b24a61" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 44379 } attrs { replace: true } } peers { permanent_uuid: "0d62c47f31894023a643c717f39abcea" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 39271 } } peers { permanent_uuid: "06a2ae4b0a294968a9c2ea6cdeaa62a3" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36373 } attrs { promote: false } } }
I20240502 01:52:34.447638 12009 raft_consensus.cc:2932] T 3cab96886c8842f399b7f5b66873e33b P 88568756bbbf4e759910daea9ddaf433 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 06a2ae4b0a294968a9c2ea6cdeaa62a3 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88568756bbbf4e759910daea9ddaf433" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 36219 } } peers { permanent_uuid: "6c8fe71c8f5e4aa3a0fd5113a9b24a61" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 44379 } attrs { replace: true } } peers { permanent_uuid: "0d62c47f31894023a643c717f39abcea" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 39271 } } peers { permanent_uuid: "06a2ae4b0a294968a9c2ea6cdeaa62a3" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36373 } attrs { promote: false } } }
I20240502 01:52:34.448415 11618 raft_consensus.cc:2932] T 3cab96886c8842f399b7f5b66873e33b P 6c8fe71c8f5e4aa3a0fd5113a9b24a61 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 06a2ae4b0a294968a9c2ea6cdeaa62a3 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88568756bbbf4e759910daea9ddaf433" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 36219 } } peers { permanent_uuid: "6c8fe71c8f5e4aa3a0fd5113a9b24a61" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 44379 } attrs { replace: true } } peers { permanent_uuid: "0d62c47f31894023a643c717f39abcea" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 39271 } } peers { permanent_uuid: "06a2ae4b0a294968a9c2ea6cdeaa62a3" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36373 } attrs { promote: false } } }
I20240502 01:52:34.448976 11474 catalog_manager.cc:5484] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea reported cstate change: config changed from index 3 to 4, 06a2ae4b0a294968a9c2ea6cdeaa62a3 (127.11.42.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0d62c47f31894023a643c717f39abcea" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88568756bbbf4e759910daea9ddaf433" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 36219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c8fe71c8f5e4aa3a0fd5113a9b24a61" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 44379 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d62c47f31894023a643c717f39abcea" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 39271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06a2ae4b0a294968a9c2ea6cdeaa62a3" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36373 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:52:34.457492 12139 consensus_queue.cc:237] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88568756bbbf4e759910daea9ddaf433" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 36219 } } peers { permanent_uuid: "0d62c47f31894023a643c717f39abcea" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 39271 } } peers { permanent_uuid: "06a2ae4b0a294968a9c2ea6cdeaa62a3" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36373 } attrs { promote: false } }
I20240502 01:52:34.459187 11879 raft_consensus.cc:1261] T 3cab96886c8842f399b7f5b66873e33b P 06a2ae4b0a294968a9c2ea6cdeaa62a3 [term 1 FOLLOWER]: Refusing update from remote peer 0d62c47f31894023a643c717f39abcea: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:52:34.459421 12009 raft_consensus.cc:1261] T 3cab96886c8842f399b7f5b66873e33b P 88568756bbbf4e759910daea9ddaf433 [term 1 FOLLOWER]: Refusing update from remote peer 0d62c47f31894023a643c717f39abcea: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:52:34.459729 12578 consensus_queue.cc:1035] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea [LEADER]: Connected to new peer: Peer: permanent_uuid: "06a2ae4b0a294968a9c2ea6cdeaa62a3" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36373 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:52:34.460394 12578 consensus_queue.cc:1035] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea [LEADER]: Connected to new peer: Peer: permanent_uuid: "88568756bbbf4e759910daea9ddaf433" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 36219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:52:34.462579 12570 raft_consensus.cc:2932] T 3cab96886c8842f399b7f5b66873e33b P 0d62c47f31894023a643c717f39abcea [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6c8fe71c8f5e4aa3a0fd5113a9b24a61 (127.11.42.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88568756bbbf4e759910daea9ddaf433" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 36219 } } peers { permanent_uuid: "0d62c47f31894023a643c717f39abcea" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 39271 } } peers { permanent_uuid: "06a2ae4b0a294968a9c2ea6cdeaa62a3" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36373 } attrs { promote: false } } }
I20240502 01:52:34.463238 12009 raft_consensus.cc:2932] T 3cab96886c8842f399b7f5b66873e33b P 88568756bbbf4e759910daea9ddaf433 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6c8fe71c8f5e4aa3a0fd5113a9b24a61 (127.11.42.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88568756bbbf4e759910daea9ddaf433" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 36219 } } peers { permanent_uuid: "0d62c47f31894023a643c717f39abcea" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 39271 } } peers { permanent_uuid: "06a2ae4b0a294968a9c2ea6cdeaa62a3" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36373 } attrs { promote: false } } }
I20240502 01:52:34.466388 11462 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3cab96886c8842f399b7f5b66873e33b with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:52:34.466958 11473 catalog_manager.cc:5484] T 3cab96886c8842f399b7f5b66873e33b P 88568756bbbf4e759910daea9ddaf433 reported cstate change: config changed from index 4 to 5, VOTER 6c8fe71c8f5e4aa3a0fd5113a9b24a61 (127.11.42.65) evicted. New cstate: current_term: 1 leader_uuid: "0d62c47f31894023a643c717f39abcea" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88568756bbbf4e759910daea9ddaf433" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 36219 } } peers { permanent_uuid: "0d62c47f31894023a643c717f39abcea" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 39271 } } peers { permanent_uuid: "06a2ae4b0a294968a9c2ea6cdeaa62a3" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36373 } attrs { promote: false } } }
I20240502 01:52:34.467597 11879 raft_consensus.cc:2932] T 3cab96886c8842f399b7f5b66873e33b P 06a2ae4b0a294968a9c2ea6cdeaa62a3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6c8fe71c8f5e4aa3a0fd5113a9b24a61 (127.11.42.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88568756bbbf4e759910daea9ddaf433" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 36219 } } peers { permanent_uuid: "0d62c47f31894023a643c717f39abcea" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 39271 } } peers { permanent_uuid: "06a2ae4b0a294968a9c2ea6cdeaa62a3" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36373 } attrs { promote: false } } }
I20240502 01:52:34.471522 11598 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cab96886c8842f399b7f5b66873e33b with delete_type TABLET_DATA_TOMBSTONED (TS 6c8fe71c8f5e4aa3a0fd5113a9b24a61 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:53850
I20240502 01:52:34.473140 12590 tablet_replica.cc:330] T 3cab96886c8842f399b7f5b66873e33b P 6c8fe71c8f5e4aa3a0fd5113a9b24a61: stopping tablet replica
I20240502 01:52:34.473635 12590 raft_consensus.cc:2229] T 3cab96886c8842f399b7f5b66873e33b P 6c8fe71c8f5e4aa3a0fd5113a9b24a61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:52:34.474151 12590 raft_consensus.cc:2258] T 3cab96886c8842f399b7f5b66873e33b P 6c8fe71c8f5e4aa3a0fd5113a9b24a61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:34.475844 12590 ts_tablet_manager.cc:1875] T 3cab96886c8842f399b7f5b66873e33b P 6c8fe71c8f5e4aa3a0fd5113a9b24a61: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:52:34.480654 12590 ts_tablet_manager.cc:1888] T 3cab96886c8842f399b7f5b66873e33b P 6c8fe71c8f5e4aa3a0fd5113a9b24a61: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:52:34.480813 12590 log.cc:1196] T 3cab96886c8842f399b7f5b66873e33b P 6c8fe71c8f5e4aa3a0fd5113a9b24a61: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/3cab96886c8842f399b7f5b66873e33b
I20240502 01:52:34.481734 11460 catalog_manager.cc:4830] TS 6c8fe71c8f5e4aa3a0fd5113a9b24a61 (127.11.42.65:44379): tablet 3cab96886c8842f399b7f5b66873e33b (table TestTable [id=863b2bda8b814df2971d7000451003af]) successfully deleted
I20240502 01:52:34.773294 11859 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:34.775480 12119 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:52:34.789533 11728 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:34.807515 11989 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:34.808040 11598 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:34.947156 11859 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:34.950408 12119 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:52:34.951462 11598 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:34.958492 11728 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:34.958887 11989 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:35.102596 12119 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:52:35.113802 11598 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:35.120967 11728 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:35.125442 11989 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:35.136333 11859 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:35.288967 11433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu pid 11666
I20240502 01:52:35.290406 11788 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:35.390372 11433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu with pid 11666
I20240502 01:52:35.855816 12119 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:52:35.866456 11859 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:35.866751 11598 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:35.870254 11989 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:35.988124 11859 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:35.993178 12119 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:52:36.003674 11598 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:36.011338 11989 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:36.169302 11859 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:36.169905 12119 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:52:36.172137 11598 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:36.184578 11989 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:36.304265 11859 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:36.307935 11598 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:36.311682 12119 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:52:36.329314 11989 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:36.466454 11433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu pid 11535
I20240502 01:52:36.467939 11657 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:36.582690 11433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu with pid 11535
I20240502 01:52:36.600503 11433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu pid 11797
I20240502 01:52:36.601639 11919 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:36.714859 11433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu with pid 11797
I20240502 01:52:36.732398 11433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu pid 11927
I20240502 01:52:36.733407 12048 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:36.847899 11433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu with pid 11927
I20240502 01:52:36.865950 11433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu pid 12057
I20240502 01:52:36.866950 12178 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:36.982445 11433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu with pid 12057
I20240502 01:52:37.000021 11433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu pid 11445
I20240502 01:52:37.001152 11506 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:37.136066 11433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu with pid 11445
2024-05-02T01:52:37Z chronyd exiting
[ OK ] RebalanceIgnoredTserversTest.Basic (10463 ms)
[----------] 1 test from RebalanceIgnoredTserversTest (10463 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.TabletServerIsDown/1
I20240502 01:52:37.177699 11433 test_util.cc:268] Using random seed: -611894985
I20240502 01:52:37.179410 11433 ts_itest-base.cc:112] Starting cluster with:
I20240502 01:52:37.179488 11433 ts_itest-base.cc:113] --------------
I20240502 01:52:37.179530 11433 ts_itest-base.cc:114] 5 tablet servers
I20240502 01:52:37.179579 11433 ts_itest-base.cc:115] 3 replicas per TS
I20240502 01:52:37.179620 11433 ts_itest-base.cc:116] --------------
2024-05-02T01:52:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T01:52:37Z Disabled control of system clock
I20240502 01:52:37.200373 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.42.126:44219
--webserver_interface=127.11.42.126
--webserver_port=0
--builtin_ntp_servers=127.11.42.84:37361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.42.126:44219
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 01:52:37.507004 12817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:37.507288 12817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:37.507421 12817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:37.514571 12817 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:37.514684 12817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 01:52:37.514761 12817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:37.514839 12817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 01:52:37.514899 12817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 01:52:37.523551 12817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:37361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.11.42.126:44219
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.42.126:44219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.11.42.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:37.524727 12817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:37.526373 12817 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:37.536435 12822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:37.536753 12823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:37.537031 12825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:37.537379 12817 server_base.cc:1034] running on GCE node
I20240502 01:52:37.538055 12817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:37.539278 12817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:37.540480 12817 hybrid_clock.cc:648] HybridClock initialized: now 1714614757540433 us; error 41 us; skew 500 ppm
I20240502 01:52:37.541085 12817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:37.549147 12817 webserver.cc:460] Webserver started at http://127.11.42.126:38943/ using document root <none> and password file <none>
I20240502 01:52:37.549827 12817 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:37.549939 12817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:37.550249 12817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:37.552723 12817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "daea235e1e50474182f69cecc600bf81"
format_stamp: "Formatted at 2024-05-02 01:52:37 on dist-test-slave-xvk5"
I20240502 01:52:37.553499 12817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "daea235e1e50474182f69cecc600bf81"
format_stamp: "Formatted at 2024-05-02 01:52:37 on dist-test-slave-xvk5"
I20240502 01:52:37.559223 12817 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 01:52:37.562669 12831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:37.563848 12817 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 01:52:37.564010 12817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/wal
uuid: "daea235e1e50474182f69cecc600bf81"
format_stamp: "Formatted at 2024-05-02 01:52:37 on dist-test-slave-xvk5"
I20240502 01:52:37.564184 12817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:37.596158 12817 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:37.596982 12817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:37.597318 12817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:37.617431 12817 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.126:44219
I20240502 01:52:37.617493 12882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.126:44219 every 8 connection(s)
I20240502 01:52:37.618739 12817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 01:52:37.622308 12883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:37.623881 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 12817
I20240502 01:52:37.624076 11433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 01:52:37.633711 12883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daea235e1e50474182f69cecc600bf81: Bootstrap starting.
I20240502 01:52:37.637372 12883 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P daea235e1e50474182f69cecc600bf81: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:37.638572 12883 log.cc:824] T 00000000000000000000000000000000 P daea235e1e50474182f69cecc600bf81: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:37.641089 12883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daea235e1e50474182f69cecc600bf81: No bootstrap required, opened a new log
I20240502 01:52:37.647051 12883 raft_consensus.cc:348] T 00000000000000000000000000000000 P daea235e1e50474182f69cecc600bf81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daea235e1e50474182f69cecc600bf81" member_type: VOTER last_known_addr { host: "127.11.42.126" port: 44219 } }
I20240502 01:52:37.647365 12883 raft_consensus.cc:374] T 00000000000000000000000000000000 P daea235e1e50474182f69cecc600bf81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:37.647459 12883 raft_consensus.cc:729] T 00000000000000000000000000000000 P daea235e1e50474182f69cecc600bf81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daea235e1e50474182f69cecc600bf81, State: Initialized, Role: FOLLOWER
I20240502 01:52:37.647997 12883 consensus_queue.cc:260] T 00000000000000000000000000000000 P daea235e1e50474182f69cecc600bf81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daea235e1e50474182f69cecc600bf81" member_type: VOTER last_known_addr { host: "127.11.42.126" port: 44219 } }
I20240502 01:52:37.648183 12883 raft_consensus.cc:388] T 00000000000000000000000000000000 P daea235e1e50474182f69cecc600bf81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:37.648303 12883 raft_consensus.cc:482] T 00000000000000000000000000000000 P daea235e1e50474182f69cecc600bf81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:37.648437 12883 raft_consensus.cc:3037] T 00000000000000000000000000000000 P daea235e1e50474182f69cecc600bf81 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:37.650561 12883 raft_consensus.cc:504] T 00000000000000000000000000000000 P daea235e1e50474182f69cecc600bf81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daea235e1e50474182f69cecc600bf81" member_type: VOTER last_known_addr { host: "127.11.42.126" port: 44219 } }
I20240502 01:52:37.651073 12883 leader_election.cc:302] T 00000000000000000000000000000000 P daea235e1e50474182f69cecc600bf81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: daea235e1e50474182f69cecc600bf81; no voters:
I20240502 01:52:37.651583 12883 leader_election.cc:288] T 00000000000000000000000000000000 P daea235e1e50474182f69cecc600bf81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:37.651706 12888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P daea235e1e50474182f69cecc600bf81 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:37.652283 12888 raft_consensus.cc:686] T 00000000000000000000000000000000 P daea235e1e50474182f69cecc600bf81 [term 1 LEADER]: Becoming Leader. State: Replica: daea235e1e50474182f69cecc600bf81, State: Running, Role: LEADER
I20240502 01:52:37.652765 12888 consensus_queue.cc:237] T 00000000000000000000000000000000 P daea235e1e50474182f69cecc600bf81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daea235e1e50474182f69cecc600bf81" member_type: VOTER last_known_addr { host: "127.11.42.126" port: 44219 } }
I20240502 01:52:37.653348 12883 sys_catalog.cc:564] T 00000000000000000000000000000000 P daea235e1e50474182f69cecc600bf81 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:52:37.656083 12889 sys_catalog.cc:455] T 00000000000000000000000000000000 P daea235e1e50474182f69cecc600bf81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "daea235e1e50474182f69cecc600bf81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daea235e1e50474182f69cecc600bf81" member_type: VOTER last_known_addr { host: "127.11.42.126" port: 44219 } } }
I20240502 01:52:37.656443 12889 sys_catalog.cc:458] T 00000000000000000000000000000000 P daea235e1e50474182f69cecc600bf81 [sys.catalog]: This master's current role is: LEADER
I20240502 01:52:37.657053 12890 sys_catalog.cc:455] T 00000000000000000000000000000000 P daea235e1e50474182f69cecc600bf81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader daea235e1e50474182f69cecc600bf81. Latest consensus state: current_term: 1 leader_uuid: "daea235e1e50474182f69cecc600bf81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daea235e1e50474182f69cecc600bf81" member_type: VOTER last_known_addr { host: "127.11.42.126" port: 44219 } } }
I20240502 01:52:37.657305 12890 sys_catalog.cc:458] T 00000000000000000000000000000000 P daea235e1e50474182f69cecc600bf81 [sys.catalog]: This master's current role is: LEADER
I20240502 01:52:37.662004 12896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:52:37.673154 12896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:52:37.687772 12896 catalog_manager.cc:1347] Generated new cluster ID: 2326af819d5242c6a138070c1f506a21
I20240502 01:52:37.687906 12896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:52:37.720060 12896 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:52:37.721774 12896 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:52:37.737562 12896 catalog_manager.cc:5857] T 00000000000000000000000000000000 P daea235e1e50474182f69cecc600bf81: Generated new TSK 0
I20240502 01:52:37.738601 12896 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:52:37.757424 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.65:0
--local_ip_for_outbound_sockets=127.11.42.65
--webserver_interface=127.11.42.65
--webserver_port=0
--tserver_master_addrs=127.11.42.126:44219
--builtin_ntp_servers=127.11.42.84:37361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 01:52:38.066704 12907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:38.066988 12907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:38.067112 12907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:38.074455 12907 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:38.074577 12907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:38.074726 12907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.65
I20240502 01:52:38.083765 12907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:37361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.11.42.65
--webserver_port=0
--tserver_master_addrs=127.11.42.126:44219
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.65
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:38.084975 12907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:38.086959 12907 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:38.098376 12912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:38.098508 12915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:38.098865 12913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:38.099241 12907 server_base.cc:1034] running on GCE node
I20240502 01:52:38.099782 12907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:38.100764 12907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:38.101963 12907 hybrid_clock.cc:648] HybridClock initialized: now 1714614758101901 us; error 56 us; skew 500 ppm
I20240502 01:52:38.102278 12907 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:38.104547 12907 webserver.cc:460] Webserver started at http://127.11.42.65:33493/ using document root <none> and password file <none>
I20240502 01:52:38.105294 12907 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:38.105407 12907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:38.105718 12907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:38.108150 12907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "4f43c062aa1b4a2a8e616a6b37f1c632"
format_stamp: "Formatted at 2024-05-02 01:52:38 on dist-test-slave-xvk5"
I20240502 01:52:38.108870 12907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "4f43c062aa1b4a2a8e616a6b37f1c632"
format_stamp: "Formatted at 2024-05-02 01:52:38 on dist-test-slave-xvk5"
I20240502 01:52:38.113544 12907 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20240502 01:52:38.116626 12922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:38.117871 12907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:52:38.118021 12907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "4f43c062aa1b4a2a8e616a6b37f1c632"
format_stamp: "Formatted at 2024-05-02 01:52:38 on dist-test-slave-xvk5"
I20240502 01:52:38.118198 12907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:38.172070 12907 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:38.172910 12907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:38.173197 12907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:38.174114 12907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:38.175568 12907 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:38.175662 12907 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:38.175765 12907 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:38.175830 12907 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:38.205191 12907 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.65:38055
I20240502 01:52:38.205250 13034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.65:38055 every 8 connection(s)
I20240502 01:52:38.206562 12907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 01:52:38.213308 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 12907
I20240502 01:52:38.213508 11433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 01:52:38.216814 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.66:0
--local_ip_for_outbound_sockets=127.11.42.66
--webserver_interface=127.11.42.66
--webserver_port=0
--tserver_master_addrs=127.11.42.126:44219
--builtin_ntp_servers=127.11.42.84:37361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:52:38.219015 13035 heartbeater.cc:347] Connected to a master server at 127.11.42.126:44219
I20240502 01:52:38.219336 13035 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:38.220268 13035 heartbeater.cc:511] Master 127.11.42.126:44219 requested a full tablet report, sending...
I20240502 01:52:38.222272 12848 ts_manager.cc:194] Registered new tserver with Master: 4f43c062aa1b4a2a8e616a6b37f1c632 (127.11.42.65:38055)
I20240502 01:52:38.223811 12848 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.65:37497
W20240502 01:52:38.519299 13039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:38.519554 13039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:38.519662 13039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:38.527062 13039 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:38.527170 13039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:38.527330 13039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.66
I20240502 01:52:38.535845 13039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:37361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.11.42.66
--webserver_port=0
--tserver_master_addrs=127.11.42.126:44219
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.66
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:38.536993 13039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:38.538619 13039 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:38.549557 13045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:38.549852 13047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:38.550094 13044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:38.550436 13039 server_base.cc:1034] running on GCE node
I20240502 01:52:38.550945 13039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:38.551748 13039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:38.552958 13039 hybrid_clock.cc:648] HybridClock initialized: now 1714614758552919 us; error 43 us; skew 500 ppm
I20240502 01:52:38.553292 13039 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:38.555373 13039 webserver.cc:460] Webserver started at http://127.11.42.66:40111/ using document root <none> and password file <none>
I20240502 01:52:38.556021 13039 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:38.556133 13039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:38.556442 13039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:38.559469 13039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "66e949f21fb040ef96f2711b1a354678"
format_stamp: "Formatted at 2024-05-02 01:52:38 on dist-test-slave-xvk5"
I20240502 01:52:38.560329 13039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "66e949f21fb040ef96f2711b1a354678"
format_stamp: "Formatted at 2024-05-02 01:52:38 on dist-test-slave-xvk5"
I20240502 01:52:38.565110 13039 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 01:52:38.568194 13053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:38.569478 13039 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:52:38.569610 13039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "66e949f21fb040ef96f2711b1a354678"
format_stamp: "Formatted at 2024-05-02 01:52:38 on dist-test-slave-xvk5"
I20240502 01:52:38.569780 13039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:38.583587 13039 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:38.584301 13039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:38.584540 13039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:38.585412 13039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:38.586741 13039 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:38.586825 13039 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:38.586921 13039 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:38.586994 13039 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:38.619974 13039 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.66:45595
I20240502 01:52:38.620019 13166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.66:45595 every 8 connection(s)
I20240502 01:52:38.621220 13039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 01:52:38.621899 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 13039
I20240502 01:52:38.622093 11433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 01:52:38.625851 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.67:0
--local_ip_for_outbound_sockets=127.11.42.67
--webserver_interface=127.11.42.67
--webserver_port=0
--tserver_master_addrs=127.11.42.126:44219
--builtin_ntp_servers=127.11.42.84:37361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:52:38.641055 13167 heartbeater.cc:347] Connected to a master server at 127.11.42.126:44219
I20240502 01:52:38.641324 13167 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:38.642062 13167 heartbeater.cc:511] Master 127.11.42.126:44219 requested a full tablet report, sending...
I20240502 01:52:38.643371 12848 ts_manager.cc:194] Registered new tserver with Master: 66e949f21fb040ef96f2711b1a354678 (127.11.42.66:45595)
I20240502 01:52:38.644106 12848 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.66:35275
W20240502 01:52:38.936808 13170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:38.937098 13170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:38.937227 13170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:38.944298 13170 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:38.944412 13170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:38.944557 13170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.67
I20240502 01:52:38.953080 13170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:37361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.11.42.67
--webserver_port=0
--tserver_master_addrs=127.11.42.126:44219
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.67
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:38.954254 13170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:38.955884 13170 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:38.967267 13178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:38.967461 13176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:38.967701 13175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:38.968012 13170 server_base.cc:1034] running on GCE node
I20240502 01:52:38.968575 13170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:38.969491 13170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:38.970685 13170 hybrid_clock.cc:648] HybridClock initialized: now 1714614758970638 us; error 40 us; skew 500 ppm
I20240502 01:52:38.970976 13170 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:38.973073 13170 webserver.cc:460] Webserver started at http://127.11.42.67:37323/ using document root <none> and password file <none>
I20240502 01:52:38.973762 13170 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:38.973870 13170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:38.974175 13170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:38.976747 13170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "c75023bb74db470eaba10148838ad6d1"
format_stamp: "Formatted at 2024-05-02 01:52:38 on dist-test-slave-xvk5"
I20240502 01:52:38.977540 13170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "c75023bb74db470eaba10148838ad6d1"
format_stamp: "Formatted at 2024-05-02 01:52:38 on dist-test-slave-xvk5"
I20240502 01:52:38.982470 13170 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240502 01:52:38.985704 13184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:38.986873 13170 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20240502 01:52:38.987025 13170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "c75023bb74db470eaba10148838ad6d1"
format_stamp: "Formatted at 2024-05-02 01:52:38 on dist-test-slave-xvk5"
I20240502 01:52:38.987198 13170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:39.010175 13170 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:39.010946 13170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:39.011197 13170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:39.012077 13170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:39.013546 13170 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:39.013638 13170 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:39.013741 13170 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:39.013813 13170 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:39.042412 13170 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.67:39343
I20240502 01:52:39.042474 13298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.67:39343 every 8 connection(s)
I20240502 01:52:39.043605 13170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 01:52:39.044189 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 13170
I20240502 01:52:39.044358 11433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 01:52:39.048971 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.68:0
--local_ip_for_outbound_sockets=127.11.42.68
--webserver_interface=127.11.42.68
--webserver_port=0
--tserver_master_addrs=127.11.42.126:44219
--builtin_ntp_servers=127.11.42.84:37361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:52:39.063482 13299 heartbeater.cc:347] Connected to a master server at 127.11.42.126:44219
I20240502 01:52:39.063818 13299 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:39.064827 13299 heartbeater.cc:511] Master 127.11.42.126:44219 requested a full tablet report, sending...
I20240502 01:52:39.066464 12848 ts_manager.cc:194] Registered new tserver with Master: c75023bb74db470eaba10148838ad6d1 (127.11.42.67:39343)
I20240502 01:52:39.067591 12848 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.67:51829
I20240502 01:52:39.226383 13035 heartbeater.cc:503] Master 127.11.42.126:44219 was elected leader, sending a full tablet report...
W20240502 01:52:39.359587 13302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:39.359833 13302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:39.359956 13302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:39.367163 13302 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:39.367296 13302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:39.367470 13302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.68
I20240502 01:52:39.376246 13302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:37361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.11.42.68
--webserver_port=0
--tserver_master_addrs=127.11.42.126:44219
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.68
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:39.377509 13302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:39.379314 13302 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:39.390731 13308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:39.390913 13307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:39.391211 13310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:39.391530 13302 server_base.cc:1034] running on GCE node
I20240502 01:52:39.392112 13302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:39.393035 13302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:39.394235 13302 hybrid_clock.cc:648] HybridClock initialized: now 1714614759394190 us; error 38 us; skew 500 ppm
I20240502 01:52:39.394537 13302 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:39.396766 13302 webserver.cc:460] Webserver started at http://127.11.42.68:34773/ using document root <none> and password file <none>
I20240502 01:52:39.397500 13302 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:39.397609 13302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:39.397917 13302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:39.400352 13302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "f53b92d52f2f4f55b826583550436916"
format_stamp: "Formatted at 2024-05-02 01:52:39 on dist-test-slave-xvk5"
I20240502 01:52:39.401105 13302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "f53b92d52f2f4f55b826583550436916"
format_stamp: "Formatted at 2024-05-02 01:52:39 on dist-test-slave-xvk5"
I20240502 01:52:39.405918 13302 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 01:52:39.409162 13316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:39.410362 13302 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:52:39.410498 13302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "f53b92d52f2f4f55b826583550436916"
format_stamp: "Formatted at 2024-05-02 01:52:39 on dist-test-slave-xvk5"
I20240502 01:52:39.410674 13302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:39.424015 13302 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:39.424753 13302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:39.424988 13302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:39.425886 13302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:39.427258 13302 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:39.427351 13302 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:39.427453 13302 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:39.427533 13302 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:39.455997 13302 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.68:39247
I20240502 01:52:39.456046 13430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.68:39247 every 8 connection(s)
I20240502 01:52:39.457260 13302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 01:52:39.457494 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 13302
I20240502 01:52:39.457664 11433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 01:52:39.461050 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.69:0
--local_ip_for_outbound_sockets=127.11.42.69
--webserver_interface=127.11.42.69
--webserver_port=0
--tserver_master_addrs=127.11.42.126:44219
--builtin_ntp_servers=127.11.42.84:37361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:52:39.470647 13431 heartbeater.cc:347] Connected to a master server at 127.11.42.126:44219
I20240502 01:52:39.470932 13431 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:39.471700 13431 heartbeater.cc:511] Master 127.11.42.126:44219 requested a full tablet report, sending...
I20240502 01:52:39.473126 12848 ts_manager.cc:194] Registered new tserver with Master: f53b92d52f2f4f55b826583550436916 (127.11.42.68:39247)
I20240502 01:52:39.473945 12848 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.68:46339
I20240502 01:52:39.646405 13167 heartbeater.cc:503] Master 127.11.42.126:44219 was elected leader, sending a full tablet report...
W20240502 01:52:39.783056 13434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:39.783360 13434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:39.783491 13434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:39.791246 13434 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:39.791369 13434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:39.791530 13434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.69
I20240502 01:52:39.800590 13434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:37361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.11.42.69
--webserver_port=0
--tserver_master_addrs=127.11.42.126:44219
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.69
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:39.801940 13434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:39.803687 13434 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:39.815615 13440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:39.815279 13439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:39.816206 13434 server_base.cc:1034] running on GCE node
W20240502 01:52:39.816306 13442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:39.816880 13434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:39.817795 13434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:39.818993 13434 hybrid_clock.cc:648] HybridClock initialized: now 1714614759818950 us; error 38 us; skew 500 ppm
I20240502 01:52:39.819307 13434 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:39.821519 13434 webserver.cc:460] Webserver started at http://127.11.42.69:46015/ using document root <none> and password file <none>
I20240502 01:52:39.822183 13434 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:39.822301 13434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:39.822616 13434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:39.826006 13434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "9d3594a9591a4084afa7ad0f940d1d1a"
format_stamp: "Formatted at 2024-05-02 01:52:39 on dist-test-slave-xvk5"
I20240502 01:52:39.826694 13434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "9d3594a9591a4084afa7ad0f940d1d1a"
format_stamp: "Formatted at 2024-05-02 01:52:39 on dist-test-slave-xvk5"
I20240502 01:52:39.831514 13434 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 01:52:39.834949 13448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:39.836127 13434 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 01:52:39.836345 13434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "9d3594a9591a4084afa7ad0f940d1d1a"
format_stamp: "Formatted at 2024-05-02 01:52:39 on dist-test-slave-xvk5"
I20240502 01:52:39.836558 13434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:39.858526 13434 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:39.859359 13434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:39.859623 13434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:39.860528 13434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:39.862108 13434 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:39.862202 13434 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:39.862310 13434 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:39.862376 13434 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:39.891824 13434 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.69:44195
I20240502 01:52:39.891918 13561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.69:44195 every 8 connection(s)
I20240502 01:52:39.893126 13434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:52:39.899154 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 13434
I20240502 01:52:39.899324 11433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 01:52:39.905476 13562 heartbeater.cc:347] Connected to a master server at 127.11.42.126:44219
I20240502 01:52:39.905753 13562 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:39.906570 13562 heartbeater.cc:511] Master 127.11.42.126:44219 requested a full tablet report, sending...
I20240502 01:52:39.907899 12848 ts_manager.cc:194] Registered new tserver with Master: 9d3594a9591a4084afa7ad0f940d1d1a (127.11.42.69:44195)
I20240502 01:52:39.908663 12848 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.69:42831
I20240502 01:52:39.914583 11433 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 01:52:39.930308 12848 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44008:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 01:52:39.992605 13496 tablet_service.cc:1467] Processing CreateTablet for tablet a5784eb4a6894fca82befc918e6e3112 (DEFAULT_TABLE table=TestTable [id=2405749e08794028b97ca25131aa8388]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:39.992853 13232 tablet_service.cc:1467] Processing CreateTablet for tablet a5784eb4a6894fca82befc918e6e3112 (DEFAULT_TABLE table=TestTable [id=2405749e08794028b97ca25131aa8388]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:39.994251 13232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5784eb4a6894fca82befc918e6e3112. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:39.994251 13496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5784eb4a6894fca82befc918e6e3112. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:39.996150 13102 tablet_service.cc:1467] Processing CreateTablet for tablet a5784eb4a6894fca82befc918e6e3112 (DEFAULT_TABLE table=TestTable [id=2405749e08794028b97ca25131aa8388]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:39.997471 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5784eb4a6894fca82befc918e6e3112. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:40.010499 13582 tablet_bootstrap.cc:492] T a5784eb4a6894fca82befc918e6e3112 P 9d3594a9591a4084afa7ad0f940d1d1a: Bootstrap starting.
I20240502 01:52:40.010823 13580 tablet_bootstrap.cc:492] T a5784eb4a6894fca82befc918e6e3112 P 66e949f21fb040ef96f2711b1a354678: Bootstrap starting.
I20240502 01:52:40.013729 13581 tablet_bootstrap.cc:492] T a5784eb4a6894fca82befc918e6e3112 P c75023bb74db470eaba10148838ad6d1: Bootstrap starting.
I20240502 01:52:40.013800 13580 tablet_bootstrap.cc:654] T a5784eb4a6894fca82befc918e6e3112 P 66e949f21fb040ef96f2711b1a354678: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:40.014876 13580 log.cc:824] T a5784eb4a6894fca82befc918e6e3112 P 66e949f21fb040ef96f2711b1a354678: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:40.015221 13582 tablet_bootstrap.cc:654] T a5784eb4a6894fca82befc918e6e3112 P 9d3594a9591a4084afa7ad0f940d1d1a: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:40.016464 13582 log.cc:824] T a5784eb4a6894fca82befc918e6e3112 P 9d3594a9591a4084afa7ad0f940d1d1a: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:40.016979 13581 tablet_bootstrap.cc:654] T a5784eb4a6894fca82befc918e6e3112 P c75023bb74db470eaba10148838ad6d1: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:40.018131 13581 log.cc:824] T a5784eb4a6894fca82befc918e6e3112 P c75023bb74db470eaba10148838ad6d1: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:40.019376 13580 tablet_bootstrap.cc:492] T a5784eb4a6894fca82befc918e6e3112 P 66e949f21fb040ef96f2711b1a354678: No bootstrap required, opened a new log
I20240502 01:52:40.019392 13582 tablet_bootstrap.cc:492] T a5784eb4a6894fca82befc918e6e3112 P 9d3594a9591a4084afa7ad0f940d1d1a: No bootstrap required, opened a new log
I20240502 01:52:40.019637 13582 ts_tablet_manager.cc:1367] T a5784eb4a6894fca82befc918e6e3112 P 9d3594a9591a4084afa7ad0f940d1d1a: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20240502 01:52:40.019637 13580 ts_tablet_manager.cc:1367] T a5784eb4a6894fca82befc918e6e3112 P 66e949f21fb040ef96f2711b1a354678: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20240502 01:52:40.020634 13581 tablet_bootstrap.cc:492] T a5784eb4a6894fca82befc918e6e3112 P c75023bb74db470eaba10148838ad6d1: No bootstrap required, opened a new log
I20240502 01:52:40.020890 13581 ts_tablet_manager.cc:1367] T a5784eb4a6894fca82befc918e6e3112 P c75023bb74db470eaba10148838ad6d1: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240502 01:52:40.026830 13581 raft_consensus.cc:348] T a5784eb4a6894fca82befc918e6e3112 P c75023bb74db470eaba10148838ad6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e949f21fb040ef96f2711b1a354678" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 45595 } } peers { permanent_uuid: "9d3594a9591a4084afa7ad0f940d1d1a" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 44195 } } peers { permanent_uuid: "c75023bb74db470eaba10148838ad6d1" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 39343 } }
I20240502 01:52:40.027150 13581 raft_consensus.cc:374] T a5784eb4a6894fca82befc918e6e3112 P c75023bb74db470eaba10148838ad6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:40.027273 13581 raft_consensus.cc:729] T a5784eb4a6894fca82befc918e6e3112 P c75023bb74db470eaba10148838ad6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c75023bb74db470eaba10148838ad6d1, State: Initialized, Role: FOLLOWER
I20240502 01:52:40.027894 13581 consensus_queue.cc:260] T a5784eb4a6894fca82befc918e6e3112 P c75023bb74db470eaba10148838ad6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e949f21fb040ef96f2711b1a354678" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 45595 } } peers { permanent_uuid: "9d3594a9591a4084afa7ad0f940d1d1a" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 44195 } } peers { permanent_uuid: "c75023bb74db470eaba10148838ad6d1" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 39343 } }
I20240502 01:52:40.028200 13582 raft_consensus.cc:348] T a5784eb4a6894fca82befc918e6e3112 P 9d3594a9591a4084afa7ad0f940d1d1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e949f21fb040ef96f2711b1a354678" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 45595 } } peers { permanent_uuid: "9d3594a9591a4084afa7ad0f940d1d1a" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 44195 } } peers { permanent_uuid: "c75023bb74db470eaba10148838ad6d1" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 39343 } }
I20240502 01:52:40.028199 13580 raft_consensus.cc:348] T a5784eb4a6894fca82befc918e6e3112 P 66e949f21fb040ef96f2711b1a354678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e949f21fb040ef96f2711b1a354678" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 45595 } } peers { permanent_uuid: "9d3594a9591a4084afa7ad0f940d1d1a" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 44195 } } peers { permanent_uuid: "c75023bb74db470eaba10148838ad6d1" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 39343 } }
I20240502 01:52:40.028587 13582 raft_consensus.cc:374] T a5784eb4a6894fca82befc918e6e3112 P 9d3594a9591a4084afa7ad0f940d1d1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:40.028589 13580 raft_consensus.cc:374] T a5784eb4a6894fca82befc918e6e3112 P 66e949f21fb040ef96f2711b1a354678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:40.028695 13580 raft_consensus.cc:729] T a5784eb4a6894fca82befc918e6e3112 P 66e949f21fb040ef96f2711b1a354678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e949f21fb040ef96f2711b1a354678, State: Initialized, Role: FOLLOWER
I20240502 01:52:40.028690 13582 raft_consensus.cc:729] T a5784eb4a6894fca82befc918e6e3112 P 9d3594a9591a4084afa7ad0f940d1d1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d3594a9591a4084afa7ad0f940d1d1a, State: Initialized, Role: FOLLOWER
I20240502 01:52:40.029268 13299 heartbeater.cc:503] Master 127.11.42.126:44219 was elected leader, sending a full tablet report...
I20240502 01:52:40.029364 13582 consensus_queue.cc:260] T a5784eb4a6894fca82befc918e6e3112 P 9d3594a9591a4084afa7ad0f940d1d1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e949f21fb040ef96f2711b1a354678" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 45595 } } peers { permanent_uuid: "9d3594a9591a4084afa7ad0f940d1d1a" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 44195 } } peers { permanent_uuid: "c75023bb74db470eaba10148838ad6d1" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 39343 } }
I20240502 01:52:40.029338 13580 consensus_queue.cc:260] T a5784eb4a6894fca82befc918e6e3112 P 66e949f21fb040ef96f2711b1a354678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e949f21fb040ef96f2711b1a354678" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 45595 } } peers { permanent_uuid: "9d3594a9591a4084afa7ad0f940d1d1a" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 44195 } } peers { permanent_uuid: "c75023bb74db470eaba10148838ad6d1" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 39343 } }
I20240502 01:52:40.029527 13581 ts_tablet_manager.cc:1398] T a5784eb4a6894fca82befc918e6e3112 P c75023bb74db470eaba10148838ad6d1: Time spent starting tablet: real 0.009s user 0.009s sys 0.001s
I20240502 01:52:40.031991 13582 ts_tablet_manager.cc:1398] T a5784eb4a6894fca82befc918e6e3112 P 9d3594a9591a4084afa7ad0f940d1d1a: Time spent starting tablet: real 0.012s user 0.010s sys 0.002s
I20240502 01:52:40.032382 13562 heartbeater.cc:503] Master 127.11.42.126:44219 was elected leader, sending a full tablet report...
I20240502 01:52:40.032435 13580 ts_tablet_manager.cc:1398] T a5784eb4a6894fca82befc918e6e3112 P 66e949f21fb040ef96f2711b1a354678: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
W20240502 01:52:40.054351 13300 tablet.cc:2369] T a5784eb4a6894fca82befc918e6e3112 P c75023bb74db470eaba10148838ad6d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:52:40.129695 13168 tablet.cc:2369] T a5784eb4a6894fca82befc918e6e3112 P 66e949f21fb040ef96f2711b1a354678: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:52:40.130335 13587 raft_consensus.cc:482] T a5784eb4a6894fca82befc918e6e3112 P 66e949f21fb040ef96f2711b1a354678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:40.130564 13587 raft_consensus.cc:504] T a5784eb4a6894fca82befc918e6e3112 P 66e949f21fb040ef96f2711b1a354678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e949f21fb040ef96f2711b1a354678" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 45595 } } peers { permanent_uuid: "9d3594a9591a4084afa7ad0f940d1d1a" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 44195 } } peers { permanent_uuid: "c75023bb74db470eaba10148838ad6d1" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 39343 } }
I20240502 01:52:40.132158 13587 leader_election.cc:288] T a5784eb4a6894fca82befc918e6e3112 P 66e949f21fb040ef96f2711b1a354678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d3594a9591a4084afa7ad0f940d1d1a (127.11.42.69:44195), c75023bb74db470eaba10148838ad6d1 (127.11.42.67:39343)
I20240502 01:52:40.139227 13253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5784eb4a6894fca82befc918e6e3112" candidate_uuid: "66e949f21fb040ef96f2711b1a354678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c75023bb74db470eaba10148838ad6d1" is_pre_election: true
I20240502 01:52:40.139232 13517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5784eb4a6894fca82befc918e6e3112" candidate_uuid: "66e949f21fb040ef96f2711b1a354678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d3594a9591a4084afa7ad0f940d1d1a" is_pre_election: true
I20240502 01:52:40.139642 13253 raft_consensus.cc:2454] T a5784eb4a6894fca82befc918e6e3112 P c75023bb74db470eaba10148838ad6d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e949f21fb040ef96f2711b1a354678 in term 0.
I20240502 01:52:40.139670 13517 raft_consensus.cc:2454] T a5784eb4a6894fca82befc918e6e3112 P 9d3594a9591a4084afa7ad0f940d1d1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e949f21fb040ef96f2711b1a354678 in term 0.
I20240502 01:52:40.140203 13058 leader_election.cc:302] T a5784eb4a6894fca82befc918e6e3112 P 66e949f21fb040ef96f2711b1a354678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e949f21fb040ef96f2711b1a354678, c75023bb74db470eaba10148838ad6d1; no voters:
I20240502 01:52:40.140566 13587 raft_consensus.cc:2781] T a5784eb4a6894fca82befc918e6e3112 P 66e949f21fb040ef96f2711b1a354678 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:40.140697 13587 raft_consensus.cc:482] T a5784eb4a6894fca82befc918e6e3112 P 66e949f21fb040ef96f2711b1a354678 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:40.141180 13587 raft_consensus.cc:3037] T a5784eb4a6894fca82befc918e6e3112 P 66e949f21fb040ef96f2711b1a354678 [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:52:40.144616 13563 tablet.cc:2369] T a5784eb4a6894fca82befc918e6e3112 P 9d3594a9591a4084afa7ad0f940d1d1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:52:40.150521 13587 raft_consensus.cc:504] T a5784eb4a6894fca82befc918e6e3112 P 66e949f21fb040ef96f2711b1a354678 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e949f21fb040ef96f2711b1a354678" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 45595 } } peers { permanent_uuid: "9d3594a9591a4084afa7ad0f940d1d1a" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 44195 } } peers { permanent_uuid: "c75023bb74db470eaba10148838ad6d1" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 39343 } }
I20240502 01:52:40.151103 13587 leader_election.cc:288] T a5784eb4a6894fca82befc918e6e3112 P 66e949f21fb040ef96f2711b1a354678 [CANDIDATE]: Term 1 election: Requested vote from peers 9d3594a9591a4084afa7ad0f940d1d1a (127.11.42.69:44195), c75023bb74db470eaba10148838ad6d1 (127.11.42.67:39343)
I20240502 01:52:40.151549 13517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5784eb4a6894fca82befc918e6e3112" candidate_uuid: "66e949f21fb040ef96f2711b1a354678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d3594a9591a4084afa7ad0f940d1d1a"
I20240502 01:52:40.151616 13253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5784eb4a6894fca82befc918e6e3112" candidate_uuid: "66e949f21fb040ef96f2711b1a354678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c75023bb74db470eaba10148838ad6d1"
I20240502 01:52:40.151721 13517 raft_consensus.cc:3037] T a5784eb4a6894fca82befc918e6e3112 P 9d3594a9591a4084afa7ad0f940d1d1a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:40.151811 13253 raft_consensus.cc:3037] T a5784eb4a6894fca82befc918e6e3112 P c75023bb74db470eaba10148838ad6d1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:40.153915 13517 raft_consensus.cc:2454] T a5784eb4a6894fca82befc918e6e3112 P 9d3594a9591a4084afa7ad0f940d1d1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e949f21fb040ef96f2711b1a354678 in term 1.
I20240502 01:52:40.154320 13055 leader_election.cc:302] T a5784eb4a6894fca82befc918e6e3112 P 66e949f21fb040ef96f2711b1a354678 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e949f21fb040ef96f2711b1a354678, 9d3594a9591a4084afa7ad0f940d1d1a; no voters:
I20240502 01:52:40.154573 13587 raft_consensus.cc:2781] T a5784eb4a6894fca82befc918e6e3112 P 66e949f21fb040ef96f2711b1a354678 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:40.154743 13253 raft_consensus.cc:2454] T a5784eb4a6894fca82befc918e6e3112 P c75023bb74db470eaba10148838ad6d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e949f21fb040ef96f2711b1a354678 in term 1.
I20240502 01:52:40.155098 13587 raft_consensus.cc:686] T a5784eb4a6894fca82befc918e6e3112 P 66e949f21fb040ef96f2711b1a354678 [term 1 LEADER]: Becoming Leader. State: Replica: 66e949f21fb040ef96f2711b1a354678, State: Running, Role: LEADER
I20240502 01:52:40.155577 13587 consensus_queue.cc:237] T a5784eb4a6894fca82befc918e6e3112 P 66e949f21fb040ef96f2711b1a354678 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e949f21fb040ef96f2711b1a354678" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 45595 } } peers { permanent_uuid: "9d3594a9591a4084afa7ad0f940d1d1a" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 44195 } } peers { permanent_uuid: "c75023bb74db470eaba10148838ad6d1" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 39343 } }
I20240502 01:52:40.160502 12847 catalog_manager.cc:5484] T a5784eb4a6894fca82befc918e6e3112 P 66e949f21fb040ef96f2711b1a354678 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66e949f21fb040ef96f2711b1a354678 (127.11.42.66). New cstate: current_term: 1 leader_uuid: "66e949f21fb040ef96f2711b1a354678" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e949f21fb040ef96f2711b1a354678" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 45595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d3594a9591a4084afa7ad0f940d1d1a" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 44195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c75023bb74db470eaba10148838ad6d1" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 39343 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:40.212884 11433 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 01:52:40.215092 11433 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 4f43c062aa1b4a2a8e616a6b37f1c632 to finish bootstrapping
I20240502 01:52:40.224421 11433 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 66e949f21fb040ef96f2711b1a354678 to finish bootstrapping
I20240502 01:52:40.231688 11433 ts_itest-base.cc:233] Waiting for 1 tablets on tserver c75023bb74db470eaba10148838ad6d1 to finish bootstrapping
I20240502 01:52:40.238678 11433 ts_itest-base.cc:233] Waiting for 0 tablets on tserver f53b92d52f2f4f55b826583550436916 to finish bootstrapping
I20240502 01:52:40.246703 11433 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 9d3594a9591a4084afa7ad0f940d1d1a to finish bootstrapping
I20240502 01:52:40.253787 11433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu pid 12907
I20240502 01:52:40.254703 13030 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:40.335433 11433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu with pid 12907
I20240502 01:52:40.476274 13431 heartbeater.cc:503] Master 127.11.42.126:44219 was elected leader, sending a full tablet report...
I20240502 01:52:40.536945 13587 consensus_queue.cc:1035] T a5784eb4a6894fca82befc918e6e3112 P 66e949f21fb040ef96f2711b1a354678 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75023bb74db470eaba10148838ad6d1" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 39343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:40.548204 13587 consensus_queue.cc:1035] T a5784eb4a6894fca82befc918e6e3112 P 66e949f21fb040ef96f2711b1a354678 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d3594a9591a4084afa7ad0f940d1d1a" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 44195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:40.799784 13496 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:40.802325 13365 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:40.803481 13232 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:40.804646 13102 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:52:40.930371 13232 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:40.931210 13496 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:40.938928 13365 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:40.939047 13102 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:52:41.094761 13102 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:52:41.094801 13496 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:41.097523 13232 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:41.105505 13365 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:41.702477 13102 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:52:41.702477 13496 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:41.710333 13365 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:41.713141 13232 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:41.833573 13102 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:52:41.837850 13232 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:41.840601 13365 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:41.841274 13496 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:42.000815 13102 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:52:42.003095 13496 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:42.006929 13232 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:42.010233 13365 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:42.154330 11433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu pid 13039
I20240502 01:52:42.155846 13162 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:42.252527 11433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu with pid 13039
I20240502 01:52:42.267889 11433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu pid 13170
I20240502 01:52:42.268931 13293 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:42.366128 11433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu with pid 13170
I20240502 01:52:42.381475 11433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu pid 13302
I20240502 01:52:42.382488 13425 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:42.472605 11433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu with pid 13302
I20240502 01:52:42.488739 11433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu pid 13434
I20240502 01:52:42.489777 13556 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:42.587095 11433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu with pid 13434
W20240502 01:52:42.603312 12835 connection.cc:535] server connection from 127.11.42.69:42831 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:52:42.603559 11433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu pid 12817
I20240502 01:52:42.604648 12878 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:42.724658 11433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu with pid 12817
2024-05-02T01:52:42Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.TabletServerIsDown/1 (5586 ms)
[----------] 1 test from RebalanceStartCriteriaTest (5586 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/1
I20240502 01:52:42.764034 11433 test_util.cc:268] Using random seed: -606308631
I20240502 01:52:42.765856 11433 ts_itest-base.cc:112] Starting cluster with:
I20240502 01:52:42.765919 11433 ts_itest-base.cc:113] --------------
I20240502 01:52:42.765977 11433 ts_itest-base.cc:114] 7 tablet servers
I20240502 01:52:42.766021 11433 ts_itest-base.cc:115] 1 replicas per TS
I20240502 01:52:42.766085 11433 ts_itest-base.cc:116] --------------
2024-05-02T01:52:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T01:52:42Z Disabled control of system clock
I20240502 01:52:42.788106 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.42.126:40815
--webserver_interface=127.11.42.126
--webserver_port=0
--builtin_ntp_servers=127.11.42.84:45633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.42.126:40815
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240502 01:52:43.099512 13804 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 01:52:43.099855 13804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:43.099952 13804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:43.100054 13804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:43.107266 13804 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:43.107380 13804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 01:52:43.107473 13804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:43.107533 13804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 01:52:43.107617 13804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 01:52:43.116621 13804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:45633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.11.42.126:40815
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.42.126:40815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.11.42.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:43.117942 13804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:43.119642 13804 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:43.130061 13809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:43.130117 13810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:43.130595 13812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:43.131064 13804 server_base.cc:1034] running on GCE node
I20240502 01:52:43.131783 13804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:43.133193 13804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:43.134418 13804 hybrid_clock.cc:648] HybridClock initialized: now 1714614763134351 us; error 38 us; skew 500 ppm
I20240502 01:52:43.134728 13804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:43.143208 13804 webserver.cc:460] Webserver started at http://127.11.42.126:35561/ using document root <none> and password file <none>
I20240502 01:52:43.143972 13804 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:43.144099 13804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:43.144425 13804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:43.147086 13804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "d00df36391f54eaf96213f38c5e602a1"
format_stamp: "Formatted at 2024-05-02 01:52:43 on dist-test-slave-xvk5"
I20240502 01:52:43.147817 13804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "d00df36391f54eaf96213f38c5e602a1"
format_stamp: "Formatted at 2024-05-02 01:52:43 on dist-test-slave-xvk5"
I20240502 01:52:43.152698 13804 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240502 01:52:43.155843 13818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:43.157086 13804 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20240502 01:52:43.157253 13804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d00df36391f54eaf96213f38c5e602a1"
format_stamp: "Formatted at 2024-05-02 01:52:43 on dist-test-slave-xvk5"
I20240502 01:52:43.157456 13804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:43.182447 13804 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:43.183209 13804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:43.183495 13804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:43.203864 13804 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.126:40815
I20240502 01:52:43.203897 13870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.126:40815 every 8 connection(s)
I20240502 01:52:43.205206 13804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 01:52:43.208870 13871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:43.213735 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 13804
I20240502 01:52:43.213956 11433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 01:52:43.218701 13871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d00df36391f54eaf96213f38c5e602a1: Bootstrap starting.
I20240502 01:52:43.221884 13871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d00df36391f54eaf96213f38c5e602a1: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:43.223066 13871 log.cc:824] T 00000000000000000000000000000000 P d00df36391f54eaf96213f38c5e602a1: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:43.225937 13871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d00df36391f54eaf96213f38c5e602a1: No bootstrap required, opened a new log
I20240502 01:52:43.232035 13871 raft_consensus.cc:348] T 00000000000000000000000000000000 P d00df36391f54eaf96213f38c5e602a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00df36391f54eaf96213f38c5e602a1" member_type: VOTER last_known_addr { host: "127.11.42.126" port: 40815 } }
I20240502 01:52:43.232319 13871 raft_consensus.cc:374] T 00000000000000000000000000000000 P d00df36391f54eaf96213f38c5e602a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:43.232393 13871 raft_consensus.cc:729] T 00000000000000000000000000000000 P d00df36391f54eaf96213f38c5e602a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d00df36391f54eaf96213f38c5e602a1, State: Initialized, Role: FOLLOWER
I20240502 01:52:43.232923 13871 consensus_queue.cc:260] T 00000000000000000000000000000000 P d00df36391f54eaf96213f38c5e602a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00df36391f54eaf96213f38c5e602a1" member_type: VOTER last_known_addr { host: "127.11.42.126" port: 40815 } }
I20240502 01:52:43.233135 13871 raft_consensus.cc:388] T 00000000000000000000000000000000 P d00df36391f54eaf96213f38c5e602a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:43.233232 13871 raft_consensus.cc:482] T 00000000000000000000000000000000 P d00df36391f54eaf96213f38c5e602a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:43.233351 13871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d00df36391f54eaf96213f38c5e602a1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:43.296842 13871 raft_consensus.cc:504] T 00000000000000000000000000000000 P d00df36391f54eaf96213f38c5e602a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00df36391f54eaf96213f38c5e602a1" member_type: VOTER last_known_addr { host: "127.11.42.126" port: 40815 } }
I20240502 01:52:43.297614 13871 leader_election.cc:302] T 00000000000000000000000000000000 P d00df36391f54eaf96213f38c5e602a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d00df36391f54eaf96213f38c5e602a1; no voters:
I20240502 01:52:43.298321 13871 leader_election.cc:288] T 00000000000000000000000000000000 P d00df36391f54eaf96213f38c5e602a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:43.298476 13876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d00df36391f54eaf96213f38c5e602a1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:43.301100 13876 raft_consensus.cc:686] T 00000000000000000000000000000000 P d00df36391f54eaf96213f38c5e602a1 [term 1 LEADER]: Becoming Leader. State: Replica: d00df36391f54eaf96213f38c5e602a1, State: Running, Role: LEADER
I20240502 01:52:43.301689 13876 consensus_queue.cc:237] T 00000000000000000000000000000000 P d00df36391f54eaf96213f38c5e602a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00df36391f54eaf96213f38c5e602a1" member_type: VOTER last_known_addr { host: "127.11.42.126" port: 40815 } }
I20240502 01:52:43.302165 13871 sys_catalog.cc:564] T 00000000000000000000000000000000 P d00df36391f54eaf96213f38c5e602a1 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:52:43.304677 13877 sys_catalog.cc:455] T 00000000000000000000000000000000 P d00df36391f54eaf96213f38c5e602a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d00df36391f54eaf96213f38c5e602a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00df36391f54eaf96213f38c5e602a1" member_type: VOTER last_known_addr { host: "127.11.42.126" port: 40815 } } }
I20240502 01:52:43.304742 13878 sys_catalog.cc:455] T 00000000000000000000000000000000 P d00df36391f54eaf96213f38c5e602a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d00df36391f54eaf96213f38c5e602a1. Latest consensus state: current_term: 1 leader_uuid: "d00df36391f54eaf96213f38c5e602a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00df36391f54eaf96213f38c5e602a1" member_type: VOTER last_known_addr { host: "127.11.42.126" port: 40815 } } }
I20240502 01:52:43.304999 13877 sys_catalog.cc:458] T 00000000000000000000000000000000 P d00df36391f54eaf96213f38c5e602a1 [sys.catalog]: This master's current role is: LEADER
I20240502 01:52:43.305058 13878 sys_catalog.cc:458] T 00000000000000000000000000000000 P d00df36391f54eaf96213f38c5e602a1 [sys.catalog]: This master's current role is: LEADER
I20240502 01:52:43.309813 13884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:52:43.314294 13884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:52:43.322539 13884 catalog_manager.cc:1347] Generated new cluster ID: 55af5da90696408ab3c6fac45783004a
I20240502 01:52:43.322638 13884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:52:43.340847 13884 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:52:43.342003 13884 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:52:43.351398 13884 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d00df36391f54eaf96213f38c5e602a1: Generated new TSK 0
I20240502 01:52:43.352077 13884 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:52:43.370363 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.65:0
--local_ip_for_outbound_sockets=127.11.42.65
--webserver_interface=127.11.42.65
--webserver_port=0
--tserver_master_addrs=127.11.42.126:40815
--builtin_ntp_servers=127.11.42.84:45633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 01:52:43.698005 13895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:43.698320 13895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:43.698448 13895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:43.706504 13895 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:43.706627 13895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:43.706777 13895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.65
I20240502 01:52:43.715742 13895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:45633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.11.42.65
--webserver_port=0
--tserver_master_addrs=127.11.42.126:40815
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.65
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:43.716998 13895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:43.719416 13895 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:43.731429 13900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:43.731971 13903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:43.732365 13901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:43.732879 13895 server_base.cc:1034] running on GCE node
I20240502 01:52:43.733506 13895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:43.734447 13895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:43.735646 13895 hybrid_clock.cc:648] HybridClock initialized: now 1714614763735594 us; error 41 us; skew 500 ppm
I20240502 01:52:43.735953 13895 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:43.738060 13895 webserver.cc:460] Webserver started at http://127.11.42.65:33553/ using document root <none> and password file <none>
I20240502 01:52:43.738768 13895 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:43.738880 13895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:43.739172 13895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:43.741755 13895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "d43426b5bc874e8482e7b3205a9ba970"
format_stamp: "Formatted at 2024-05-02 01:52:43 on dist-test-slave-xvk5"
I20240502 01:52:43.742487 13895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "d43426b5bc874e8482e7b3205a9ba970"
format_stamp: "Formatted at 2024-05-02 01:52:43 on dist-test-slave-xvk5"
I20240502 01:52:43.747401 13895 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 01:52:43.750622 13909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:43.751859 13895 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240502 01:52:43.752015 13895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "d43426b5bc874e8482e7b3205a9ba970"
format_stamp: "Formatted at 2024-05-02 01:52:43 on dist-test-slave-xvk5"
I20240502 01:52:43.752189 13895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:43.776409 13895 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:43.777222 13895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:43.777469 13895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:43.778398 13895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:43.779855 13895 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:43.779940 13895 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:43.780045 13895 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:43.780118 13895 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:43.810300 13895 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.65:40383
I20240502 01:52:43.810381 14023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.65:40383 every 8 connection(s)
I20240502 01:52:43.811576 13895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 01:52:43.816524 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 13895
I20240502 01:52:43.816751 11433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 01:52:43.820161 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.66:0
--local_ip_for_outbound_sockets=127.11.42.66
--webserver_interface=127.11.42.66
--webserver_port=0
--tserver_master_addrs=127.11.42.126:40815
--builtin_ntp_servers=127.11.42.84:45633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:52:43.824120 14024 heartbeater.cc:347] Connected to a master server at 127.11.42.126:40815
I20240502 01:52:43.824431 14024 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:43.825397 14024 heartbeater.cc:511] Master 127.11.42.126:40815 requested a full tablet report, sending...
I20240502 01:52:43.827399 13836 ts_manager.cc:194] Registered new tserver with Master: d43426b5bc874e8482e7b3205a9ba970 (127.11.42.65:40383)
I20240502 01:52:43.828805 13836 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.65:41769
W20240502 01:52:44.121646 14028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:44.121912 14028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:44.122037 14028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:44.129209 14028 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:44.129305 14028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:44.129451 14028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.66
I20240502 01:52:44.137881 14028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:45633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.11.42.66
--webserver_port=0
--tserver_master_addrs=127.11.42.126:40815
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.66
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:44.139009 14028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:44.140610 14028 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:44.151654 14033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:44.152119 14036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:44.152483 14034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:44.152837 14028 server_base.cc:1034] running on GCE node
I20240502 01:52:44.153393 14028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:44.154208 14028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:44.155391 14028 hybrid_clock.cc:648] HybridClock initialized: now 1714614764155349 us; error 46 us; skew 500 ppm
I20240502 01:52:44.155658 14028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:44.157722 14028 webserver.cc:460] Webserver started at http://127.11.42.66:40309/ using document root <none> and password file <none>
I20240502 01:52:44.158371 14028 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:44.158476 14028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:44.158763 14028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:44.161116 14028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "1478241a03934d029dc7030480423cab"
format_stamp: "Formatted at 2024-05-02 01:52:44 on dist-test-slave-xvk5"
I20240502 01:52:44.161859 14028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "1478241a03934d029dc7030480423cab"
format_stamp: "Formatted at 2024-05-02 01:52:44 on dist-test-slave-xvk5"
I20240502 01:52:44.166505 14028 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20240502 01:52:44.169503 14042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:44.170655 14028 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20240502 01:52:44.170792 14028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "1478241a03934d029dc7030480423cab"
format_stamp: "Formatted at 2024-05-02 01:52:44 on dist-test-slave-xvk5"
I20240502 01:52:44.170965 14028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:44.186282 14028 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:44.186962 14028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:44.187218 14028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:44.188055 14028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:44.189409 14028 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:44.189500 14028 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:44.189599 14028 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:44.189663 14028 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:44.223239 14028 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.66:34613
I20240502 01:52:44.223294 14155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.66:34613 every 8 connection(s)
I20240502 01:52:44.224386 14028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 01:52:44.225170 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 14028
I20240502 01:52:44.225353 11433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 01:52:44.229955 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.67:0
--local_ip_for_outbound_sockets=127.11.42.67
--webserver_interface=127.11.42.67
--webserver_port=0
--tserver_master_addrs=127.11.42.126:40815
--builtin_ntp_servers=127.11.42.84:45633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:52:44.237282 14156 heartbeater.cc:347] Connected to a master server at 127.11.42.126:40815
I20240502 01:52:44.237535 14156 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:44.238266 14156 heartbeater.cc:511] Master 127.11.42.126:40815 requested a full tablet report, sending...
I20240502 01:52:44.239529 13836 ts_manager.cc:194] Registered new tserver with Master: 1478241a03934d029dc7030480423cab (127.11.42.66:34613)
I20240502 01:52:44.240262 13836 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.66:38813
W20240502 01:52:44.534219 14159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:44.534487 14159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:44.534612 14159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:44.541788 14159 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:44.541901 14159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:44.542042 14159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.67
I20240502 01:52:44.550484 14159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:45633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.11.42.67
--webserver_port=0
--tserver_master_addrs=127.11.42.126:40815
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.67
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:44.551642 14159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:44.553290 14159 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:44.565073 14164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:44.565599 14165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:44.565990 14168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:44.566603 14159 server_base.cc:1034] running on GCE node
I20240502 01:52:44.567149 14159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:44.568017 14159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:44.569193 14159 hybrid_clock.cc:648] HybridClock initialized: now 1714614764569146 us; error 43 us; skew 500 ppm
I20240502 01:52:44.569476 14159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:44.571501 14159 webserver.cc:460] Webserver started at http://127.11.42.67:33451/ using document root <none> and password file <none>
I20240502 01:52:44.572180 14159 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:44.572283 14159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:44.572578 14159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:44.575141 14159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "020e5f8ebb004e2ab6fddaece6663ece"
format_stamp: "Formatted at 2024-05-02 01:52:44 on dist-test-slave-xvk5"
I20240502 01:52:44.575892 14159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "020e5f8ebb004e2ab6fddaece6663ece"
format_stamp: "Formatted at 2024-05-02 01:52:44 on dist-test-slave-xvk5"
I20240502 01:52:44.581605 14159 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20240502 01:52:44.584743 14174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:44.585963 14159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:52:44.586102 14159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "020e5f8ebb004e2ab6fddaece6663ece"
format_stamp: "Formatted at 2024-05-02 01:52:44 on dist-test-slave-xvk5"
I20240502 01:52:44.586282 14159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:44.605928 14159 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:44.606720 14159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:44.606977 14159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:44.607887 14159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:44.609324 14159 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:44.609423 14159 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:44.609548 14159 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:44.609617 14159 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:44.637621 14159 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.67:37505
I20240502 01:52:44.637682 14287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.67:37505 every 8 connection(s)
I20240502 01:52:44.638824 14159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 01:52:44.645686 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 14159
I20240502 01:52:44.645900 11433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 01:52:44.649215 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.68:0
--local_ip_for_outbound_sockets=127.11.42.68
--webserver_interface=127.11.42.68
--webserver_port=0
--tserver_master_addrs=127.11.42.126:40815
--builtin_ntp_servers=127.11.42.84:45633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:52:44.650930 14288 heartbeater.cc:347] Connected to a master server at 127.11.42.126:40815
I20240502 01:52:44.651239 14288 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:44.652163 14288 heartbeater.cc:511] Master 127.11.42.126:40815 requested a full tablet report, sending...
I20240502 01:52:44.653577 13836 ts_manager.cc:194] Registered new tserver with Master: 020e5f8ebb004e2ab6fddaece6663ece (127.11.42.67:37505)
I20240502 01:52:44.654323 13836 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.67:52467
I20240502 01:52:44.831878 14024 heartbeater.cc:503] Master 127.11.42.126:40815 was elected leader, sending a full tablet report...
W20240502 01:52:44.951866 14291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:44.952121 14291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:44.952212 14291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:44.959460 14291 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:44.959568 14291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:44.959687 14291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.68
I20240502 01:52:44.968698 14291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:45633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.11.42.68
--webserver_port=0
--tserver_master_addrs=127.11.42.126:40815
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.68
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:44.969898 14291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:44.971554 14291 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:44.983062 14296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:44.983548 14297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:44.983956 14299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:44.984462 14291 server_base.cc:1034] running on GCE node
I20240502 01:52:44.984982 14291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:44.985867 14291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:44.987051 14291 hybrid_clock.cc:648] HybridClock initialized: now 1714614764987004 us; error 37 us; skew 500 ppm
I20240502 01:52:44.987354 14291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:44.989375 14291 webserver.cc:460] Webserver started at http://127.11.42.68:36969/ using document root <none> and password file <none>
I20240502 01:52:44.990103 14291 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:44.990211 14291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:44.990515 14291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:44.992972 14291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "1607e211ee8c48de9fa56e78a3b01418"
format_stamp: "Formatted at 2024-05-02 01:52:44 on dist-test-slave-xvk5"
I20240502 01:52:44.993731 14291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "1607e211ee8c48de9fa56e78a3b01418"
format_stamp: "Formatted at 2024-05-02 01:52:44 on dist-test-slave-xvk5"
I20240502 01:52:44.998289 14291 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 01:52:45.001379 14305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:45.002575 14291 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20240502 01:52:45.002710 14291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "1607e211ee8c48de9fa56e78a3b01418"
format_stamp: "Formatted at 2024-05-02 01:52:44 on dist-test-slave-xvk5"
I20240502 01:52:45.002885 14291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:45.019495 14291 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:45.020213 14291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:45.020479 14291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:45.021384 14291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:45.022764 14291 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:45.022858 14291 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:45.022964 14291 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:45.023031 14291 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:45.052155 14291 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.68:41779
I20240502 01:52:45.052215 14417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.68:41779 every 8 connection(s)
I20240502 01:52:45.053416 14291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 01:52:45.053872 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 14291
I20240502 01:52:45.054040 11433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 01:52:45.057689 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.69:0
--local_ip_for_outbound_sockets=127.11.42.69
--webserver_interface=127.11.42.69
--webserver_port=0
--tserver_master_addrs=127.11.42.126:40815
--builtin_ntp_servers=127.11.42.84:45633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:52:45.067200 14418 heartbeater.cc:347] Connected to a master server at 127.11.42.126:40815
I20240502 01:52:45.067477 14418 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:45.068241 14418 heartbeater.cc:511] Master 127.11.42.126:40815 requested a full tablet report, sending...
I20240502 01:52:45.069590 13836 ts_manager.cc:194] Registered new tserver with Master: 1607e211ee8c48de9fa56e78a3b01418 (127.11.42.68:41779)
I20240502 01:52:45.070344 13836 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.68:55523
I20240502 01:52:45.242587 14156 heartbeater.cc:503] Master 127.11.42.126:40815 was elected leader, sending a full tablet report...
W20240502 01:52:45.364249 14421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:45.364519 14421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:45.364651 14421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:45.371805 14421 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:45.371915 14421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:45.372058 14421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.69
I20240502 01:52:45.380789 14421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:45633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.11.42.69
--webserver_port=0
--tserver_master_addrs=127.11.42.126:40815
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.69
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:45.381963 14421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:45.383527 14421 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:45.394691 14426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:45.395042 14427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:45.395427 14429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:45.395815 14421 server_base.cc:1034] running on GCE node
I20240502 01:52:45.396333 14421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:45.397169 14421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:45.398377 14421 hybrid_clock.cc:648] HybridClock initialized: now 1714614765398341 us; error 33 us; skew 500 ppm
I20240502 01:52:45.398651 14421 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:45.400635 14421 webserver.cc:460] Webserver started at http://127.11.42.69:36569/ using document root <none> and password file <none>
I20240502 01:52:45.401347 14421 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:45.401453 14421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:45.401744 14421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:45.404254 14421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "95e7c90cdc634d64a66dad6805e1680c"
format_stamp: "Formatted at 2024-05-02 01:52:45 on dist-test-slave-xvk5"
I20240502 01:52:45.404960 14421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "95e7c90cdc634d64a66dad6805e1680c"
format_stamp: "Formatted at 2024-05-02 01:52:45 on dist-test-slave-xvk5"
I20240502 01:52:45.409802 14421 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 01:52:45.412917 14435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:45.414136 14421 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 01:52:45.414285 14421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "95e7c90cdc634d64a66dad6805e1680c"
format_stamp: "Formatted at 2024-05-02 01:52:45 on dist-test-slave-xvk5"
I20240502 01:52:45.414461 14421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:45.453250 14421 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:45.454035 14421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:45.454299 14421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:45.455170 14421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:45.456663 14421 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:45.456754 14421 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:45.456859 14421 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:45.456925 14421 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:45.486271 14421 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.69:43981
I20240502 01:52:45.486454 14548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.69:43981 every 8 connection(s)
I20240502 01:52:45.487749 14421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:52:45.495041 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 14421
I20240502 01:52:45.495240 11433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 01:52:45.498555 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.70:0
--local_ip_for_outbound_sockets=127.11.42.70
--webserver_interface=127.11.42.70
--webserver_port=0
--tserver_master_addrs=127.11.42.126:40815
--builtin_ntp_servers=127.11.42.84:45633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:52:45.500278 14549 heartbeater.cc:347] Connected to a master server at 127.11.42.126:40815
I20240502 01:52:45.500586 14549 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:45.501521 14549 heartbeater.cc:511] Master 127.11.42.126:40815 requested a full tablet report, sending...
I20240502 01:52:45.502935 13836 ts_manager.cc:194] Registered new tserver with Master: 95e7c90cdc634d64a66dad6805e1680c (127.11.42.69:43981)
I20240502 01:52:45.503976 13836 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.69:58981
I20240502 01:52:45.656530 14288 heartbeater.cc:503] Master 127.11.42.126:40815 was elected leader, sending a full tablet report...
W20240502 01:52:45.816502 14552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:45.816758 14552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:45.816880 14552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:45.824229 14552 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:45.824359 14552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:45.824505 14552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.70
I20240502 01:52:45.833068 14552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:45633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.11.42.70
--webserver_port=0
--tserver_master_addrs=127.11.42.126:40815
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.70
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:45.834393 14552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:45.836216 14552 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:45.848707 14557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:45.849239 14558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:45.849529 14560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:45.849947 14552 server_base.cc:1034] running on GCE node
I20240502 01:52:45.850481 14552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:45.851356 14552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:45.852563 14552 hybrid_clock.cc:648] HybridClock initialized: now 1714614765852517 us; error 40 us; skew 500 ppm
I20240502 01:52:45.852839 14552 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:45.854974 14552 webserver.cc:460] Webserver started at http://127.11.42.70:46747/ using document root <none> and password file <none>
I20240502 01:52:45.855649 14552 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:45.855746 14552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:45.856041 14552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:45.858511 14552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "2ee638444633463b943c881866708482"
format_stamp: "Formatted at 2024-05-02 01:52:45 on dist-test-slave-xvk5"
I20240502 01:52:45.859208 14552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "2ee638444633463b943c881866708482"
format_stamp: "Formatted at 2024-05-02 01:52:45 on dist-test-slave-xvk5"
I20240502 01:52:45.863957 14552 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20240502 01:52:45.867234 14566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:45.868538 14552 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240502 01:52:45.868700 14552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "2ee638444633463b943c881866708482"
format_stamp: "Formatted at 2024-05-02 01:52:45 on dist-test-slave-xvk5"
I20240502 01:52:45.868870 14552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:45.888423 14552 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:45.889190 14552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:45.889437 14552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:45.890347 14552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:45.891840 14552 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:45.891932 14552 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:45.892031 14552 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:45.892094 14552 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:45.929958 14552 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.70:45255
I20240502 01:52:45.930001 14679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.70:45255 every 8 connection(s)
I20240502 01:52:45.931406 14552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 01:52:45.941639 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 14552
I20240502 01:52:45.941802 11433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 01:52:45.944130 14680 heartbeater.cc:347] Connected to a master server at 127.11.42.126:40815
I20240502 01:52:45.944423 14680 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:45.945425 14680 heartbeater.cc:511] Master 127.11.42.126:40815 requested a full tablet report, sending...
I20240502 01:52:45.945528 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.71:0
--local_ip_for_outbound_sockets=127.11.42.71
--webserver_interface=127.11.42.71
--webserver_port=0
--tserver_master_addrs=127.11.42.126:40815
--builtin_ntp_servers=127.11.42.84:45633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:52:45.946926 13836 ts_manager.cc:194] Registered new tserver with Master: 2ee638444633463b943c881866708482 (127.11.42.70:45255)
I20240502 01:52:45.947881 13836 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.70:49319
I20240502 01:52:46.072607 14418 heartbeater.cc:503] Master 127.11.42.126:40815 was elected leader, sending a full tablet report...
W20240502 01:52:46.279896 14683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:46.280153 14683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:46.280256 14683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:46.287261 14683 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:46.287359 14683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:46.287474 14683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.71
I20240502 01:52:46.295969 14683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:45633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.11.42.71
--webserver_port=0
--tserver_master_addrs=127.11.42.126:40815
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.71
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:46.297178 14683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:46.298794 14683 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:46.309940 14688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:46.310451 14691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:46.310863 14689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:46.311367 14683 server_base.cc:1034] running on GCE node
I20240502 01:52:46.311875 14683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:46.312717 14683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:46.313900 14683 hybrid_clock.cc:648] HybridClock initialized: now 1714614766313859 us; error 37 us; skew 500 ppm
I20240502 01:52:46.314178 14683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:46.316177 14683 webserver.cc:460] Webserver started at http://127.11.42.71:44441/ using document root <none> and password file <none>
I20240502 01:52:46.316828 14683 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:46.316932 14683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:46.317273 14683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:46.319711 14683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "39e5e5c1a5e74915800d96f720aee8c4"
format_stamp: "Formatted at 2024-05-02 01:52:46 on dist-test-slave-xvk5"
I20240502 01:52:46.320423 14683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "39e5e5c1a5e74915800d96f720aee8c4"
format_stamp: "Formatted at 2024-05-02 01:52:46 on dist-test-slave-xvk5"
I20240502 01:52:46.325170 14683 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20240502 01:52:46.328194 14697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:46.329411 14683 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:52:46.329551 14683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "39e5e5c1a5e74915800d96f720aee8c4"
format_stamp: "Formatted at 2024-05-02 01:52:46 on dist-test-slave-xvk5"
I20240502 01:52:46.329721 14683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:46.342133 14683 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:46.342798 14683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:46.343037 14683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:46.343839 14683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:46.345201 14683 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:46.345304 14683 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:46.345398 14683 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:46.345464 14683 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:46.374303 14683 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.71:37071
I20240502 01:52:46.374356 14811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.71:37071 every 8 connection(s)
I20240502 01:52:46.375560 14683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 01:52:46.384850 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 14683
I20240502 01:52:46.385103 11433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 01:52:46.387585 14812 heartbeater.cc:347] Connected to a master server at 127.11.42.126:40815
I20240502 01:52:46.387899 14812 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:46.388844 14812 heartbeater.cc:511] Master 127.11.42.126:40815 requested a full tablet report, sending...
I20240502 01:52:46.390153 13836 ts_manager.cc:194] Registered new tserver with Master: 39e5e5c1a5e74915800d96f720aee8c4 (127.11.42.71:37071)
I20240502 01:52:46.390879 13836 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.71:45759
I20240502 01:52:46.400938 11433 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 01:52:46.415901 13836 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43710:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 01:52:46.441885 14615 tablet_service.cc:1467] Processing CreateTablet for tablet 563e1017b8774231bd29a1f8e628b4dc (DEFAULT_TABLE table=TestTable [id=4cfb76b46ca047699368d9096ba5e420]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:46.443213 14615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 563e1017b8774231bd29a1f8e628b4dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:46.452448 14826 tablet_bootstrap.cc:492] T 563e1017b8774231bd29a1f8e628b4dc P 2ee638444633463b943c881866708482: Bootstrap starting.
I20240502 01:52:46.455036 14826 tablet_bootstrap.cc:654] T 563e1017b8774231bd29a1f8e628b4dc P 2ee638444633463b943c881866708482: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:46.456034 14826 log.cc:824] T 563e1017b8774231bd29a1f8e628b4dc P 2ee638444633463b943c881866708482: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:46.458215 14826 tablet_bootstrap.cc:492] T 563e1017b8774231bd29a1f8e628b4dc P 2ee638444633463b943c881866708482: No bootstrap required, opened a new log
I20240502 01:52:46.458447 14826 ts_tablet_manager.cc:1367] T 563e1017b8774231bd29a1f8e628b4dc P 2ee638444633463b943c881866708482: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 01:52:46.463977 14826 raft_consensus.cc:348] T 563e1017b8774231bd29a1f8e628b4dc P 2ee638444633463b943c881866708482 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ee638444633463b943c881866708482" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 45255 } }
I20240502 01:52:46.464308 14826 raft_consensus.cc:374] T 563e1017b8774231bd29a1f8e628b4dc P 2ee638444633463b943c881866708482 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:46.464407 14826 raft_consensus.cc:729] T 563e1017b8774231bd29a1f8e628b4dc P 2ee638444633463b943c881866708482 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ee638444633463b943c881866708482, State: Initialized, Role: FOLLOWER
I20240502 01:52:46.464923 14826 consensus_queue.cc:260] T 563e1017b8774231bd29a1f8e628b4dc P 2ee638444633463b943c881866708482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ee638444633463b943c881866708482" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 45255 } }
I20240502 01:52:46.465135 14826 raft_consensus.cc:388] T 563e1017b8774231bd29a1f8e628b4dc P 2ee638444633463b943c881866708482 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:46.465265 14826 raft_consensus.cc:482] T 563e1017b8774231bd29a1f8e628b4dc P 2ee638444633463b943c881866708482 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:46.465420 14826 raft_consensus.cc:3037] T 563e1017b8774231bd29a1f8e628b4dc P 2ee638444633463b943c881866708482 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:46.467255 14826 raft_consensus.cc:504] T 563e1017b8774231bd29a1f8e628b4dc P 2ee638444633463b943c881866708482 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ee638444633463b943c881866708482" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 45255 } }
I20240502 01:52:46.467684 14826 leader_election.cc:302] T 563e1017b8774231bd29a1f8e628b4dc P 2ee638444633463b943c881866708482 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ee638444633463b943c881866708482; no voters:
I20240502 01:52:46.468201 14826 leader_election.cc:288] T 563e1017b8774231bd29a1f8e628b4dc P 2ee638444633463b943c881866708482 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:46.468361 14828 raft_consensus.cc:2781] T 563e1017b8774231bd29a1f8e628b4dc P 2ee638444633463b943c881866708482 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:46.469203 14680 heartbeater.cc:503] Master 127.11.42.126:40815 was elected leader, sending a full tablet report...
I20240502 01:52:46.469473 14828 raft_consensus.cc:686] T 563e1017b8774231bd29a1f8e628b4dc P 2ee638444633463b943c881866708482 [term 1 LEADER]: Becoming Leader. State: Replica: 2ee638444633463b943c881866708482, State: Running, Role: LEADER
I20240502 01:52:46.470018 14828 consensus_queue.cc:237] T 563e1017b8774231bd29a1f8e628b4dc P 2ee638444633463b943c881866708482 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ee638444633463b943c881866708482" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 45255 } }
I20240502 01:52:46.470249 14826 ts_tablet_manager.cc:1398] T 563e1017b8774231bd29a1f8e628b4dc P 2ee638444633463b943c881866708482: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20240502 01:52:46.474845 13836 catalog_manager.cc:5484] T 563e1017b8774231bd29a1f8e628b4dc P 2ee638444633463b943c881866708482 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ee638444633463b943c881866708482 (127.11.42.70). New cstate: current_term: 1 leader_uuid: "2ee638444633463b943c881866708482" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ee638444633463b943c881866708482" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 45255 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:46.484906 11433 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 01:52:46.486791 11433 ts_itest-base.cc:233] Waiting for 0 tablets on tserver d43426b5bc874e8482e7b3205a9ba970 to finish bootstrapping
I20240502 01:52:46.495749 11433 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 1478241a03934d029dc7030480423cab to finish bootstrapping
I20240502 01:52:46.503607 11433 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 020e5f8ebb004e2ab6fddaece6663ece to finish bootstrapping
I20240502 01:52:46.506647 14549 heartbeater.cc:503] Master 127.11.42.126:40815 was elected leader, sending a full tablet report...
I20240502 01:52:46.511860 11433 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 1607e211ee8c48de9fa56e78a3b01418 to finish bootstrapping
I20240502 01:52:46.519579 11433 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 95e7c90cdc634d64a66dad6805e1680c to finish bootstrapping
I20240502 01:52:46.527067 11433 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 2ee638444633463b943c881866708482 to finish bootstrapping
I20240502 01:52:46.533537 11433 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 39e5e5c1a5e74915800d96f720aee8c4 to finish bootstrapping
I20240502 01:52:46.541345 11433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu pid 14159
I20240502 01:52:46.542243 14283 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:46.621528 11433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu with pid 14159
I20240502 01:52:46.635872 11433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu pid 14291
I20240502 01:52:46.636925 14413 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:46.715761 11433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu with pid 14291
I20240502 01:52:46.729403 11433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu pid 14421
I20240502 01:52:46.730437 14544 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:46.808647 11433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu with pid 14421
I20240502 01:52:46.822646 11433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu pid 14552
I20240502 01:52:46.823669 14675 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:46.908299 11433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu with pid 14552
I20240502 01:52:46.922295 11433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu pid 14683
I20240502 01:52:46.923301 14807 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:47.001645 11433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu with pid 14683
I20240502 01:52:50.766942 13835 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43710:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:52:50.785873 14090 tablet_service.cc:1467] Processing CreateTablet for tablet aa387d2b1dcc42ddb34005f08dde2d8d (DEFAULT_TABLE table=rebalance_test_table_0 [id=cd22e1a0d15d492fb0d2e238ceb715ea]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:50.786092 13956 tablet_service.cc:1467] Processing CreateTablet for tablet 62ad914a819c4c7a85856be3c1855270 (DEFAULT_TABLE table=rebalance_test_table_0 [id=cd22e1a0d15d492fb0d2e238ceb715ea]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:50.787366 14090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa387d2b1dcc42ddb34005f08dde2d8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:50.787467 13956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62ad914a819c4c7a85856be3c1855270. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:50.786080 13957 tablet_service.cc:1467] Processing CreateTablet for tablet 67bc0cd4c3b94782ac1d9f150fcdbd03 (DEFAULT_TABLE table=rebalance_test_table_0 [id=cd22e1a0d15d492fb0d2e238ceb715ea]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:50.789489 13957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67bc0cd4c3b94782ac1d9f150fcdbd03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:50.797698 14854 tablet_bootstrap.cc:492] T aa387d2b1dcc42ddb34005f08dde2d8d P 1478241a03934d029dc7030480423cab: Bootstrap starting.
I20240502 01:52:50.799801 14855 tablet_bootstrap.cc:492] T 62ad914a819c4c7a85856be3c1855270 P d43426b5bc874e8482e7b3205a9ba970: Bootstrap starting.
I20240502 01:52:50.800963 14854 tablet_bootstrap.cc:654] T aa387d2b1dcc42ddb34005f08dde2d8d P 1478241a03934d029dc7030480423cab: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:50.802287 14854 log.cc:824] T aa387d2b1dcc42ddb34005f08dde2d8d P 1478241a03934d029dc7030480423cab: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:50.803028 14855 tablet_bootstrap.cc:654] T 62ad914a819c4c7a85856be3c1855270 P d43426b5bc874e8482e7b3205a9ba970: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:50.804096 14855 log.cc:824] T 62ad914a819c4c7a85856be3c1855270 P d43426b5bc874e8482e7b3205a9ba970: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:50.805137 14854 tablet_bootstrap.cc:492] T aa387d2b1dcc42ddb34005f08dde2d8d P 1478241a03934d029dc7030480423cab: No bootstrap required, opened a new log
I20240502 01:52:50.805409 14854 ts_tablet_manager.cc:1367] T aa387d2b1dcc42ddb34005f08dde2d8d P 1478241a03934d029dc7030480423cab: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240502 01:52:50.806742 14855 tablet_bootstrap.cc:492] T 62ad914a819c4c7a85856be3c1855270 P d43426b5bc874e8482e7b3205a9ba970: No bootstrap required, opened a new log
I20240502 01:52:50.806988 14855 ts_tablet_manager.cc:1367] T 62ad914a819c4c7a85856be3c1855270 P d43426b5bc874e8482e7b3205a9ba970: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20240502 01:52:50.813679 14854 raft_consensus.cc:348] T aa387d2b1dcc42ddb34005f08dde2d8d P 1478241a03934d029dc7030480423cab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } }
I20240502 01:52:50.814049 14854 raft_consensus.cc:374] T aa387d2b1dcc42ddb34005f08dde2d8d P 1478241a03934d029dc7030480423cab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:50.814150 14854 raft_consensus.cc:729] T aa387d2b1dcc42ddb34005f08dde2d8d P 1478241a03934d029dc7030480423cab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1478241a03934d029dc7030480423cab, State: Initialized, Role: FOLLOWER
I20240502 01:52:50.814814 14854 consensus_queue.cc:260] T aa387d2b1dcc42ddb34005f08dde2d8d P 1478241a03934d029dc7030480423cab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } }
I20240502 01:52:50.815053 14854 raft_consensus.cc:388] T aa387d2b1dcc42ddb34005f08dde2d8d P 1478241a03934d029dc7030480423cab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:50.815182 14854 raft_consensus.cc:482] T aa387d2b1dcc42ddb34005f08dde2d8d P 1478241a03934d029dc7030480423cab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:50.815349 14854 raft_consensus.cc:3037] T aa387d2b1dcc42ddb34005f08dde2d8d P 1478241a03934d029dc7030480423cab [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:50.815508 14855 raft_consensus.cc:348] T 62ad914a819c4c7a85856be3c1855270 P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } }
I20240502 01:52:50.815865 14855 raft_consensus.cc:374] T 62ad914a819c4c7a85856be3c1855270 P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:50.815968 14855 raft_consensus.cc:729] T 62ad914a819c4c7a85856be3c1855270 P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43426b5bc874e8482e7b3205a9ba970, State: Initialized, Role: FOLLOWER
I20240502 01:52:50.816586 14855 consensus_queue.cc:260] T 62ad914a819c4c7a85856be3c1855270 P d43426b5bc874e8482e7b3205a9ba970 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } }
I20240502 01:52:50.816860 14855 raft_consensus.cc:388] T 62ad914a819c4c7a85856be3c1855270 P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:50.817059 14855 raft_consensus.cc:482] T 62ad914a819c4c7a85856be3c1855270 P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:50.817245 14855 raft_consensus.cc:3037] T 62ad914a819c4c7a85856be3c1855270 P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:50.818136 14854 raft_consensus.cc:504] T aa387d2b1dcc42ddb34005f08dde2d8d P 1478241a03934d029dc7030480423cab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } }
I20240502 01:52:50.818655 14854 leader_election.cc:302] T aa387d2b1dcc42ddb34005f08dde2d8d P 1478241a03934d029dc7030480423cab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1478241a03934d029dc7030480423cab; no voters:
I20240502 01:52:50.819262 14854 leader_election.cc:288] T aa387d2b1dcc42ddb34005f08dde2d8d P 1478241a03934d029dc7030480423cab [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:50.819393 14858 raft_consensus.cc:2781] T aa387d2b1dcc42ddb34005f08dde2d8d P 1478241a03934d029dc7030480423cab [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:50.819942 14855 raft_consensus.cc:504] T 62ad914a819c4c7a85856be3c1855270 P d43426b5bc874e8482e7b3205a9ba970 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } }
I20240502 01:52:50.820473 14855 leader_election.cc:302] T 62ad914a819c4c7a85856be3c1855270 P d43426b5bc874e8482e7b3205a9ba970 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d43426b5bc874e8482e7b3205a9ba970; no voters:
I20240502 01:52:50.820762 14858 raft_consensus.cc:686] T aa387d2b1dcc42ddb34005f08dde2d8d P 1478241a03934d029dc7030480423cab [term 1 LEADER]: Becoming Leader. State: Replica: 1478241a03934d029dc7030480423cab, State: Running, Role: LEADER
I20240502 01:52:50.820920 14854 ts_tablet_manager.cc:1398] T aa387d2b1dcc42ddb34005f08dde2d8d P 1478241a03934d029dc7030480423cab: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20240502 01:52:50.821219 14855 leader_election.cc:288] T 62ad914a819c4c7a85856be3c1855270 P d43426b5bc874e8482e7b3205a9ba970 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:50.821327 14861 raft_consensus.cc:2781] T 62ad914a819c4c7a85856be3c1855270 P d43426b5bc874e8482e7b3205a9ba970 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:50.821331 14858 consensus_queue.cc:237] T aa387d2b1dcc42ddb34005f08dde2d8d P 1478241a03934d029dc7030480423cab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } }
I20240502 01:52:50.822326 14861 raft_consensus.cc:686] T 62ad914a819c4c7a85856be3c1855270 P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Becoming Leader. State: Replica: d43426b5bc874e8482e7b3205a9ba970, State: Running, Role: LEADER
I20240502 01:52:50.822716 14855 ts_tablet_manager.cc:1398] T 62ad914a819c4c7a85856be3c1855270 P d43426b5bc874e8482e7b3205a9ba970: Time spent starting tablet: real 0.016s user 0.005s sys 0.011s
I20240502 01:52:50.822840 14861 consensus_queue.cc:237] T 62ad914a819c4c7a85856be3c1855270 P d43426b5bc874e8482e7b3205a9ba970 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } }
I20240502 01:52:50.823047 14855 tablet_bootstrap.cc:492] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P d43426b5bc874e8482e7b3205a9ba970: Bootstrap starting.
I20240502 01:52:50.824204 14855 tablet_bootstrap.cc:654] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P d43426b5bc874e8482e7b3205a9ba970: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:50.827512 13835 catalog_manager.cc:5484] T 62ad914a819c4c7a85856be3c1855270 P d43426b5bc874e8482e7b3205a9ba970 reported cstate change: term changed from 0 to 1, leader changed from <none> to d43426b5bc874e8482e7b3205a9ba970 (127.11.42.65). New cstate: current_term: 1 leader_uuid: "d43426b5bc874e8482e7b3205a9ba970" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:50.831530 14855 tablet_bootstrap.cc:492] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P d43426b5bc874e8482e7b3205a9ba970: No bootstrap required, opened a new log
I20240502 01:52:50.831677 14855 ts_tablet_manager.cc:1367] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P d43426b5bc874e8482e7b3205a9ba970: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240502 01:52:50.832551 14855 raft_consensus.cc:348] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } }
I20240502 01:52:50.832712 14855 raft_consensus.cc:374] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:50.832816 14855 raft_consensus.cc:729] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43426b5bc874e8482e7b3205a9ba970, State: Initialized, Role: FOLLOWER
I20240502 01:52:50.833112 14855 consensus_queue.cc:260] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P d43426b5bc874e8482e7b3205a9ba970 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } }
I20240502 01:52:50.833319 14855 raft_consensus.cc:388] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:50.833401 14855 raft_consensus.cc:482] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:50.833233 13835 catalog_manager.cc:5484] T aa387d2b1dcc42ddb34005f08dde2d8d P 1478241a03934d029dc7030480423cab reported cstate change: term changed from 0 to 1, leader changed from <none> to 1478241a03934d029dc7030480423cab (127.11.42.66). New cstate: current_term: 1 leader_uuid: "1478241a03934d029dc7030480423cab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:50.833501 14855 raft_consensus.cc:3037] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:50.835490 14855 raft_consensus.cc:504] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P d43426b5bc874e8482e7b3205a9ba970 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } }
I20240502 01:52:50.835691 14855 leader_election.cc:302] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P d43426b5bc874e8482e7b3205a9ba970 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d43426b5bc874e8482e7b3205a9ba970; no voters:
I20240502 01:52:50.835853 14855 leader_election.cc:288] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P d43426b5bc874e8482e7b3205a9ba970 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:50.835920 14861 raft_consensus.cc:2781] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P d43426b5bc874e8482e7b3205a9ba970 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:50.836164 14861 raft_consensus.cc:686] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Becoming Leader. State: Replica: d43426b5bc874e8482e7b3205a9ba970, State: Running, Role: LEADER
I20240502 01:52:50.836266 14855 ts_tablet_manager.cc:1398] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P d43426b5bc874e8482e7b3205a9ba970: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:52:50.836413 14861 consensus_queue.cc:237] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P d43426b5bc874e8482e7b3205a9ba970 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } }
I20240502 01:52:50.838405 13836 catalog_manager.cc:5484] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P d43426b5bc874e8482e7b3205a9ba970 reported cstate change: term changed from 0 to 1, leader changed from <none> to d43426b5bc874e8482e7b3205a9ba970 (127.11.42.65). New cstate: current_term: 1 leader_uuid: "d43426b5bc874e8482e7b3205a9ba970" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } health_report { overall_health: HEALTHY } } }
W20240502 01:52:51.150266 14870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:51.150535 14870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:51.157778 14870 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 122.111 ms
time per row: 0.0610553 ms
I20240502 01:52:51.417384 13836 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43710:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:52:51.425297 14090 tablet_service.cc:1467] Processing CreateTablet for tablet 65d44d1da4bf4366860fc5d1a1e0785c (DEFAULT_TABLE table=rebalance_test_table_1 [id=9069de6d20e94872b794e4757f1a1d1a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:51.425423 13956 tablet_service.cc:1467] Processing CreateTablet for tablet 86219a30419046dcb0384f5e539cdf15 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9069de6d20e94872b794e4757f1a1d1a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:51.425421 14089 tablet_service.cc:1467] Processing CreateTablet for tablet f9f4aab07f984168bdfec24f908c124d (DEFAULT_TABLE table=rebalance_test_table_1 [id=9069de6d20e94872b794e4757f1a1d1a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:51.425782 13956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86219a30419046dcb0384f5e539cdf15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:51.425786 14090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65d44d1da4bf4366860fc5d1a1e0785c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:51.426025 14089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9f4aab07f984168bdfec24f908c124d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:51.432006 14910 tablet_bootstrap.cc:492] T f9f4aab07f984168bdfec24f908c124d P 1478241a03934d029dc7030480423cab: Bootstrap starting.
I20240502 01:52:51.432499 14911 tablet_bootstrap.cc:492] T 86219a30419046dcb0384f5e539cdf15 P d43426b5bc874e8482e7b3205a9ba970: Bootstrap starting.
I20240502 01:52:51.433712 14910 tablet_bootstrap.cc:654] T f9f4aab07f984168bdfec24f908c124d P 1478241a03934d029dc7030480423cab: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:51.433993 14911 tablet_bootstrap.cc:654] T 86219a30419046dcb0384f5e539cdf15 P d43426b5bc874e8482e7b3205a9ba970: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:51.435808 14910 tablet_bootstrap.cc:492] T f9f4aab07f984168bdfec24f908c124d P 1478241a03934d029dc7030480423cab: No bootstrap required, opened a new log
I20240502 01:52:51.435806 14911 tablet_bootstrap.cc:492] T 86219a30419046dcb0384f5e539cdf15 P d43426b5bc874e8482e7b3205a9ba970: No bootstrap required, opened a new log
I20240502 01:52:51.435945 14910 ts_tablet_manager.cc:1367] T f9f4aab07f984168bdfec24f908c124d P 1478241a03934d029dc7030480423cab: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:52:51.435946 14911 ts_tablet_manager.cc:1367] T 86219a30419046dcb0384f5e539cdf15 P d43426b5bc874e8482e7b3205a9ba970: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:52:51.436698 14911 raft_consensus.cc:348] T 86219a30419046dcb0384f5e539cdf15 P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } }
I20240502 01:52:51.436880 14911 raft_consensus.cc:374] T 86219a30419046dcb0384f5e539cdf15 P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:51.436879 14910 raft_consensus.cc:348] T f9f4aab07f984168bdfec24f908c124d P 1478241a03934d029dc7030480423cab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } }
I20240502 01:52:51.436978 14911 raft_consensus.cc:729] T 86219a30419046dcb0384f5e539cdf15 P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43426b5bc874e8482e7b3205a9ba970, State: Initialized, Role: FOLLOWER
I20240502 01:52:51.437073 14910 raft_consensus.cc:374] T f9f4aab07f984168bdfec24f908c124d P 1478241a03934d029dc7030480423cab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:51.437202 14910 raft_consensus.cc:729] T f9f4aab07f984168bdfec24f908c124d P 1478241a03934d029dc7030480423cab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1478241a03934d029dc7030480423cab, State: Initialized, Role: FOLLOWER
I20240502 01:52:51.437251 14911 consensus_queue.cc:260] T 86219a30419046dcb0384f5e539cdf15 P d43426b5bc874e8482e7b3205a9ba970 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } }
I20240502 01:52:51.437449 14911 raft_consensus.cc:388] T 86219a30419046dcb0384f5e539cdf15 P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:51.437448 14910 consensus_queue.cc:260] T f9f4aab07f984168bdfec24f908c124d P 1478241a03934d029dc7030480423cab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } }
I20240502 01:52:51.437530 14911 raft_consensus.cc:482] T 86219a30419046dcb0384f5e539cdf15 P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:51.437633 14911 raft_consensus.cc:3037] T 86219a30419046dcb0384f5e539cdf15 P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:51.437633 14910 raft_consensus.cc:388] T f9f4aab07f984168bdfec24f908c124d P 1478241a03934d029dc7030480423cab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:51.437705 14910 raft_consensus.cc:482] T f9f4aab07f984168bdfec24f908c124d P 1478241a03934d029dc7030480423cab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:51.437811 14910 raft_consensus.cc:3037] T f9f4aab07f984168bdfec24f908c124d P 1478241a03934d029dc7030480423cab [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:51.440165 14911 raft_consensus.cc:504] T 86219a30419046dcb0384f5e539cdf15 P d43426b5bc874e8482e7b3205a9ba970 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } }
I20240502 01:52:51.440265 14910 raft_consensus.cc:504] T f9f4aab07f984168bdfec24f908c124d P 1478241a03934d029dc7030480423cab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } }
I20240502 01:52:51.440434 14911 leader_election.cc:302] T 86219a30419046dcb0384f5e539cdf15 P d43426b5bc874e8482e7b3205a9ba970 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d43426b5bc874e8482e7b3205a9ba970; no voters:
I20240502 01:52:51.440472 14910 leader_election.cc:302] T f9f4aab07f984168bdfec24f908c124d P 1478241a03934d029dc7030480423cab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1478241a03934d029dc7030480423cab; no voters:
I20240502 01:52:51.440632 14910 leader_election.cc:288] T f9f4aab07f984168bdfec24f908c124d P 1478241a03934d029dc7030480423cab [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:51.440634 14911 leader_election.cc:288] T 86219a30419046dcb0384f5e539cdf15 P d43426b5bc874e8482e7b3205a9ba970 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:51.440723 14858 raft_consensus.cc:2781] T f9f4aab07f984168bdfec24f908c124d P 1478241a03934d029dc7030480423cab [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:51.440742 14861 raft_consensus.cc:2781] T 86219a30419046dcb0384f5e539cdf15 P d43426b5bc874e8482e7b3205a9ba970 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:51.441401 14911 ts_tablet_manager.cc:1398] T 86219a30419046dcb0384f5e539cdf15 P d43426b5bc874e8482e7b3205a9ba970: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240502 01:52:51.443542 14861 raft_consensus.cc:686] T 86219a30419046dcb0384f5e539cdf15 P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Becoming Leader. State: Replica: d43426b5bc874e8482e7b3205a9ba970, State: Running, Role: LEADER
I20240502 01:52:51.443866 14858 raft_consensus.cc:686] T f9f4aab07f984168bdfec24f908c124d P 1478241a03934d029dc7030480423cab [term 1 LEADER]: Becoming Leader. State: Replica: 1478241a03934d029dc7030480423cab, State: Running, Role: LEADER
I20240502 01:52:51.443832 14861 consensus_queue.cc:237] T 86219a30419046dcb0384f5e539cdf15 P d43426b5bc874e8482e7b3205a9ba970 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } }
I20240502 01:52:51.444124 14858 consensus_queue.cc:237] T f9f4aab07f984168bdfec24f908c124d P 1478241a03934d029dc7030480423cab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } }
I20240502 01:52:51.446161 14910 ts_tablet_manager.cc:1398] T f9f4aab07f984168bdfec24f908c124d P 1478241a03934d029dc7030480423cab: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20240502 01:52:51.446187 13836 catalog_manager.cc:5484] T 86219a30419046dcb0384f5e539cdf15 P d43426b5bc874e8482e7b3205a9ba970 reported cstate change: term changed from 0 to 1, leader changed from <none> to d43426b5bc874e8482e7b3205a9ba970 (127.11.42.65). New cstate: current_term: 1 leader_uuid: "d43426b5bc874e8482e7b3205a9ba970" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:51.446417 14910 tablet_bootstrap.cc:492] T 65d44d1da4bf4366860fc5d1a1e0785c P 1478241a03934d029dc7030480423cab: Bootstrap starting.
I20240502 01:52:51.447059 13834 catalog_manager.cc:5484] T f9f4aab07f984168bdfec24f908c124d P 1478241a03934d029dc7030480423cab reported cstate change: term changed from 0 to 1, leader changed from <none> to 1478241a03934d029dc7030480423cab (127.11.42.66). New cstate: current_term: 1 leader_uuid: "1478241a03934d029dc7030480423cab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:51.447914 14910 tablet_bootstrap.cc:654] T 65d44d1da4bf4366860fc5d1a1e0785c P 1478241a03934d029dc7030480423cab: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:51.450047 14910 tablet_bootstrap.cc:492] T 65d44d1da4bf4366860fc5d1a1e0785c P 1478241a03934d029dc7030480423cab: No bootstrap required, opened a new log
I20240502 01:52:51.450203 14910 ts_tablet_manager.cc:1367] T 65d44d1da4bf4366860fc5d1a1e0785c P 1478241a03934d029dc7030480423cab: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:52:51.451320 14910 raft_consensus.cc:348] T 65d44d1da4bf4366860fc5d1a1e0785c P 1478241a03934d029dc7030480423cab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } }
I20240502 01:52:51.451548 14910 raft_consensus.cc:374] T 65d44d1da4bf4366860fc5d1a1e0785c P 1478241a03934d029dc7030480423cab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:51.451633 14910 raft_consensus.cc:729] T 65d44d1da4bf4366860fc5d1a1e0785c P 1478241a03934d029dc7030480423cab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1478241a03934d029dc7030480423cab, State: Initialized, Role: FOLLOWER
I20240502 01:52:51.451861 14910 consensus_queue.cc:260] T 65d44d1da4bf4366860fc5d1a1e0785c P 1478241a03934d029dc7030480423cab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } }
I20240502 01:52:51.451995 14910 raft_consensus.cc:388] T 65d44d1da4bf4366860fc5d1a1e0785c P 1478241a03934d029dc7030480423cab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:51.452083 14910 raft_consensus.cc:482] T 65d44d1da4bf4366860fc5d1a1e0785c P 1478241a03934d029dc7030480423cab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:51.452164 14910 raft_consensus.cc:3037] T 65d44d1da4bf4366860fc5d1a1e0785c P 1478241a03934d029dc7030480423cab [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:51.454061 14910 raft_consensus.cc:504] T 65d44d1da4bf4366860fc5d1a1e0785c P 1478241a03934d029dc7030480423cab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } }
I20240502 01:52:51.454257 14910 leader_election.cc:302] T 65d44d1da4bf4366860fc5d1a1e0785c P 1478241a03934d029dc7030480423cab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1478241a03934d029dc7030480423cab; no voters:
I20240502 01:52:51.454471 14910 leader_election.cc:288] T 65d44d1da4bf4366860fc5d1a1e0785c P 1478241a03934d029dc7030480423cab [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:51.454525 14916 raft_consensus.cc:2781] T 65d44d1da4bf4366860fc5d1a1e0785c P 1478241a03934d029dc7030480423cab [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:51.454692 14916 raft_consensus.cc:686] T 65d44d1da4bf4366860fc5d1a1e0785c P 1478241a03934d029dc7030480423cab [term 1 LEADER]: Becoming Leader. State: Replica: 1478241a03934d029dc7030480423cab, State: Running, Role: LEADER
I20240502 01:52:51.454986 14910 ts_tablet_manager.cc:1398] T 65d44d1da4bf4366860fc5d1a1e0785c P 1478241a03934d029dc7030480423cab: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240502 01:52:51.454970 14916 consensus_queue.cc:237] T 65d44d1da4bf4366860fc5d1a1e0785c P 1478241a03934d029dc7030480423cab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } }
I20240502 01:52:51.456979 13834 catalog_manager.cc:5484] T 65d44d1da4bf4366860fc5d1a1e0785c P 1478241a03934d029dc7030480423cab reported cstate change: term changed from 0 to 1, leader changed from <none> to 1478241a03934d029dc7030480423cab (127.11.42.66). New cstate: current_term: 1 leader_uuid: "1478241a03934d029dc7030480423cab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } health_report { overall_health: HEALTHY } } }
W20240502 01:52:51.775424 14921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:51.775696 14921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:51.782794 14921 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 106.431 ms
time per row: 0.0532157 ms
I20240502 01:52:52.023685 13834 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43710:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:52:52.031881 13956 tablet_service.cc:1467] Processing CreateTablet for tablet 0eb60206b8cb40d3b30cdc1f47d67f94 (DEFAULT_TABLE table=rebalance_test_table_2 [id=3c737a3ccdb1430496def9e37e6428d9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:52.032048 14090 tablet_service.cc:1467] Processing CreateTablet for tablet 2d8fb038394d4ad981d208e52b19672f (DEFAULT_TABLE table=rebalance_test_table_2 [id=3c737a3ccdb1430496def9e37e6428d9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:52.032142 13957 tablet_service.cc:1467] Processing CreateTablet for tablet c0620ccc47634258a7962d91cc7bb060 (DEFAULT_TABLE table=rebalance_test_table_2 [id=3c737a3ccdb1430496def9e37e6428d9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:52.032371 13956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0eb60206b8cb40d3b30cdc1f47d67f94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:52.032512 14090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d8fb038394d4ad981d208e52b19672f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:52.032581 13957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0620ccc47634258a7962d91cc7bb060. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:52.037847 14963 tablet_bootstrap.cc:492] T 2d8fb038394d4ad981d208e52b19672f P 1478241a03934d029dc7030480423cab: Bootstrap starting.
I20240502 01:52:52.039023 14964 tablet_bootstrap.cc:492] T 0eb60206b8cb40d3b30cdc1f47d67f94 P d43426b5bc874e8482e7b3205a9ba970: Bootstrap starting.
I20240502 01:52:52.039206 14963 tablet_bootstrap.cc:654] T 2d8fb038394d4ad981d208e52b19672f P 1478241a03934d029dc7030480423cab: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:52.040374 14964 tablet_bootstrap.cc:654] T 0eb60206b8cb40d3b30cdc1f47d67f94 P d43426b5bc874e8482e7b3205a9ba970: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:52.040805 14963 tablet_bootstrap.cc:492] T 2d8fb038394d4ad981d208e52b19672f P 1478241a03934d029dc7030480423cab: No bootstrap required, opened a new log
I20240502 01:52:52.041214 14963 ts_tablet_manager.cc:1367] T 2d8fb038394d4ad981d208e52b19672f P 1478241a03934d029dc7030480423cab: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 01:52:52.041951 14963 raft_consensus.cc:348] T 2d8fb038394d4ad981d208e52b19672f P 1478241a03934d029dc7030480423cab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } }
I20240502 01:52:52.042088 14963 raft_consensus.cc:374] T 2d8fb038394d4ad981d208e52b19672f P 1478241a03934d029dc7030480423cab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:52.042150 14963 raft_consensus.cc:729] T 2d8fb038394d4ad981d208e52b19672f P 1478241a03934d029dc7030480423cab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1478241a03934d029dc7030480423cab, State: Initialized, Role: FOLLOWER
I20240502 01:52:52.042174 14964 tablet_bootstrap.cc:492] T 0eb60206b8cb40d3b30cdc1f47d67f94 P d43426b5bc874e8482e7b3205a9ba970: No bootstrap required, opened a new log
I20240502 01:52:52.042292 14964 ts_tablet_manager.cc:1367] T 0eb60206b8cb40d3b30cdc1f47d67f94 P d43426b5bc874e8482e7b3205a9ba970: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 01:52:52.042357 14963 consensus_queue.cc:260] T 2d8fb038394d4ad981d208e52b19672f P 1478241a03934d029dc7030480423cab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } }
I20240502 01:52:52.042513 14963 raft_consensus.cc:388] T 2d8fb038394d4ad981d208e52b19672f P 1478241a03934d029dc7030480423cab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:52.042598 14963 raft_consensus.cc:482] T 2d8fb038394d4ad981d208e52b19672f P 1478241a03934d029dc7030480423cab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:52.042716 14963 raft_consensus.cc:3037] T 2d8fb038394d4ad981d208e52b19672f P 1478241a03934d029dc7030480423cab [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:52.042939 14964 raft_consensus.cc:348] T 0eb60206b8cb40d3b30cdc1f47d67f94 P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } }
I20240502 01:52:52.043069 14964 raft_consensus.cc:374] T 0eb60206b8cb40d3b30cdc1f47d67f94 P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:52.043155 14964 raft_consensus.cc:729] T 0eb60206b8cb40d3b30cdc1f47d67f94 P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43426b5bc874e8482e7b3205a9ba970, State: Initialized, Role: FOLLOWER
I20240502 01:52:52.043357 14964 consensus_queue.cc:260] T 0eb60206b8cb40d3b30cdc1f47d67f94 P d43426b5bc874e8482e7b3205a9ba970 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } }
I20240502 01:52:52.043506 14964 raft_consensus.cc:388] T 0eb60206b8cb40d3b30cdc1f47d67f94 P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:52.043581 14964 raft_consensus.cc:482] T 0eb60206b8cb40d3b30cdc1f47d67f94 P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:52.043680 14964 raft_consensus.cc:3037] T 0eb60206b8cb40d3b30cdc1f47d67f94 P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:52.044478 14963 raft_consensus.cc:504] T 2d8fb038394d4ad981d208e52b19672f P 1478241a03934d029dc7030480423cab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } }
I20240502 01:52:52.044682 14963 leader_election.cc:302] T 2d8fb038394d4ad981d208e52b19672f P 1478241a03934d029dc7030480423cab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1478241a03934d029dc7030480423cab; no voters:
I20240502 01:52:52.044891 14963 leader_election.cc:288] T 2d8fb038394d4ad981d208e52b19672f P 1478241a03934d029dc7030480423cab [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:52.044981 14914 raft_consensus.cc:2781] T 2d8fb038394d4ad981d208e52b19672f P 1478241a03934d029dc7030480423cab [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:52.045146 14914 raft_consensus.cc:686] T 2d8fb038394d4ad981d208e52b19672f P 1478241a03934d029dc7030480423cab [term 1 LEADER]: Becoming Leader. State: Replica: 1478241a03934d029dc7030480423cab, State: Running, Role: LEADER
I20240502 01:52:52.045408 14914 consensus_queue.cc:237] T 2d8fb038394d4ad981d208e52b19672f P 1478241a03934d029dc7030480423cab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } }
I20240502 01:52:52.045717 14964 raft_consensus.cc:504] T 0eb60206b8cb40d3b30cdc1f47d67f94 P d43426b5bc874e8482e7b3205a9ba970 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } }
I20240502 01:52:52.045961 14964 leader_election.cc:302] T 0eb60206b8cb40d3b30cdc1f47d67f94 P d43426b5bc874e8482e7b3205a9ba970 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d43426b5bc874e8482e7b3205a9ba970; no voters:
I20240502 01:52:52.046156 14964 leader_election.cc:288] T 0eb60206b8cb40d3b30cdc1f47d67f94 P d43426b5bc874e8482e7b3205a9ba970 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:52.046316 14861 raft_consensus.cc:2781] T 0eb60206b8cb40d3b30cdc1f47d67f94 P d43426b5bc874e8482e7b3205a9ba970 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:52.046818 14963 ts_tablet_manager.cc:1398] T 2d8fb038394d4ad981d208e52b19672f P 1478241a03934d029dc7030480423cab: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 01:52:52.047554 13834 catalog_manager.cc:5484] T 2d8fb038394d4ad981d208e52b19672f P 1478241a03934d029dc7030480423cab reported cstate change: term changed from 0 to 1, leader changed from <none> to 1478241a03934d029dc7030480423cab (127.11.42.66). New cstate: current_term: 1 leader_uuid: "1478241a03934d029dc7030480423cab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:52.048552 14861 raft_consensus.cc:686] T 0eb60206b8cb40d3b30cdc1f47d67f94 P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Becoming Leader. State: Replica: d43426b5bc874e8482e7b3205a9ba970, State: Running, Role: LEADER
I20240502 01:52:52.048839 14861 consensus_queue.cc:237] T 0eb60206b8cb40d3b30cdc1f47d67f94 P d43426b5bc874e8482e7b3205a9ba970 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } }
I20240502 01:52:52.049177 14964 ts_tablet_manager.cc:1398] T 0eb60206b8cb40d3b30cdc1f47d67f94 P d43426b5bc874e8482e7b3205a9ba970: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240502 01:52:52.049430 14964 tablet_bootstrap.cc:492] T c0620ccc47634258a7962d91cc7bb060 P d43426b5bc874e8482e7b3205a9ba970: Bootstrap starting.
I20240502 01:52:52.050902 14964 tablet_bootstrap.cc:654] T c0620ccc47634258a7962d91cc7bb060 P d43426b5bc874e8482e7b3205a9ba970: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:52.052057 13832 catalog_manager.cc:5484] T 0eb60206b8cb40d3b30cdc1f47d67f94 P d43426b5bc874e8482e7b3205a9ba970 reported cstate change: term changed from 0 to 1, leader changed from <none> to d43426b5bc874e8482e7b3205a9ba970 (127.11.42.65). New cstate: current_term: 1 leader_uuid: "d43426b5bc874e8482e7b3205a9ba970" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:52.052937 14964 tablet_bootstrap.cc:492] T c0620ccc47634258a7962d91cc7bb060 P d43426b5bc874e8482e7b3205a9ba970: No bootstrap required, opened a new log
I20240502 01:52:52.053095 14964 ts_tablet_manager.cc:1367] T c0620ccc47634258a7962d91cc7bb060 P d43426b5bc874e8482e7b3205a9ba970: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:52:52.053907 14964 raft_consensus.cc:348] T c0620ccc47634258a7962d91cc7bb060 P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } }
I20240502 01:52:52.054044 14964 raft_consensus.cc:374] T c0620ccc47634258a7962d91cc7bb060 P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:52.054111 14964 raft_consensus.cc:729] T c0620ccc47634258a7962d91cc7bb060 P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43426b5bc874e8482e7b3205a9ba970, State: Initialized, Role: FOLLOWER
I20240502 01:52:52.054338 14964 consensus_queue.cc:260] T c0620ccc47634258a7962d91cc7bb060 P d43426b5bc874e8482e7b3205a9ba970 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } }
I20240502 01:52:52.054471 14964 raft_consensus.cc:388] T c0620ccc47634258a7962d91cc7bb060 P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:52.054564 14964 raft_consensus.cc:482] T c0620ccc47634258a7962d91cc7bb060 P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:52.054646 14964 raft_consensus.cc:3037] T c0620ccc47634258a7962d91cc7bb060 P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:52.056481 14964 raft_consensus.cc:504] T c0620ccc47634258a7962d91cc7bb060 P d43426b5bc874e8482e7b3205a9ba970 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } }
I20240502 01:52:52.056679 14964 leader_election.cc:302] T c0620ccc47634258a7962d91cc7bb060 P d43426b5bc874e8482e7b3205a9ba970 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d43426b5bc874e8482e7b3205a9ba970; no voters:
I20240502 01:52:52.056865 14964 leader_election.cc:288] T c0620ccc47634258a7962d91cc7bb060 P d43426b5bc874e8482e7b3205a9ba970 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:52.056912 14861 raft_consensus.cc:2781] T c0620ccc47634258a7962d91cc7bb060 P d43426b5bc874e8482e7b3205a9ba970 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:52.057137 14861 raft_consensus.cc:686] T c0620ccc47634258a7962d91cc7bb060 P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Becoming Leader. State: Replica: d43426b5bc874e8482e7b3205a9ba970, State: Running, Role: LEADER
I20240502 01:52:52.057348 14964 ts_tablet_manager.cc:1398] T c0620ccc47634258a7962d91cc7bb060 P d43426b5bc874e8482e7b3205a9ba970: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240502 01:52:52.057401 14861 consensus_queue.cc:237] T c0620ccc47634258a7962d91cc7bb060 P d43426b5bc874e8482e7b3205a9ba970 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } }
I20240502 01:52:52.059192 13832 catalog_manager.cc:5484] T c0620ccc47634258a7962d91cc7bb060 P d43426b5bc874e8482e7b3205a9ba970 reported cstate change: term changed from 0 to 1, leader changed from <none> to d43426b5bc874e8482e7b3205a9ba970 (127.11.42.65). New cstate: current_term: 1 leader_uuid: "d43426b5bc874e8482e7b3205a9ba970" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } health_report { overall_health: HEALTHY } } }
W20240502 01:52:52.369725 14974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:52.370005 14974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:52.377110 14974 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 110.351 ms
time per row: 0.0551755 ms
I20240502 01:52:52.620514 13834 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43710:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:52:52.628494 14090 tablet_service.cc:1467] Processing CreateTablet for tablet 5bc0ff324caa436196f22d90360c9247 (DEFAULT_TABLE table=rebalance_test_table_3 [id=fb174a9542bb4a20bf8666a08c8a114e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:52.628636 14089 tablet_service.cc:1467] Processing CreateTablet for tablet 6eb1311f3c054aec9a306922f25c6b89 (DEFAULT_TABLE table=rebalance_test_table_3 [id=fb174a9542bb4a20bf8666a08c8a114e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:52.628973 14090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bc0ff324caa436196f22d90360c9247. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:52.628976 13956 tablet_service.cc:1467] Processing CreateTablet for tablet d22aa9aee27e43a79dd19934de05c43c (DEFAULT_TABLE table=rebalance_test_table_3 [id=fb174a9542bb4a20bf8666a08c8a114e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:52.629305 14089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eb1311f3c054aec9a306922f25c6b89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:52.629410 13956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d22aa9aee27e43a79dd19934de05c43c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:52.633527 15015 tablet_bootstrap.cc:492] T d22aa9aee27e43a79dd19934de05c43c P d43426b5bc874e8482e7b3205a9ba970: Bootstrap starting.
I20240502 01:52:52.634835 15015 tablet_bootstrap.cc:654] T d22aa9aee27e43a79dd19934de05c43c P d43426b5bc874e8482e7b3205a9ba970: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:52.636173 15016 tablet_bootstrap.cc:492] T 5bc0ff324caa436196f22d90360c9247 P 1478241a03934d029dc7030480423cab: Bootstrap starting.
I20240502 01:52:52.636853 15015 tablet_bootstrap.cc:492] T d22aa9aee27e43a79dd19934de05c43c P d43426b5bc874e8482e7b3205a9ba970: No bootstrap required, opened a new log
I20240502 01:52:52.637001 15015 ts_tablet_manager.cc:1367] T d22aa9aee27e43a79dd19934de05c43c P d43426b5bc874e8482e7b3205a9ba970: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:52:52.637670 15016 tablet_bootstrap.cc:654] T 5bc0ff324caa436196f22d90360c9247 P 1478241a03934d029dc7030480423cab: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:52.637722 15015 raft_consensus.cc:348] T d22aa9aee27e43a79dd19934de05c43c P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } }
I20240502 01:52:52.637898 15015 raft_consensus.cc:374] T d22aa9aee27e43a79dd19934de05c43c P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:52.637992 15015 raft_consensus.cc:729] T d22aa9aee27e43a79dd19934de05c43c P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43426b5bc874e8482e7b3205a9ba970, State: Initialized, Role: FOLLOWER
I20240502 01:52:52.638239 15015 consensus_queue.cc:260] T d22aa9aee27e43a79dd19934de05c43c P d43426b5bc874e8482e7b3205a9ba970 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } }
I20240502 01:52:52.638430 15015 raft_consensus.cc:388] T d22aa9aee27e43a79dd19934de05c43c P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:52.638507 15015 raft_consensus.cc:482] T d22aa9aee27e43a79dd19934de05c43c P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:52.638608 15015 raft_consensus.cc:3037] T d22aa9aee27e43a79dd19934de05c43c P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:52.639448 15016 tablet_bootstrap.cc:492] T 5bc0ff324caa436196f22d90360c9247 P 1478241a03934d029dc7030480423cab: No bootstrap required, opened a new log
I20240502 01:52:52.639580 15016 ts_tablet_manager.cc:1367] T 5bc0ff324caa436196f22d90360c9247 P 1478241a03934d029dc7030480423cab: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 01:52:52.640250 15016 raft_consensus.cc:348] T 5bc0ff324caa436196f22d90360c9247 P 1478241a03934d029dc7030480423cab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } }
I20240502 01:52:52.640416 15016 raft_consensus.cc:374] T 5bc0ff324caa436196f22d90360c9247 P 1478241a03934d029dc7030480423cab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:52.640498 15016 raft_consensus.cc:729] T 5bc0ff324caa436196f22d90360c9247 P 1478241a03934d029dc7030480423cab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1478241a03934d029dc7030480423cab, State: Initialized, Role: FOLLOWER
I20240502 01:52:52.640702 15016 consensus_queue.cc:260] T 5bc0ff324caa436196f22d90360c9247 P 1478241a03934d029dc7030480423cab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } }
I20240502 01:52:52.640890 15016 raft_consensus.cc:388] T 5bc0ff324caa436196f22d90360c9247 P 1478241a03934d029dc7030480423cab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:52.640980 15016 raft_consensus.cc:482] T 5bc0ff324caa436196f22d90360c9247 P 1478241a03934d029dc7030480423cab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:52.640951 15015 raft_consensus.cc:504] T d22aa9aee27e43a79dd19934de05c43c P d43426b5bc874e8482e7b3205a9ba970 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } }
I20240502 01:52:52.641119 15016 raft_consensus.cc:3037] T 5bc0ff324caa436196f22d90360c9247 P 1478241a03934d029dc7030480423cab [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:52.641196 15015 leader_election.cc:302] T d22aa9aee27e43a79dd19934de05c43c P d43426b5bc874e8482e7b3205a9ba970 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d43426b5bc874e8482e7b3205a9ba970; no voters:
I20240502 01:52:52.641386 15015 leader_election.cc:288] T d22aa9aee27e43a79dd19934de05c43c P d43426b5bc874e8482e7b3205a9ba970 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:52.641464 14970 raft_consensus.cc:2781] T d22aa9aee27e43a79dd19934de05c43c P d43426b5bc874e8482e7b3205a9ba970 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:52.642099 15015 ts_tablet_manager.cc:1398] T d22aa9aee27e43a79dd19934de05c43c P d43426b5bc874e8482e7b3205a9ba970: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20240502 01:52:52.642124 14970 raft_consensus.cc:686] T d22aa9aee27e43a79dd19934de05c43c P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Becoming Leader. State: Replica: d43426b5bc874e8482e7b3205a9ba970, State: Running, Role: LEADER
I20240502 01:52:52.642398 14970 consensus_queue.cc:237] T d22aa9aee27e43a79dd19934de05c43c P d43426b5bc874e8482e7b3205a9ba970 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } }
I20240502 01:52:52.643268 15016 raft_consensus.cc:504] T 5bc0ff324caa436196f22d90360c9247 P 1478241a03934d029dc7030480423cab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } }
I20240502 01:52:52.643505 15016 leader_election.cc:302] T 5bc0ff324caa436196f22d90360c9247 P 1478241a03934d029dc7030480423cab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1478241a03934d029dc7030480423cab; no voters:
I20240502 01:52:52.643692 15016 leader_election.cc:288] T 5bc0ff324caa436196f22d90360c9247 P 1478241a03934d029dc7030480423cab [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:52.643805 14914 raft_consensus.cc:2781] T 5bc0ff324caa436196f22d90360c9247 P 1478241a03934d029dc7030480423cab [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:52.644783 13834 catalog_manager.cc:5484] T d22aa9aee27e43a79dd19934de05c43c P d43426b5bc874e8482e7b3205a9ba970 reported cstate change: term changed from 0 to 1, leader changed from <none> to d43426b5bc874e8482e7b3205a9ba970 (127.11.42.65). New cstate: current_term: 1 leader_uuid: "d43426b5bc874e8482e7b3205a9ba970" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:52.645890 14914 raft_consensus.cc:686] T 5bc0ff324caa436196f22d90360c9247 P 1478241a03934d029dc7030480423cab [term 1 LEADER]: Becoming Leader. State: Replica: 1478241a03934d029dc7030480423cab, State: Running, Role: LEADER
I20240502 01:52:52.646140 14914 consensus_queue.cc:237] T 5bc0ff324caa436196f22d90360c9247 P 1478241a03934d029dc7030480423cab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } }
I20240502 01:52:52.647545 15016 ts_tablet_manager.cc:1398] T 5bc0ff324caa436196f22d90360c9247 P 1478241a03934d029dc7030480423cab: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240502 01:52:52.647773 15016 tablet_bootstrap.cc:492] T 6eb1311f3c054aec9a306922f25c6b89 P 1478241a03934d029dc7030480423cab: Bootstrap starting.
I20240502 01:52:52.648626 13834 catalog_manager.cc:5484] T 5bc0ff324caa436196f22d90360c9247 P 1478241a03934d029dc7030480423cab reported cstate change: term changed from 0 to 1, leader changed from <none> to 1478241a03934d029dc7030480423cab (127.11.42.66). New cstate: current_term: 1 leader_uuid: "1478241a03934d029dc7030480423cab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:52.650303 15016 tablet_bootstrap.cc:654] T 6eb1311f3c054aec9a306922f25c6b89 P 1478241a03934d029dc7030480423cab: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:52.651849 15016 tablet_bootstrap.cc:492] T 6eb1311f3c054aec9a306922f25c6b89 P 1478241a03934d029dc7030480423cab: No bootstrap required, opened a new log
I20240502 01:52:52.651978 15016 ts_tablet_manager.cc:1367] T 6eb1311f3c054aec9a306922f25c6b89 P 1478241a03934d029dc7030480423cab: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:52:52.652489 15016 raft_consensus.cc:348] T 6eb1311f3c054aec9a306922f25c6b89 P 1478241a03934d029dc7030480423cab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } }
I20240502 01:52:52.652631 15016 raft_consensus.cc:374] T 6eb1311f3c054aec9a306922f25c6b89 P 1478241a03934d029dc7030480423cab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:52.652695 15016 raft_consensus.cc:729] T 6eb1311f3c054aec9a306922f25c6b89 P 1478241a03934d029dc7030480423cab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1478241a03934d029dc7030480423cab, State: Initialized, Role: FOLLOWER
I20240502 01:52:52.652909 15016 consensus_queue.cc:260] T 6eb1311f3c054aec9a306922f25c6b89 P 1478241a03934d029dc7030480423cab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } }
I20240502 01:52:52.653107 15016 raft_consensus.cc:388] T 6eb1311f3c054aec9a306922f25c6b89 P 1478241a03934d029dc7030480423cab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:52.653209 15016 raft_consensus.cc:482] T 6eb1311f3c054aec9a306922f25c6b89 P 1478241a03934d029dc7030480423cab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:52.653291 15016 raft_consensus.cc:3037] T 6eb1311f3c054aec9a306922f25c6b89 P 1478241a03934d029dc7030480423cab [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:52.655244 15016 raft_consensus.cc:504] T 6eb1311f3c054aec9a306922f25c6b89 P 1478241a03934d029dc7030480423cab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } }
I20240502 01:52:52.655437 15016 leader_election.cc:302] T 6eb1311f3c054aec9a306922f25c6b89 P 1478241a03934d029dc7030480423cab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1478241a03934d029dc7030480423cab; no voters:
I20240502 01:52:52.655622 15016 leader_election.cc:288] T 6eb1311f3c054aec9a306922f25c6b89 P 1478241a03934d029dc7030480423cab [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:52.655694 15022 raft_consensus.cc:2781] T 6eb1311f3c054aec9a306922f25c6b89 P 1478241a03934d029dc7030480423cab [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:52.655925 15022 raft_consensus.cc:686] T 6eb1311f3c054aec9a306922f25c6b89 P 1478241a03934d029dc7030480423cab [term 1 LEADER]: Becoming Leader. State: Replica: 1478241a03934d029dc7030480423cab, State: Running, Role: LEADER
I20240502 01:52:52.656018 15016 ts_tablet_manager.cc:1398] T 6eb1311f3c054aec9a306922f25c6b89 P 1478241a03934d029dc7030480423cab: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 01:52:52.656186 15022 consensus_queue.cc:237] T 6eb1311f3c054aec9a306922f25c6b89 P 1478241a03934d029dc7030480423cab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } }
I20240502 01:52:52.658004 13834 catalog_manager.cc:5484] T 6eb1311f3c054aec9a306922f25c6b89 P 1478241a03934d029dc7030480423cab reported cstate change: term changed from 0 to 1, leader changed from <none> to 1478241a03934d029dc7030480423cab (127.11.42.66). New cstate: current_term: 1 leader_uuid: "1478241a03934d029dc7030480423cab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } health_report { overall_health: HEALTHY } } }
W20240502 01:52:52.971654 15027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:52.971940 15027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:52.979118 15027 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 106.811 ms
time per row: 0.0534055 ms
I20240502 01:52:53.223212 13832 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43710:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:52:53.232508 13956 tablet_service.cc:1467] Processing CreateTablet for tablet 6ed541d2e5ba4bbab19869c10116142e (DEFAULT_TABLE table=rebalance_test_table_4 [id=0c0532d94ab24438b74e82726a95b9e8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:53.232664 14090 tablet_service.cc:1467] Processing CreateTablet for tablet 954145d4672146ba8d26a25540de0325 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0c0532d94ab24438b74e82726a95b9e8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:53.232967 13956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ed541d2e5ba4bbab19869c10116142e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:53.233126 14090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 954145d4672146ba8d26a25540de0325. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:53.237663 13957 tablet_service.cc:1467] Processing CreateTablet for tablet 50496c7daf804faa8d7bfaf128839e2c (DEFAULT_TABLE table=rebalance_test_table_4 [id=0c0532d94ab24438b74e82726a95b9e8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:53.238001 13957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50496c7daf804faa8d7bfaf128839e2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:53.243257 15066 tablet_bootstrap.cc:492] T 6ed541d2e5ba4bbab19869c10116142e P d43426b5bc874e8482e7b3205a9ba970: Bootstrap starting.
I20240502 01:52:53.243582 15067 tablet_bootstrap.cc:492] T 954145d4672146ba8d26a25540de0325 P 1478241a03934d029dc7030480423cab: Bootstrap starting.
I20240502 01:52:53.245052 15067 tablet_bootstrap.cc:654] T 954145d4672146ba8d26a25540de0325 P 1478241a03934d029dc7030480423cab: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:53.245332 15066 tablet_bootstrap.cc:654] T 6ed541d2e5ba4bbab19869c10116142e P d43426b5bc874e8482e7b3205a9ba970: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:53.247222 15066 tablet_bootstrap.cc:492] T 6ed541d2e5ba4bbab19869c10116142e P d43426b5bc874e8482e7b3205a9ba970: No bootstrap required, opened a new log
I20240502 01:52:53.247359 15066 ts_tablet_manager.cc:1367] T 6ed541d2e5ba4bbab19869c10116142e P d43426b5bc874e8482e7b3205a9ba970: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:52:53.248031 15066 raft_consensus.cc:348] T 6ed541d2e5ba4bbab19869c10116142e P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } }
I20240502 01:52:53.248198 15066 raft_consensus.cc:374] T 6ed541d2e5ba4bbab19869c10116142e P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:53.248299 15066 raft_consensus.cc:729] T 6ed541d2e5ba4bbab19869c10116142e P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43426b5bc874e8482e7b3205a9ba970, State: Initialized, Role: FOLLOWER
I20240502 01:52:53.248509 15066 consensus_queue.cc:260] T 6ed541d2e5ba4bbab19869c10116142e P d43426b5bc874e8482e7b3205a9ba970 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } }
I20240502 01:52:53.248678 15066 raft_consensus.cc:388] T 6ed541d2e5ba4bbab19869c10116142e P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:53.248744 15066 raft_consensus.cc:482] T 6ed541d2e5ba4bbab19869c10116142e P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:53.248836 15066 raft_consensus.cc:3037] T 6ed541d2e5ba4bbab19869c10116142e P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:53.250030 15067 tablet_bootstrap.cc:492] T 954145d4672146ba8d26a25540de0325 P 1478241a03934d029dc7030480423cab: No bootstrap required, opened a new log
I20240502 01:52:53.250452 15067 ts_tablet_manager.cc:1367] T 954145d4672146ba8d26a25540de0325 P 1478241a03934d029dc7030480423cab: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240502 01:52:53.251137 15067 raft_consensus.cc:348] T 954145d4672146ba8d26a25540de0325 P 1478241a03934d029dc7030480423cab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } }
I20240502 01:52:53.251282 15067 raft_consensus.cc:374] T 954145d4672146ba8d26a25540de0325 P 1478241a03934d029dc7030480423cab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:53.251314 15066 raft_consensus.cc:504] T 6ed541d2e5ba4bbab19869c10116142e P d43426b5bc874e8482e7b3205a9ba970 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } }
I20240502 01:52:53.251384 15067 raft_consensus.cc:729] T 954145d4672146ba8d26a25540de0325 P 1478241a03934d029dc7030480423cab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1478241a03934d029dc7030480423cab, State: Initialized, Role: FOLLOWER
I20240502 01:52:53.251554 15066 leader_election.cc:302] T 6ed541d2e5ba4bbab19869c10116142e P d43426b5bc874e8482e7b3205a9ba970 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d43426b5bc874e8482e7b3205a9ba970; no voters:
I20240502 01:52:53.251616 15067 consensus_queue.cc:260] T 954145d4672146ba8d26a25540de0325 P 1478241a03934d029dc7030480423cab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } }
I20240502 01:52:53.251744 15066 leader_election.cc:288] T 6ed541d2e5ba4bbab19869c10116142e P d43426b5bc874e8482e7b3205a9ba970 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:53.251823 14970 raft_consensus.cc:2781] T 6ed541d2e5ba4bbab19869c10116142e P d43426b5bc874e8482e7b3205a9ba970 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:53.251857 15067 raft_consensus.cc:388] T 954145d4672146ba8d26a25540de0325 P 1478241a03934d029dc7030480423cab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:53.251935 15067 raft_consensus.cc:482] T 954145d4672146ba8d26a25540de0325 P 1478241a03934d029dc7030480423cab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:53.252070 15067 raft_consensus.cc:3037] T 954145d4672146ba8d26a25540de0325 P 1478241a03934d029dc7030480423cab [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:53.252219 14970 raft_consensus.cc:686] T 6ed541d2e5ba4bbab19869c10116142e P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Becoming Leader. State: Replica: d43426b5bc874e8482e7b3205a9ba970, State: Running, Role: LEADER
I20240502 01:52:53.252449 14970 consensus_queue.cc:237] T 6ed541d2e5ba4bbab19869c10116142e P d43426b5bc874e8482e7b3205a9ba970 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } }
I20240502 01:52:53.254791 13832 catalog_manager.cc:5484] T 6ed541d2e5ba4bbab19869c10116142e P d43426b5bc874e8482e7b3205a9ba970 reported cstate change: term changed from 0 to 1, leader changed from <none> to d43426b5bc874e8482e7b3205a9ba970 (127.11.42.65). New cstate: current_term: 1 leader_uuid: "d43426b5bc874e8482e7b3205a9ba970" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:53.255581 15067 raft_consensus.cc:504] T 954145d4672146ba8d26a25540de0325 P 1478241a03934d029dc7030480423cab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } }
I20240502 01:52:53.255884 15067 leader_election.cc:302] T 954145d4672146ba8d26a25540de0325 P 1478241a03934d029dc7030480423cab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1478241a03934d029dc7030480423cab; no voters:
I20240502 01:52:53.257737 14914 raft_consensus.cc:2781] T 954145d4672146ba8d26a25540de0325 P 1478241a03934d029dc7030480423cab [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:53.257890 14914 raft_consensus.cc:686] T 954145d4672146ba8d26a25540de0325 P 1478241a03934d029dc7030480423cab [term 1 LEADER]: Becoming Leader. State: Replica: 1478241a03934d029dc7030480423cab, State: Running, Role: LEADER
I20240502 01:52:53.258086 14914 consensus_queue.cc:237] T 954145d4672146ba8d26a25540de0325 P 1478241a03934d029dc7030480423cab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } }
I20240502 01:52:53.258734 15067 leader_election.cc:288] T 954145d4672146ba8d26a25540de0325 P 1478241a03934d029dc7030480423cab [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:53.258961 15066 ts_tablet_manager.cc:1398] T 6ed541d2e5ba4bbab19869c10116142e P d43426b5bc874e8482e7b3205a9ba970: Time spent starting tablet: real 0.011s user 0.001s sys 0.008s
I20240502 01:52:53.259172 15067 ts_tablet_manager.cc:1398] T 954145d4672146ba8d26a25540de0325 P 1478241a03934d029dc7030480423cab: Time spent starting tablet: real 0.009s user 0.001s sys 0.003s
I20240502 01:52:53.259253 15066 tablet_bootstrap.cc:492] T 50496c7daf804faa8d7bfaf128839e2c P d43426b5bc874e8482e7b3205a9ba970: Bootstrap starting.
I20240502 01:52:53.260053 13832 catalog_manager.cc:5484] T 954145d4672146ba8d26a25540de0325 P 1478241a03934d029dc7030480423cab reported cstate change: term changed from 0 to 1, leader changed from <none> to 1478241a03934d029dc7030480423cab (127.11.42.66). New cstate: current_term: 1 leader_uuid: "1478241a03934d029dc7030480423cab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:53.260581 15066 tablet_bootstrap.cc:654] T 50496c7daf804faa8d7bfaf128839e2c P d43426b5bc874e8482e7b3205a9ba970: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:53.263655 15066 tablet_bootstrap.cc:492] T 50496c7daf804faa8d7bfaf128839e2c P d43426b5bc874e8482e7b3205a9ba970: No bootstrap required, opened a new log
I20240502 01:52:53.263828 15066 ts_tablet_manager.cc:1367] T 50496c7daf804faa8d7bfaf128839e2c P d43426b5bc874e8482e7b3205a9ba970: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240502 01:52:53.264575 15066 raft_consensus.cc:348] T 50496c7daf804faa8d7bfaf128839e2c P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } }
I20240502 01:52:53.264739 15066 raft_consensus.cc:374] T 50496c7daf804faa8d7bfaf128839e2c P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:53.264853 15066 raft_consensus.cc:729] T 50496c7daf804faa8d7bfaf128839e2c P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43426b5bc874e8482e7b3205a9ba970, State: Initialized, Role: FOLLOWER
I20240502 01:52:53.265091 15066 consensus_queue.cc:260] T 50496c7daf804faa8d7bfaf128839e2c P d43426b5bc874e8482e7b3205a9ba970 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } }
I20240502 01:52:53.265221 15066 raft_consensus.cc:388] T 50496c7daf804faa8d7bfaf128839e2c P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:53.265337 15066 raft_consensus.cc:482] T 50496c7daf804faa8d7bfaf128839e2c P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:53.265435 15066 raft_consensus.cc:3037] T 50496c7daf804faa8d7bfaf128839e2c P d43426b5bc874e8482e7b3205a9ba970 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:53.267392 15066 raft_consensus.cc:504] T 50496c7daf804faa8d7bfaf128839e2c P d43426b5bc874e8482e7b3205a9ba970 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } }
I20240502 01:52:53.267568 15066 leader_election.cc:302] T 50496c7daf804faa8d7bfaf128839e2c P d43426b5bc874e8482e7b3205a9ba970 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d43426b5bc874e8482e7b3205a9ba970; no voters:
I20240502 01:52:53.267710 15066 leader_election.cc:288] T 50496c7daf804faa8d7bfaf128839e2c P d43426b5bc874e8482e7b3205a9ba970 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:53.267807 15072 raft_consensus.cc:2781] T 50496c7daf804faa8d7bfaf128839e2c P d43426b5bc874e8482e7b3205a9ba970 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:53.268005 15072 raft_consensus.cc:686] T 50496c7daf804faa8d7bfaf128839e2c P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Becoming Leader. State: Replica: d43426b5bc874e8482e7b3205a9ba970, State: Running, Role: LEADER
I20240502 01:52:53.268136 15066 ts_tablet_manager.cc:1398] T 50496c7daf804faa8d7bfaf128839e2c P d43426b5bc874e8482e7b3205a9ba970: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:52:53.268311 15072 consensus_queue.cc:237] T 50496c7daf804faa8d7bfaf128839e2c P d43426b5bc874e8482e7b3205a9ba970 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } }
I20240502 01:52:53.270347 13832 catalog_manager.cc:5484] T 50496c7daf804faa8d7bfaf128839e2c P d43426b5bc874e8482e7b3205a9ba970 reported cstate change: term changed from 0 to 1, leader changed from <none> to d43426b5bc874e8482e7b3205a9ba970 (127.11.42.65). New cstate: current_term: 1 leader_uuid: "d43426b5bc874e8482e7b3205a9ba970" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } health_report { overall_health: HEALTHY } } }
W20240502 01:52:53.581961 15076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:53.582245 15076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:53.589286 15076 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 118.815 ms
time per row: 0.0594076 ms
I20240502 01:52:53.838953 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.67:37505
--local_ip_for_outbound_sockets=127.11.42.67
--tserver_master_addrs=127.11.42.126:40815
--webserver_port=33451
--webserver_interface=127.11.42.67
--builtin_ntp_servers=127.11.42.84:45633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 01:52:54.146332 15114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:54.146591 15114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:54.146714 15114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:54.153895 15114 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:54.154014 15114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:54.154151 15114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.67
I20240502 01:52:54.162698 15114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:45633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.67:37505
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.11.42.67
--webserver_port=33451
--tserver_master_addrs=127.11.42.126:40815
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.67
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:54.163906 15114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:54.165604 15114 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:54.176762 15119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:54.177269 15120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:54.177660 15122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:54.178098 15114 server_base.cc:1034] running on GCE node
I20240502 01:52:54.178624 15114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:54.179458 15114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:54.180672 15114 hybrid_clock.cc:648] HybridClock initialized: now 1714614774180630 us; error 30 us; skew 500 ppm
I20240502 01:52:54.180977 15114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:54.183077 15114 webserver.cc:460] Webserver started at http://127.11.42.67:33451/ using document root <none> and password file <none>
I20240502 01:52:54.183729 15114 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:54.183852 15114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:54.188638 15114 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20240502 01:52:54.191346 15128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:54.192600 15114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240502 01:52:54.192745 15114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "020e5f8ebb004e2ab6fddaece6663ece"
format_stamp: "Formatted at 2024-05-02 01:52:44 on dist-test-slave-xvk5"
I20240502 01:52:54.193552 15114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:54.210477 15114 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:54.211354 15114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:54.211642 15114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:54.212644 15114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:54.214407 15114 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:54.214493 15114 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:54.214602 15114 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:54.214658 15114 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:54.248113 15114 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.67:37505
I20240502 01:52:54.248155 15241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.67:37505 every 8 connection(s)
I20240502 01:52:54.249471 15114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 01:52:54.253696 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 15114
I20240502 01:52:54.254245 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.68:41779
--local_ip_for_outbound_sockets=127.11.42.68
--tserver_master_addrs=127.11.42.126:40815
--webserver_port=36969
--webserver_interface=127.11.42.68
--builtin_ntp_servers=127.11.42.84:45633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:52:54.262086 15242 heartbeater.cc:347] Connected to a master server at 127.11.42.126:40815
I20240502 01:52:54.262403 15242 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:54.263306 15242 heartbeater.cc:511] Master 127.11.42.126:40815 requested a full tablet report, sending...
I20240502 01:52:54.264941 13834 ts_manager.cc:194] Re-registered known tserver with Master: 020e5f8ebb004e2ab6fddaece6663ece (127.11.42.67:37505)
I20240502 01:52:54.266033 13834 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.67:54329
W20240502 01:52:54.564882 15246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:54.565155 15246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:54.565253 15246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:54.572319 15246 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:54.572427 15246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:54.572566 15246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.68
I20240502 01:52:54.581136 15246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:45633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.68:41779
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.11.42.68
--webserver_port=36969
--tserver_master_addrs=127.11.42.126:40815
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.68
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:54.582345 15246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:54.583981 15246 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:54.595000 15251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:54.595402 15254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:54.595835 15252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:54.596361 15246 server_base.cc:1034] running on GCE node
I20240502 01:52:54.596880 15246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:54.597793 15246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:54.598985 15246 hybrid_clock.cc:648] HybridClock initialized: now 1714614774598941 us; error 35 us; skew 500 ppm
I20240502 01:52:54.599288 15246 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:54.601334 15246 webserver.cc:460] Webserver started at http://127.11.42.68:36969/ using document root <none> and password file <none>
I20240502 01:52:54.601989 15246 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:54.602108 15246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:54.607024 15246 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 01:52:54.609833 15260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:54.611498 15246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:52:54.611665 15246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "1607e211ee8c48de9fa56e78a3b01418"
format_stamp: "Formatted at 2024-05-02 01:52:44 on dist-test-slave-xvk5"
I20240502 01:52:54.612560 15246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:54.635677 15246 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:54.636394 15246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:54.636631 15246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:54.637547 15246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:54.638923 15246 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:54.638993 15246 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:54.639084 15246 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:54.639125 15246 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:54.667865 15246 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.68:41779
I20240502 01:52:54.667922 15373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.68:41779 every 8 connection(s)
I20240502 01:52:54.669204 15246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 01:52:54.671275 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 15246
I20240502 01:52:54.671995 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.69:43981
--local_ip_for_outbound_sockets=127.11.42.69
--tserver_master_addrs=127.11.42.126:40815
--webserver_port=36569
--webserver_interface=127.11.42.69
--builtin_ntp_servers=127.11.42.84:45633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:52:54.682592 15374 heartbeater.cc:347] Connected to a master server at 127.11.42.126:40815
I20240502 01:52:54.682916 15374 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:54.683933 15374 heartbeater.cc:511] Master 127.11.42.126:40815 requested a full tablet report, sending...
I20240502 01:52:54.685510 13834 ts_manager.cc:194] Re-registered known tserver with Master: 1607e211ee8c48de9fa56e78a3b01418 (127.11.42.68:41779)
I20240502 01:52:54.686308 13834 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.68:54935
W20240502 01:52:54.998080 15377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:54.998380 15377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:54.998512 15377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:55.005661 15377 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:55.005764 15377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:55.005909 15377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.69
I20240502 01:52:55.014393 15377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:45633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.69:43981
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.11.42.69
--webserver_port=36569
--tserver_master_addrs=127.11.42.126:40815
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.69
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:55.015571 15377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:55.017246 15377 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:55.028210 15382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:55.028656 15383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:55.029078 15385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:55.029487 15377 server_base.cc:1034] running on GCE node
I20240502 01:52:55.029978 15377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:55.030774 15377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:55.031966 15377 hybrid_clock.cc:648] HybridClock initialized: now 1714614775031919 us; error 43 us; skew 500 ppm
I20240502 01:52:55.032263 15377 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:55.034245 15377 webserver.cc:460] Webserver started at http://127.11.42.69:36569/ using document root <none> and password file <none>
I20240502 01:52:55.034883 15377 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:55.035003 15377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:55.039644 15377 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 01:52:55.042114 15391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:55.043264 15377 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20240502 01:52:55.043414 15377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "95e7c90cdc634d64a66dad6805e1680c"
format_stamp: "Formatted at 2024-05-02 01:52:45 on dist-test-slave-xvk5"
I20240502 01:52:55.044142 15377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:55.069646 15377 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:55.070360 15377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:55.070626 15377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:55.071466 15377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:55.072849 15377 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:55.072923 15377 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:55.073064 15377 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:55.073119 15377 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20240502 01:52:55.101500 15377 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.69:43981
I20240502 01:52:55.101560 15503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.69:43981 every 8 connection(s)
I20240502 01:52:55.102869 15377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:52:55.109143 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 15377
I20240502 01:52:55.109865 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.70:45255
--local_ip_for_outbound_sockets=127.11.42.70
--tserver_master_addrs=127.11.42.126:40815
--webserver_port=46747
--webserver_interface=127.11.42.70
--builtin_ntp_servers=127.11.42.84:45633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:52:55.115931 15504 heartbeater.cc:347] Connected to a master server at 127.11.42.126:40815
I20240502 01:52:55.116214 15504 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:55.117043 15504 heartbeater.cc:511] Master 127.11.42.126:40815 requested a full tablet report, sending...
I20240502 01:52:55.118408 13834 ts_manager.cc:194] Re-registered known tserver with Master: 95e7c90cdc634d64a66dad6805e1680c (127.11.42.69:43981)
I20240502 01:52:55.119175 13834 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.69:38215
I20240502 01:52:55.268970 15242 heartbeater.cc:503] Master 127.11.42.126:40815 was elected leader, sending a full tablet report...
W20240502 01:52:55.416041 15508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:55.416316 15508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:55.416451 15508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:55.423674 15508 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:55.423801 15508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:55.423947 15508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.70
I20240502 01:52:55.432451 15508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:45633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.70:45255
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.11.42.70
--webserver_port=46747
--tserver_master_addrs=127.11.42.126:40815
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.70
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:55.433636 15508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:55.435249 15508 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:55.446372 15513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:55.446878 15514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:55.447177 15516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:55.447731 15508 server_base.cc:1034] running on GCE node
I20240502 01:52:55.448292 15508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:55.449141 15508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:55.450354 15508 hybrid_clock.cc:648] HybridClock initialized: now 1714614775450301 us; error 45 us; skew 500 ppm
I20240502 01:52:55.450640 15508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:55.452677 15508 webserver.cc:460] Webserver started at http://127.11.42.70:46747/ using document root <none> and password file <none>
I20240502 01:52:55.453387 15508 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:55.453505 15508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:55.458312 15508 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20240502 01:52:55.460898 15522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:55.462100 15508 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 01:52:55.462249 15508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "2ee638444633463b943c881866708482"
format_stamp: "Formatted at 2024-05-02 01:52:45 on dist-test-slave-xvk5"
I20240502 01:52:55.463042 15508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:55.498447 15508 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:55.499533 15508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:55.499886 15508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:55.501082 15508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:55.503535 15529 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240502 01:52:55.510377 15508 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240502 01:52:55.510494 15508 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20240502 01:52:55.510604 15508 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240502 01:52:55.513397 15508 ts_tablet_manager.cc:590] Registered 1 tablets
I20240502 01:52:55.513490 15508 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s user 0.000s sys 0.003s
I20240502 01:52:55.513798 15529 tablet_bootstrap.cc:492] T 563e1017b8774231bd29a1f8e628b4dc P 2ee638444633463b943c881866708482: Bootstrap starting.
I20240502 01:52:55.530364 15529 log.cc:824] T 563e1017b8774231bd29a1f8e628b4dc P 2ee638444633463b943c881866708482: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:55.534653 15529 tablet_bootstrap.cc:492] T 563e1017b8774231bd29a1f8e628b4dc P 2ee638444633463b943c881866708482: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:52:55.535390 15529 tablet_bootstrap.cc:492] T 563e1017b8774231bd29a1f8e628b4dc P 2ee638444633463b943c881866708482: Bootstrap complete.
I20240502 01:52:55.536376 15529 ts_tablet_manager.cc:1367] T 563e1017b8774231bd29a1f8e628b4dc P 2ee638444633463b943c881866708482: Time spent bootstrapping tablet: real 0.023s user 0.017s sys 0.005s
I20240502 01:52:55.544993 15529 raft_consensus.cc:348] T 563e1017b8774231bd29a1f8e628b4dc P 2ee638444633463b943c881866708482 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ee638444633463b943c881866708482" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 45255 } }
I20240502 01:52:55.545872 15529 raft_consensus.cc:729] T 563e1017b8774231bd29a1f8e628b4dc P 2ee638444633463b943c881866708482 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ee638444633463b943c881866708482, State: Initialized, Role: FOLLOWER
I20240502 01:52:55.546867 15529 consensus_queue.cc:260] T 563e1017b8774231bd29a1f8e628b4dc P 2ee638444633463b943c881866708482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ee638444633463b943c881866708482" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 45255 } }
I20240502 01:52:55.547147 15529 raft_consensus.cc:388] T 563e1017b8774231bd29a1f8e628b4dc P 2ee638444633463b943c881866708482 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:55.547348 15529 raft_consensus.cc:482] T 563e1017b8774231bd29a1f8e628b4dc P 2ee638444633463b943c881866708482 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:55.547580 15529 raft_consensus.cc:3037] T 563e1017b8774231bd29a1f8e628b4dc P 2ee638444633463b943c881866708482 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:55.552971 15529 raft_consensus.cc:504] T 563e1017b8774231bd29a1f8e628b4dc P 2ee638444633463b943c881866708482 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ee638444633463b943c881866708482" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 45255 } }
I20240502 01:52:55.553639 15529 leader_election.cc:302] T 563e1017b8774231bd29a1f8e628b4dc P 2ee638444633463b943c881866708482 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ee638444633463b943c881866708482; no voters:
I20240502 01:52:55.554423 15529 leader_election.cc:288] T 563e1017b8774231bd29a1f8e628b4dc P 2ee638444633463b943c881866708482 [CANDIDATE]: Term 2 election: Requested vote from peers
I20240502 01:52:55.554559 15629 raft_consensus.cc:2781] T 563e1017b8774231bd29a1f8e628b4dc P 2ee638444633463b943c881866708482 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:52:55.558856 15629 raft_consensus.cc:686] T 563e1017b8774231bd29a1f8e628b4dc P 2ee638444633463b943c881866708482 [term 2 LEADER]: Becoming Leader. State: Replica: 2ee638444633463b943c881866708482, State: Running, Role: LEADER
I20240502 01:52:55.559417 15629 consensus_queue.cc:237] T 563e1017b8774231bd29a1f8e628b4dc P 2ee638444633463b943c881866708482 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ee638444633463b943c881866708482" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 45255 } }
I20240502 01:52:55.562517 15529 ts_tablet_manager.cc:1398] T 563e1017b8774231bd29a1f8e628b4dc P 2ee638444633463b943c881866708482: Time spent starting tablet: real 0.026s user 0.017s sys 0.008s
I20240502 01:52:55.566670 15508 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.70:45255
I20240502 01:52:55.566741 15641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.70:45255 every 8 connection(s)
I20240502 01:52:55.567942 15508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 01:52:55.576524 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 15508
I20240502 01:52:55.577208 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.71:37071
--local_ip_for_outbound_sockets=127.11.42.71
--tserver_master_addrs=127.11.42.126:40815
--webserver_port=44441
--webserver_interface=127.11.42.71
--builtin_ntp_servers=127.11.42.84:45633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:52:55.578543 15642 heartbeater.cc:347] Connected to a master server at 127.11.42.126:40815
I20240502 01:52:55.578805 15642 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:55.579543 15642 heartbeater.cc:511] Master 127.11.42.126:40815 requested a full tablet report, sending...
I20240502 01:52:55.581887 13834 ts_manager.cc:194] Re-registered known tserver with Master: 2ee638444633463b943c881866708482 (127.11.42.70:45255)
I20240502 01:52:55.582299 13834 catalog_manager.cc:5484] T 563e1017b8774231bd29a1f8e628b4dc P 2ee638444633463b943c881866708482 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "2ee638444633463b943c881866708482" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ee638444633463b943c881866708482" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 45255 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:55.587244 13834 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.70:35783
I20240502 01:52:55.589731 15642 heartbeater.cc:503] Master 127.11.42.126:40815 was elected leader, sending a full tablet report...
I20240502 01:52:55.689157 15374 heartbeater.cc:503] Master 127.11.42.126:40815 was elected leader, sending a full tablet report...
W20240502 01:52:55.892808 15645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:55.893095 15645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:55.893220 15645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:55.900514 15645 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:55.900635 15645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:55.900779 15645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.71
I20240502 01:52:55.909379 15645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:45633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.71:37071
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.11.42.71
--webserver_port=44441
--tserver_master_addrs=127.11.42.126:40815
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.71
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:55.910640 15645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:55.912323 15645 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:55.923882 15655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:55.924392 15654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:55.924793 15657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:55.925341 15645 server_base.cc:1034] running on GCE node
I20240502 01:52:55.925860 15645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:55.926676 15645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:55.927868 15645 hybrid_clock.cc:648] HybridClock initialized: now 1714614775927818 us; error 48 us; skew 500 ppm
I20240502 01:52:55.928148 15645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:55.930195 15645 webserver.cc:460] Webserver started at http://127.11.42.71:44441/ using document root <none> and password file <none>
I20240502 01:52:55.930851 15645 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:55.930968 15645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:55.935638 15645 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 01:52:55.938124 15663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:55.939244 15645 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:52:55.939396 15645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "39e5e5c1a5e74915800d96f720aee8c4"
format_stamp: "Formatted at 2024-05-02 01:52:46 on dist-test-slave-xvk5"
I20240502 01:52:55.940146 15645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:55.959321 15645 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:55.960026 15645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:55.960274 15645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:55.961165 15645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:55.962605 15645 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:55.962678 15645 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:55.962797 15645 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:55.962846 15645 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:55.992012 15645 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.71:37071
I20240502 01:52:55.992053 15775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.71:37071 every 8 connection(s)
I20240502 01:52:55.993409 15645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 01:52:56.003134 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 15645
I20240502 01:52:56.008963 15776 heartbeater.cc:347] Connected to a master server at 127.11.42.126:40815
I20240502 01:52:56.009305 15776 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:56.010279 15776 heartbeater.cc:511] Master 127.11.42.126:40815 requested a full tablet report, sending...
I20240502 01:52:56.011713 13834 ts_manager.cc:194] Re-registered known tserver with Master: 39e5e5c1a5e74915800d96f720aee8c4 (127.11.42.71:37071)
I20240502 01:52:56.012517 13834 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.71:49957
I20240502 01:52:56.121429 15504 heartbeater.cc:503] Master 127.11.42.126:40815 was elected leader, sending a full tablet report...
I20240502 01:52:56.459758 14090 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:52:56.469386 15176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:56.474900 13957 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:52:56.474911 15711 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:56.475708 15308 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:56.503114 15572 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:52:56.517258 15439 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:56.709142 15176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:56.719357 14090 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:52:56.739382 15308 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:56.752348 15711 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:56.755962 15572 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:52:56.771781 15439 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:56.772279 13957 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:52:56.850009 14111 consensus_queue.cc:237] T 5bc0ff324caa436196f22d90360c9247 P 1478241a03934d029dc7030480423cab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "020e5f8ebb004e2ab6fddaece6663ece" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 37505 } attrs { promote: true } }
I20240502 01:52:56.852708 15866 raft_consensus.cc:2932] T 5bc0ff324caa436196f22d90360c9247 P 1478241a03934d029dc7030480423cab [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 020e5f8ebb004e2ab6fddaece6663ece (127.11.42.67) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "020e5f8ebb004e2ab6fddaece6663ece" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 37505 } attrs { promote: true } } }
I20240502 01:52:56.859040 13834 catalog_manager.cc:5484] T 5bc0ff324caa436196f22d90360c9247 P 1478241a03934d029dc7030480423cab reported cstate change: config changed from index -1 to 4, NON_VOTER 020e5f8ebb004e2ab6fddaece6663ece (127.11.42.67) added. New cstate: current_term: 1 leader_uuid: "1478241a03934d029dc7030480423cab" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "020e5f8ebb004e2ab6fddaece6663ece" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 37505 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:52:56.862560 14043 consensus_peers.cc:485] T 5bc0ff324caa436196f22d90360c9247 P 1478241a03934d029dc7030480423cab -> Peer 020e5f8ebb004e2ab6fddaece6663ece (127.11.42.67:37505): Couldn't send request to peer 020e5f8ebb004e2ab6fddaece6663ece. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5bc0ff324caa436196f22d90360c9247. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:52:56.871301 13977 consensus_queue.cc:237] T c0620ccc47634258a7962d91cc7bb060 P d43426b5bc874e8482e7b3205a9ba970 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "1607e211ee8c48de9fa56e78a3b01418" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 41779 } attrs { promote: true } }
I20240502 01:52:56.873958 15880 raft_consensus.cc:2932] T c0620ccc47634258a7962d91cc7bb060 P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1607e211ee8c48de9fa56e78a3b01418 (127.11.42.68) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "1607e211ee8c48de9fa56e78a3b01418" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 41779 } attrs { promote: true } } }
I20240502 01:52:56.879798 13834 catalog_manager.cc:5484] T c0620ccc47634258a7962d91cc7bb060 P d43426b5bc874e8482e7b3205a9ba970 reported cstate change: config changed from index -1 to 4, NON_VOTER 1607e211ee8c48de9fa56e78a3b01418 (127.11.42.68) added. New cstate: current_term: 1 leader_uuid: "d43426b5bc874e8482e7b3205a9ba970" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1607e211ee8c48de9fa56e78a3b01418" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 41779 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:52:56.883651 13911 consensus_peers.cc:485] T c0620ccc47634258a7962d91cc7bb060 P d43426b5bc874e8482e7b3205a9ba970 -> Peer 1607e211ee8c48de9fa56e78a3b01418 (127.11.42.68:41779): Couldn't send request to peer 1607e211ee8c48de9fa56e78a3b01418. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0620ccc47634258a7962d91cc7bb060. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:52:56.890519 14111 consensus_queue.cc:237] T aa387d2b1dcc42ddb34005f08dde2d8d P 1478241a03934d029dc7030480423cab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "95e7c90cdc634d64a66dad6805e1680c" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 43981 } attrs { promote: true } }
I20240502 01:52:56.892287 15866 raft_consensus.cc:2932] T aa387d2b1dcc42ddb34005f08dde2d8d P 1478241a03934d029dc7030480423cab [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 95e7c90cdc634d64a66dad6805e1680c (127.11.42.69) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "95e7c90cdc634d64a66dad6805e1680c" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 43981 } attrs { promote: true } } }
I20240502 01:52:56.897917 13834 catalog_manager.cc:5484] T aa387d2b1dcc42ddb34005f08dde2d8d P 1478241a03934d029dc7030480423cab reported cstate change: config changed from index -1 to 4, NON_VOTER 95e7c90cdc634d64a66dad6805e1680c (127.11.42.69) added. New cstate: current_term: 1 leader_uuid: "1478241a03934d029dc7030480423cab" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95e7c90cdc634d64a66dad6805e1680c" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 43981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:52:56.901940 14043 consensus_peers.cc:485] T aa387d2b1dcc42ddb34005f08dde2d8d P 1478241a03934d029dc7030480423cab -> Peer 95e7c90cdc634d64a66dad6805e1680c (127.11.42.69:43981): Couldn't send request to peer 95e7c90cdc634d64a66dad6805e1680c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aa387d2b1dcc42ddb34005f08dde2d8d. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:52:56.913378 13977 consensus_queue.cc:237] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P d43426b5bc874e8482e7b3205a9ba970 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "39e5e5c1a5e74915800d96f720aee8c4" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 37071 } attrs { promote: true } }
I20240502 01:52:56.914983 15880 raft_consensus.cc:2932] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 39e5e5c1a5e74915800d96f720aee8c4 (127.11.42.71) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "39e5e5c1a5e74915800d96f720aee8c4" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 37071 } attrs { promote: true } } }
I20240502 01:52:56.919548 13834 catalog_manager.cc:5484] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P d43426b5bc874e8482e7b3205a9ba970 reported cstate change: config changed from index -1 to 4, NON_VOTER 39e5e5c1a5e74915800d96f720aee8c4 (127.11.42.71) added. New cstate: current_term: 1 leader_uuid: "d43426b5bc874e8482e7b3205a9ba970" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39e5e5c1a5e74915800d96f720aee8c4" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 37071 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:52:56.922596 13910 consensus_peers.cc:485] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P d43426b5bc874e8482e7b3205a9ba970 -> Peer 39e5e5c1a5e74915800d96f720aee8c4 (127.11.42.71:37071): Couldn't send request to peer 39e5e5c1a5e74915800d96f720aee8c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 67bc0cd4c3b94782ac1d9f150fcdbd03. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:52:56.931732 14111 consensus_queue.cc:237] T f9f4aab07f984168bdfec24f908c124d P 1478241a03934d029dc7030480423cab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "2ee638444633463b943c881866708482" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 45255 } attrs { promote: true } }
I20240502 01:52:56.933945 15867 raft_consensus.cc:2932] T f9f4aab07f984168bdfec24f908c124d P 1478241a03934d029dc7030480423cab [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2ee638444633463b943c881866708482 (127.11.42.70) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "2ee638444633463b943c881866708482" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 45255 } attrs { promote: true } } }
I20240502 01:52:56.939069 13834 catalog_manager.cc:5484] T f9f4aab07f984168bdfec24f908c124d P 1478241a03934d029dc7030480423cab reported cstate change: config changed from index -1 to 4, NON_VOTER 2ee638444633463b943c881866708482 (127.11.42.70) added. New cstate: current_term: 1 leader_uuid: "1478241a03934d029dc7030480423cab" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ee638444633463b943c881866708482" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 45255 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:52:56.941771 14043 consensus_peers.cc:485] T f9f4aab07f984168bdfec24f908c124d P 1478241a03934d029dc7030480423cab -> Peer 2ee638444633463b943c881866708482 (127.11.42.70:45255): Couldn't send request to peer 2ee638444633463b943c881866708482. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f9f4aab07f984168bdfec24f908c124d. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:52:56.954200 13977 consensus_queue.cc:237] T 0eb60206b8cb40d3b30cdc1f47d67f94 P d43426b5bc874e8482e7b3205a9ba970 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "020e5f8ebb004e2ab6fddaece6663ece" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 37505 } attrs { promote: true } }
I20240502 01:52:56.956189 15879 raft_consensus.cc:2932] T 0eb60206b8cb40d3b30cdc1f47d67f94 P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 020e5f8ebb004e2ab6fddaece6663ece (127.11.42.67) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "020e5f8ebb004e2ab6fddaece6663ece" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 37505 } attrs { promote: true } } }
I20240502 01:52:56.961241 13834 catalog_manager.cc:5484] T 0eb60206b8cb40d3b30cdc1f47d67f94 P d43426b5bc874e8482e7b3205a9ba970 reported cstate change: config changed from index -1 to 4, NON_VOTER 020e5f8ebb004e2ab6fddaece6663ece (127.11.42.67) added. New cstate: current_term: 1 leader_uuid: "d43426b5bc874e8482e7b3205a9ba970" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "020e5f8ebb004e2ab6fddaece6663ece" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 37505 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:52:56.962842 13910 consensus_peers.cc:485] T 0eb60206b8cb40d3b30cdc1f47d67f94 P d43426b5bc874e8482e7b3205a9ba970 -> Peer 020e5f8ebb004e2ab6fddaece6663ece (127.11.42.67:37505): Couldn't send request to peer 020e5f8ebb004e2ab6fddaece6663ece. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0eb60206b8cb40d3b30cdc1f47d67f94. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:52:56.972468 14111 consensus_queue.cc:237] T 6eb1311f3c054aec9a306922f25c6b89 P 1478241a03934d029dc7030480423cab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "1607e211ee8c48de9fa56e78a3b01418" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 41779 } attrs { promote: true } }
I20240502 01:52:56.974357 15866 raft_consensus.cc:2932] T 6eb1311f3c054aec9a306922f25c6b89 P 1478241a03934d029dc7030480423cab [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1607e211ee8c48de9fa56e78a3b01418 (127.11.42.68) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "1607e211ee8c48de9fa56e78a3b01418" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 41779 } attrs { promote: true } } }
I20240502 01:52:56.979350 13834 catalog_manager.cc:5484] T 6eb1311f3c054aec9a306922f25c6b89 P 1478241a03934d029dc7030480423cab reported cstate change: config changed from index -1 to 4, NON_VOTER 1607e211ee8c48de9fa56e78a3b01418 (127.11.42.68) added. New cstate: current_term: 1 leader_uuid: "1478241a03934d029dc7030480423cab" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1607e211ee8c48de9fa56e78a3b01418" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 41779 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:52:56.981520 14044 consensus_peers.cc:485] T 6eb1311f3c054aec9a306922f25c6b89 P 1478241a03934d029dc7030480423cab -> Peer 1607e211ee8c48de9fa56e78a3b01418 (127.11.42.68:41779): Couldn't send request to peer 1607e211ee8c48de9fa56e78a3b01418. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6eb1311f3c054aec9a306922f25c6b89. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:52:56.990589 13977 consensus_queue.cc:237] T 50496c7daf804faa8d7bfaf128839e2c P d43426b5bc874e8482e7b3205a9ba970 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "95e7c90cdc634d64a66dad6805e1680c" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 43981 } attrs { promote: true } }
I20240502 01:52:56.992463 15879 raft_consensus.cc:2932] T 50496c7daf804faa8d7bfaf128839e2c P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 95e7c90cdc634d64a66dad6805e1680c (127.11.42.69) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "95e7c90cdc634d64a66dad6805e1680c" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 43981 } attrs { promote: true } } }
I20240502 01:52:56.997920 13834 catalog_manager.cc:5484] T 50496c7daf804faa8d7bfaf128839e2c P d43426b5bc874e8482e7b3205a9ba970 reported cstate change: config changed from index -1 to 4, NON_VOTER 95e7c90cdc634d64a66dad6805e1680c (127.11.42.69) added. New cstate: current_term: 1 leader_uuid: "d43426b5bc874e8482e7b3205a9ba970" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95e7c90cdc634d64a66dad6805e1680c" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 43981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:52:56.999526 13910 consensus_peers.cc:485] T 50496c7daf804faa8d7bfaf128839e2c P d43426b5bc874e8482e7b3205a9ba970 -> Peer 95e7c90cdc634d64a66dad6805e1680c (127.11.42.69:43981): Couldn't send request to peer 95e7c90cdc634d64a66dad6805e1680c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 50496c7daf804faa8d7bfaf128839e2c. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:52:57.012578 13977 consensus_queue.cc:237] T 6ed541d2e5ba4bbab19869c10116142e P d43426b5bc874e8482e7b3205a9ba970 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "39e5e5c1a5e74915800d96f720aee8c4" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 37071 } attrs { promote: true } }
I20240502 01:52:57.014127 15879 raft_consensus.cc:2932] T 6ed541d2e5ba4bbab19869c10116142e P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 39e5e5c1a5e74915800d96f720aee8c4 (127.11.42.71) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "39e5e5c1a5e74915800d96f720aee8c4" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 37071 } attrs { promote: true } } }
W20240502 01:52:57.014463 13910 consensus_peers.cc:485] T 6ed541d2e5ba4bbab19869c10116142e P d43426b5bc874e8482e7b3205a9ba970 -> Peer 39e5e5c1a5e74915800d96f720aee8c4 (127.11.42.71:37071): Couldn't send request to peer 39e5e5c1a5e74915800d96f720aee8c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6ed541d2e5ba4bbab19869c10116142e. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:52:57.015167 15776 heartbeater.cc:503] Master 127.11.42.126:40815 was elected leader, sending a full tablet report...
I20240502 01:52:57.018873 13834 catalog_manager.cc:5484] T 6ed541d2e5ba4bbab19869c10116142e P d43426b5bc874e8482e7b3205a9ba970 reported cstate change: config changed from index -1 to 4, NON_VOTER 39e5e5c1a5e74915800d96f720aee8c4 (127.11.42.71) added. New cstate: current_term: 1 leader_uuid: "d43426b5bc874e8482e7b3205a9ba970" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39e5e5c1a5e74915800d96f720aee8c4" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 37071 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:57.331442 15991 ts_tablet_manager.cc:902] T 5bc0ff324caa436196f22d90360c9247 P 020e5f8ebb004e2ab6fddaece6663ece: Initiating tablet copy from peer 1478241a03934d029dc7030480423cab (127.11.42.66:34613)
I20240502 01:52:57.333856 15991 tablet_copy_client.cc:308] T 5bc0ff324caa436196f22d90360c9247 P 020e5f8ebb004e2ab6fddaece6663ece: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.66:34613
I20240502 01:52:57.340772 14131 tablet_copy_service.cc:140] P 1478241a03934d029dc7030480423cab: Received BeginTabletCopySession request for tablet 5bc0ff324caa436196f22d90360c9247 from peer 020e5f8ebb004e2ab6fddaece6663ece ({username='slave'} at 127.11.42.67:51505)
I20240502 01:52:57.341063 14131 tablet_copy_service.cc:161] P 1478241a03934d029dc7030480423cab: Beginning new tablet copy session on tablet 5bc0ff324caa436196f22d90360c9247 from peer 020e5f8ebb004e2ab6fddaece6663ece at {username='slave'} at 127.11.42.67:51505: session id = 020e5f8ebb004e2ab6fddaece6663ece-5bc0ff324caa436196f22d90360c9247
I20240502 01:52:57.343976 14131 tablet_copy_source_session.cc:205] T 5bc0ff324caa436196f22d90360c9247 P 1478241a03934d029dc7030480423cab: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:52:57.349575 15991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bc0ff324caa436196f22d90360c9247. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:57.362973 15991 tablet_copy_client.cc:783] T 5bc0ff324caa436196f22d90360c9247 P 020e5f8ebb004e2ab6fddaece6663ece: tablet copy: Starting download of 0 data blocks...
I20240502 01:52:57.363242 16014 ts_tablet_manager.cc:902] T f9f4aab07f984168bdfec24f908c124d P 2ee638444633463b943c881866708482: Initiating tablet copy from peer 1478241a03934d029dc7030480423cab (127.11.42.66:34613)
I20240502 01:52:57.363456 15991 tablet_copy_client.cc:647] T 5bc0ff324caa436196f22d90360c9247 P 020e5f8ebb004e2ab6fddaece6663ece: tablet copy: Starting download of 1 WAL segments...
I20240502 01:52:57.366338 15991 tablet_copy_client.cc:523] T 5bc0ff324caa436196f22d90360c9247 P 020e5f8ebb004e2ab6fddaece6663ece: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:52:57.371237 15991 tablet_bootstrap.cc:492] T 5bc0ff324caa436196f22d90360c9247 P 020e5f8ebb004e2ab6fddaece6663ece: Bootstrap starting.
I20240502 01:52:57.380798 16014 tablet_copy_client.cc:308] T f9f4aab07f984168bdfec24f908c124d P 2ee638444633463b943c881866708482: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.66:34613
I20240502 01:52:57.386013 15991 log.cc:824] T 5bc0ff324caa436196f22d90360c9247 P 020e5f8ebb004e2ab6fddaece6663ece: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:57.394438 14131 tablet_copy_service.cc:140] P 1478241a03934d029dc7030480423cab: Received BeginTabletCopySession request for tablet f9f4aab07f984168bdfec24f908c124d from peer 2ee638444633463b943c881866708482 ({username='slave'} at 127.11.42.70:57485)
I20240502 01:52:57.394619 14131 tablet_copy_service.cc:161] P 1478241a03934d029dc7030480423cab: Beginning new tablet copy session on tablet f9f4aab07f984168bdfec24f908c124d from peer 2ee638444633463b943c881866708482 at {username='slave'} at 127.11.42.70:57485: session id = 2ee638444633463b943c881866708482-f9f4aab07f984168bdfec24f908c124d
I20240502 01:52:57.397256 14131 tablet_copy_source_session.cc:205] T f9f4aab07f984168bdfec24f908c124d P 1478241a03934d029dc7030480423cab: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:52:57.399539 16014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9f4aab07f984168bdfec24f908c124d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:57.405790 16014 tablet_copy_client.cc:783] T f9f4aab07f984168bdfec24f908c124d P 2ee638444633463b943c881866708482: tablet copy: Starting download of 0 data blocks...
I20240502 01:52:57.406124 16014 tablet_copy_client.cc:647] T f9f4aab07f984168bdfec24f908c124d P 2ee638444633463b943c881866708482: tablet copy: Starting download of 1 WAL segments...
I20240502 01:52:57.408527 16014 tablet_copy_client.cc:523] T f9f4aab07f984168bdfec24f908c124d P 2ee638444633463b943c881866708482: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:52:57.411410 16014 tablet_bootstrap.cc:492] T f9f4aab07f984168bdfec24f908c124d P 2ee638444633463b943c881866708482: Bootstrap starting.
I20240502 01:52:57.421641 15991 tablet_bootstrap.cc:492] T 5bc0ff324caa436196f22d90360c9247 P 020e5f8ebb004e2ab6fddaece6663ece: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:52:57.422096 15991 tablet_bootstrap.cc:492] T 5bc0ff324caa436196f22d90360c9247 P 020e5f8ebb004e2ab6fddaece6663ece: Bootstrap complete.
I20240502 01:52:57.422492 15991 ts_tablet_manager.cc:1367] T 5bc0ff324caa436196f22d90360c9247 P 020e5f8ebb004e2ab6fddaece6663ece: Time spent bootstrapping tablet: real 0.051s user 0.045s sys 0.004s
I20240502 01:52:57.427546 15991 raft_consensus.cc:348] T 5bc0ff324caa436196f22d90360c9247 P 020e5f8ebb004e2ab6fddaece6663ece [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "020e5f8ebb004e2ab6fddaece6663ece" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 37505 } attrs { promote: true } }
I20240502 01:52:57.428130 15991 raft_consensus.cc:729] T 5bc0ff324caa436196f22d90360c9247 P 020e5f8ebb004e2ab6fddaece6663ece [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 020e5f8ebb004e2ab6fddaece6663ece, State: Initialized, Role: LEARNER
I20240502 01:52:57.428697 15991 consensus_queue.cc:260] T 5bc0ff324caa436196f22d90360c9247 P 020e5f8ebb004e2ab6fddaece6663ece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "020e5f8ebb004e2ab6fddaece6663ece" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 37505 } attrs { promote: true } }
I20240502 01:52:57.430488 16030 ts_tablet_manager.cc:902] T c0620ccc47634258a7962d91cc7bb060 P 1607e211ee8c48de9fa56e78a3b01418: Initiating tablet copy from peer d43426b5bc874e8482e7b3205a9ba970 (127.11.42.65:40383)
I20240502 01:52:57.430619 15991 ts_tablet_manager.cc:1398] T 5bc0ff324caa436196f22d90360c9247 P 020e5f8ebb004e2ab6fddaece6663ece: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20240502 01:52:57.433295 14131 tablet_copy_service.cc:342] P 1478241a03934d029dc7030480423cab: Request end of tablet copy session 020e5f8ebb004e2ab6fddaece6663ece-5bc0ff324caa436196f22d90360c9247 received from {username='slave'} at 127.11.42.67:51505
I20240502 01:52:57.433494 14131 tablet_copy_service.cc:433] P 1478241a03934d029dc7030480423cab: ending tablet copy session 020e5f8ebb004e2ab6fddaece6663ece-5bc0ff324caa436196f22d90360c9247 on tablet 5bc0ff324caa436196f22d90360c9247 with peer 020e5f8ebb004e2ab6fddaece6663ece
I20240502 01:52:57.436247 16030 tablet_copy_client.cc:308] T c0620ccc47634258a7962d91cc7bb060 P 1607e211ee8c48de9fa56e78a3b01418: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.65:40383
I20240502 01:52:57.444757 13998 tablet_copy_service.cc:140] P d43426b5bc874e8482e7b3205a9ba970: Received BeginTabletCopySession request for tablet c0620ccc47634258a7962d91cc7bb060 from peer 1607e211ee8c48de9fa56e78a3b01418 ({username='slave'} at 127.11.42.68:40443)
I20240502 01:52:57.445122 13998 tablet_copy_service.cc:161] P d43426b5bc874e8482e7b3205a9ba970: Beginning new tablet copy session on tablet c0620ccc47634258a7962d91cc7bb060 from peer 1607e211ee8c48de9fa56e78a3b01418 at {username='slave'} at 127.11.42.68:40443: session id = 1607e211ee8c48de9fa56e78a3b01418-c0620ccc47634258a7962d91cc7bb060
I20240502 01:52:57.449436 13998 tablet_copy_source_session.cc:205] T c0620ccc47634258a7962d91cc7bb060 P d43426b5bc874e8482e7b3205a9ba970: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:52:57.454541 16014 tablet_bootstrap.cc:492] T f9f4aab07f984168bdfec24f908c124d P 2ee638444633463b943c881866708482: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:52:57.454875 16030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0620ccc47634258a7962d91cc7bb060. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:57.455034 16014 tablet_bootstrap.cc:492] T f9f4aab07f984168bdfec24f908c124d P 2ee638444633463b943c881866708482: Bootstrap complete.
I20240502 01:52:57.455367 16014 ts_tablet_manager.cc:1367] T f9f4aab07f984168bdfec24f908c124d P 2ee638444633463b943c881866708482: Time spent bootstrapping tablet: real 0.044s user 0.043s sys 0.000s
I20240502 01:52:57.456534 16014 raft_consensus.cc:348] T f9f4aab07f984168bdfec24f908c124d P 2ee638444633463b943c881866708482 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "2ee638444633463b943c881866708482" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 45255 } attrs { promote: true } }
I20240502 01:52:57.456835 16014 raft_consensus.cc:729] T f9f4aab07f984168bdfec24f908c124d P 2ee638444633463b943c881866708482 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2ee638444633463b943c881866708482, State: Initialized, Role: LEARNER
I20240502 01:52:57.457141 16014 consensus_queue.cc:260] T f9f4aab07f984168bdfec24f908c124d P 2ee638444633463b943c881866708482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "2ee638444633463b943c881866708482" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 45255 } attrs { promote: true } }
I20240502 01:52:57.458336 16014 ts_tablet_manager.cc:1398] T f9f4aab07f984168bdfec24f908c124d P 2ee638444633463b943c881866708482: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20240502 01:52:57.459601 16037 ts_tablet_manager.cc:902] T aa387d2b1dcc42ddb34005f08dde2d8d P 95e7c90cdc634d64a66dad6805e1680c: Initiating tablet copy from peer 1478241a03934d029dc7030480423cab (127.11.42.66:34613)
I20240502 01:52:57.460084 14131 tablet_copy_service.cc:342] P 1478241a03934d029dc7030480423cab: Request end of tablet copy session 2ee638444633463b943c881866708482-f9f4aab07f984168bdfec24f908c124d received from {username='slave'} at 127.11.42.70:57485
I20240502 01:52:57.460228 14131 tablet_copy_service.cc:433] P 1478241a03934d029dc7030480423cab: ending tablet copy session 2ee638444633463b943c881866708482-f9f4aab07f984168bdfec24f908c124d on tablet f9f4aab07f984168bdfec24f908c124d with peer 2ee638444633463b943c881866708482
I20240502 01:52:57.470147 16030 tablet_copy_client.cc:783] T c0620ccc47634258a7962d91cc7bb060 P 1607e211ee8c48de9fa56e78a3b01418: tablet copy: Starting download of 0 data blocks...
I20240502 01:52:57.470562 16030 tablet_copy_client.cc:647] T c0620ccc47634258a7962d91cc7bb060 P 1607e211ee8c48de9fa56e78a3b01418: tablet copy: Starting download of 1 WAL segments...
I20240502 01:52:57.471189 16042 ts_tablet_manager.cc:902] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P 39e5e5c1a5e74915800d96f720aee8c4: Initiating tablet copy from peer d43426b5bc874e8482e7b3205a9ba970 (127.11.42.65:40383)
I20240502 01:52:57.473141 16043 ts_tablet_manager.cc:902] T 50496c7daf804faa8d7bfaf128839e2c P 95e7c90cdc634d64a66dad6805e1680c: Initiating tablet copy from peer d43426b5bc874e8482e7b3205a9ba970 (127.11.42.65:40383)
I20240502 01:52:57.473351 16030 tablet_copy_client.cc:523] T c0620ccc47634258a7962d91cc7bb060 P 1607e211ee8c48de9fa56e78a3b01418: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:52:57.473531 16037 tablet_copy_client.cc:308] T aa387d2b1dcc42ddb34005f08dde2d8d P 95e7c90cdc634d64a66dad6805e1680c: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.66:34613
I20240502 01:52:57.477923 16030 tablet_bootstrap.cc:492] T c0620ccc47634258a7962d91cc7bb060 P 1607e211ee8c48de9fa56e78a3b01418: Bootstrap starting.
I20240502 01:52:57.486456 14131 tablet_copy_service.cc:140] P 1478241a03934d029dc7030480423cab: Received BeginTabletCopySession request for tablet aa387d2b1dcc42ddb34005f08dde2d8d from peer 95e7c90cdc634d64a66dad6805e1680c ({username='slave'} at 127.11.42.69:46487)
I20240502 01:52:57.486630 14131 tablet_copy_service.cc:161] P 1478241a03934d029dc7030480423cab: Beginning new tablet copy session on tablet aa387d2b1dcc42ddb34005f08dde2d8d from peer 95e7c90cdc634d64a66dad6805e1680c at {username='slave'} at 127.11.42.69:46487: session id = 95e7c90cdc634d64a66dad6805e1680c-aa387d2b1dcc42ddb34005f08dde2d8d
I20240502 01:52:57.489930 14131 tablet_copy_source_session.cc:205] T aa387d2b1dcc42ddb34005f08dde2d8d P 1478241a03934d029dc7030480423cab: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:52:57.492156 16030 log.cc:824] T c0620ccc47634258a7962d91cc7bb060 P 1607e211ee8c48de9fa56e78a3b01418: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:57.495321 16037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa387d2b1dcc42ddb34005f08dde2d8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:57.495990 15991 ts_tablet_manager.cc:902] T 0eb60206b8cb40d3b30cdc1f47d67f94 P 020e5f8ebb004e2ab6fddaece6663ece: Initiating tablet copy from peer d43426b5bc874e8482e7b3205a9ba970 (127.11.42.65:40383)
I20240502 01:52:57.509733 16043 tablet_copy_client.cc:308] T 50496c7daf804faa8d7bfaf128839e2c P 95e7c90cdc634d64a66dad6805e1680c: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.65:40383
I20240502 01:52:57.513763 16037 tablet_copy_client.cc:783] T aa387d2b1dcc42ddb34005f08dde2d8d P 95e7c90cdc634d64a66dad6805e1680c: tablet copy: Starting download of 0 data blocks...
I20240502 01:52:57.514253 16037 tablet_copy_client.cc:647] T aa387d2b1dcc42ddb34005f08dde2d8d P 95e7c90cdc634d64a66dad6805e1680c: tablet copy: Starting download of 1 WAL segments...
I20240502 01:52:57.516259 15991 tablet_copy_client.cc:308] T 0eb60206b8cb40d3b30cdc1f47d67f94 P 020e5f8ebb004e2ab6fddaece6663ece: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.65:40383
I20240502 01:52:57.516858 16037 tablet_copy_client.cc:523] T aa387d2b1dcc42ddb34005f08dde2d8d P 95e7c90cdc634d64a66dad6805e1680c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:52:57.521818 16037 tablet_bootstrap.cc:492] T aa387d2b1dcc42ddb34005f08dde2d8d P 95e7c90cdc634d64a66dad6805e1680c: Bootstrap starting.
I20240502 01:52:57.526829 13998 tablet_copy_service.cc:140] P d43426b5bc874e8482e7b3205a9ba970: Received BeginTabletCopySession request for tablet 50496c7daf804faa8d7bfaf128839e2c from peer 95e7c90cdc634d64a66dad6805e1680c ({username='slave'} at 127.11.42.69:32905)
I20240502 01:52:57.527030 13998 tablet_copy_service.cc:161] P d43426b5bc874e8482e7b3205a9ba970: Beginning new tablet copy session on tablet 50496c7daf804faa8d7bfaf128839e2c from peer 95e7c90cdc634d64a66dad6805e1680c at {username='slave'} at 127.11.42.69:32905: session id = 95e7c90cdc634d64a66dad6805e1680c-50496c7daf804faa8d7bfaf128839e2c
I20240502 01:52:57.529844 13997 tablet_copy_service.cc:140] P d43426b5bc874e8482e7b3205a9ba970: Received BeginTabletCopySession request for tablet 0eb60206b8cb40d3b30cdc1f47d67f94 from peer 020e5f8ebb004e2ab6fddaece6663ece ({username='slave'} at 127.11.42.67:38529)
I20240502 01:52:57.530038 13997 tablet_copy_service.cc:161] P d43426b5bc874e8482e7b3205a9ba970: Beginning new tablet copy session on tablet 0eb60206b8cb40d3b30cdc1f47d67f94 from peer 020e5f8ebb004e2ab6fddaece6663ece at {username='slave'} at 127.11.42.67:38529: session id = 020e5f8ebb004e2ab6fddaece6663ece-0eb60206b8cb40d3b30cdc1f47d67f94
I20240502 01:52:57.530478 13998 tablet_copy_source_session.cc:205] T 50496c7daf804faa8d7bfaf128839e2c P d43426b5bc874e8482e7b3205a9ba970: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:52:57.532076 16043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50496c7daf804faa8d7bfaf128839e2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:57.533371 13997 tablet_copy_source_session.cc:205] T 0eb60206b8cb40d3b30cdc1f47d67f94 P d43426b5bc874e8482e7b3205a9ba970: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:52:57.534013 16042 tablet_copy_client.cc:308] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P 39e5e5c1a5e74915800d96f720aee8c4: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.65:40383
I20240502 01:52:57.534925 15991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0eb60206b8cb40d3b30cdc1f47d67f94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:57.536974 16037 log.cc:824] T aa387d2b1dcc42ddb34005f08dde2d8d P 95e7c90cdc634d64a66dad6805e1680c: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:57.542639 15991 tablet_copy_client.cc:783] T 0eb60206b8cb40d3b30cdc1f47d67f94 P 020e5f8ebb004e2ab6fddaece6663ece: tablet copy: Starting download of 0 data blocks...
I20240502 01:52:57.542991 15991 tablet_copy_client.cc:647] T 0eb60206b8cb40d3b30cdc1f47d67f94 P 020e5f8ebb004e2ab6fddaece6663ece: tablet copy: Starting download of 1 WAL segments...
I20240502 01:52:57.543112 16043 tablet_copy_client.cc:783] T 50496c7daf804faa8d7bfaf128839e2c P 95e7c90cdc634d64a66dad6805e1680c: tablet copy: Starting download of 0 data blocks...
I20240502 01:52:57.543362 16043 tablet_copy_client.cc:647] T 50496c7daf804faa8d7bfaf128839e2c P 95e7c90cdc634d64a66dad6805e1680c: tablet copy: Starting download of 1 WAL segments...
I20240502 01:52:57.545266 15991 tablet_copy_client.cc:523] T 0eb60206b8cb40d3b30cdc1f47d67f94 P 020e5f8ebb004e2ab6fddaece6663ece: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:52:57.545658 16043 tablet_copy_client.cc:523] T 50496c7daf804faa8d7bfaf128839e2c P 95e7c90cdc634d64a66dad6805e1680c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:52:57.549113 15991 tablet_bootstrap.cc:492] T 0eb60206b8cb40d3b30cdc1f47d67f94 P 020e5f8ebb004e2ab6fddaece6663ece: Bootstrap starting.
I20240502 01:52:57.551105 16043 tablet_bootstrap.cc:492] T 50496c7daf804faa8d7bfaf128839e2c P 95e7c90cdc634d64a66dad6805e1680c: Bootstrap starting.
I20240502 01:52:57.569388 13997 tablet_copy_service.cc:140] P d43426b5bc874e8482e7b3205a9ba970: Received BeginTabletCopySession request for tablet 67bc0cd4c3b94782ac1d9f150fcdbd03 from peer 39e5e5c1a5e74915800d96f720aee8c4 ({username='slave'} at 127.11.42.71:39761)
I20240502 01:52:57.569581 13997 tablet_copy_service.cc:161] P d43426b5bc874e8482e7b3205a9ba970: Beginning new tablet copy session on tablet 67bc0cd4c3b94782ac1d9f150fcdbd03 from peer 39e5e5c1a5e74915800d96f720aee8c4 at {username='slave'} at 127.11.42.71:39761: session id = 39e5e5c1a5e74915800d96f720aee8c4-67bc0cd4c3b94782ac1d9f150fcdbd03
I20240502 01:52:57.572293 16030 tablet_bootstrap.cc:492] T c0620ccc47634258a7962d91cc7bb060 P 1607e211ee8c48de9fa56e78a3b01418: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:52:57.572814 13997 tablet_copy_source_session.cc:205] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P d43426b5bc874e8482e7b3205a9ba970: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:52:57.572866 16030 tablet_bootstrap.cc:492] T c0620ccc47634258a7962d91cc7bb060 P 1607e211ee8c48de9fa56e78a3b01418: Bootstrap complete.
I20240502 01:52:57.573339 16030 ts_tablet_manager.cc:1367] T c0620ccc47634258a7962d91cc7bb060 P 1607e211ee8c48de9fa56e78a3b01418: Time spent bootstrapping tablet: real 0.096s user 0.064s sys 0.008s
I20240502 01:52:57.577659 16063 ts_tablet_manager.cc:902] T 6eb1311f3c054aec9a306922f25c6b89 P 1607e211ee8c48de9fa56e78a3b01418: Initiating tablet copy from peer 1478241a03934d029dc7030480423cab (127.11.42.66:34613)
I20240502 01:52:57.578872 16042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67bc0cd4c3b94782ac1d9f150fcdbd03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:57.583688 16030 raft_consensus.cc:348] T c0620ccc47634258a7962d91cc7bb060 P 1607e211ee8c48de9fa56e78a3b01418 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "1607e211ee8c48de9fa56e78a3b01418" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 41779 } attrs { promote: true } }
I20240502 01:52:57.589224 16030 raft_consensus.cc:729] T c0620ccc47634258a7962d91cc7bb060 P 1607e211ee8c48de9fa56e78a3b01418 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1607e211ee8c48de9fa56e78a3b01418, State: Initialized, Role: LEARNER
I20240502 01:52:57.590221 16030 consensus_queue.cc:260] T c0620ccc47634258a7962d91cc7bb060 P 1607e211ee8c48de9fa56e78a3b01418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "1607e211ee8c48de9fa56e78a3b01418" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 41779 } attrs { promote: true } }
I20240502 01:52:57.602311 16030 ts_tablet_manager.cc:1398] T c0620ccc47634258a7962d91cc7bb060 P 1607e211ee8c48de9fa56e78a3b01418: Time spent starting tablet: real 0.029s user 0.009s sys 0.002s
I20240502 01:52:57.603616 16063 tablet_copy_client.cc:308] T 6eb1311f3c054aec9a306922f25c6b89 P 1607e211ee8c48de9fa56e78a3b01418: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.66:34613
I20240502 01:52:57.613086 14131 tablet_copy_service.cc:140] P 1478241a03934d029dc7030480423cab: Received BeginTabletCopySession request for tablet 6eb1311f3c054aec9a306922f25c6b89 from peer 1607e211ee8c48de9fa56e78a3b01418 ({username='slave'} at 127.11.42.68:40023)
I20240502 01:52:57.613309 14131 tablet_copy_service.cc:161] P 1478241a03934d029dc7030480423cab: Beginning new tablet copy session on tablet 6eb1311f3c054aec9a306922f25c6b89 from peer 1607e211ee8c48de9fa56e78a3b01418 at {username='slave'} at 127.11.42.68:40023: session id = 1607e211ee8c48de9fa56e78a3b01418-6eb1311f3c054aec9a306922f25c6b89
I20240502 01:52:57.616518 13997 tablet_copy_service.cc:342] P d43426b5bc874e8482e7b3205a9ba970: Request end of tablet copy session 1607e211ee8c48de9fa56e78a3b01418-c0620ccc47634258a7962d91cc7bb060 received from {username='slave'} at 127.11.42.68:40443
I20240502 01:52:57.616695 13997 tablet_copy_service.cc:433] P d43426b5bc874e8482e7b3205a9ba970: ending tablet copy session 1607e211ee8c48de9fa56e78a3b01418-c0620ccc47634258a7962d91cc7bb060 on tablet c0620ccc47634258a7962d91cc7bb060 with peer 1607e211ee8c48de9fa56e78a3b01418
I20240502 01:52:57.617599 14131 tablet_copy_source_session.cc:205] T 6eb1311f3c054aec9a306922f25c6b89 P 1478241a03934d029dc7030480423cab: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:52:57.619730 16063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eb1311f3c054aec9a306922f25c6b89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:57.622248 16070 ts_tablet_manager.cc:902] T 6ed541d2e5ba4bbab19869c10116142e P 39e5e5c1a5e74915800d96f720aee8c4: Initiating tablet copy from peer d43426b5bc874e8482e7b3205a9ba970 (127.11.42.65:40383)
I20240502 01:52:57.628540 16063 tablet_copy_client.cc:783] T 6eb1311f3c054aec9a306922f25c6b89 P 1607e211ee8c48de9fa56e78a3b01418: tablet copy: Starting download of 0 data blocks...
I20240502 01:52:57.628890 16063 tablet_copy_client.cc:647] T 6eb1311f3c054aec9a306922f25c6b89 P 1607e211ee8c48de9fa56e78a3b01418: tablet copy: Starting download of 1 WAL segments...
I20240502 01:52:57.628903 15991 tablet_bootstrap.cc:492] T 0eb60206b8cb40d3b30cdc1f47d67f94 P 020e5f8ebb004e2ab6fddaece6663ece: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:52:57.629381 15991 tablet_bootstrap.cc:492] T 0eb60206b8cb40d3b30cdc1f47d67f94 P 020e5f8ebb004e2ab6fddaece6663ece: Bootstrap complete.
I20240502 01:52:57.629738 15991 ts_tablet_manager.cc:1367] T 0eb60206b8cb40d3b30cdc1f47d67f94 P 020e5f8ebb004e2ab6fddaece6663ece: Time spent bootstrapping tablet: real 0.081s user 0.045s sys 0.012s
I20240502 01:52:57.630797 16063 tablet_copy_client.cc:523] T 6eb1311f3c054aec9a306922f25c6b89 P 1607e211ee8c48de9fa56e78a3b01418: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:52:57.631225 15991 raft_consensus.cc:348] T 0eb60206b8cb40d3b30cdc1f47d67f94 P 020e5f8ebb004e2ab6fddaece6663ece [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "020e5f8ebb004e2ab6fddaece6663ece" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 37505 } attrs { promote: true } }
I20240502 01:52:57.631585 15991 raft_consensus.cc:729] T 0eb60206b8cb40d3b30cdc1f47d67f94 P 020e5f8ebb004e2ab6fddaece6663ece [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 020e5f8ebb004e2ab6fddaece6663ece, State: Initialized, Role: LEARNER
I20240502 01:52:57.631829 15991 consensus_queue.cc:260] T 0eb60206b8cb40d3b30cdc1f47d67f94 P 020e5f8ebb004e2ab6fddaece6663ece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "020e5f8ebb004e2ab6fddaece6663ece" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 37505 } attrs { promote: true } }
I20240502 01:52:57.632495 15991 ts_tablet_manager.cc:1398] T 0eb60206b8cb40d3b30cdc1f47d67f94 P 020e5f8ebb004e2ab6fddaece6663ece: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:52:57.633244 13997 tablet_copy_service.cc:342] P d43426b5bc874e8482e7b3205a9ba970: Request end of tablet copy session 020e5f8ebb004e2ab6fddaece6663ece-0eb60206b8cb40d3b30cdc1f47d67f94 received from {username='slave'} at 127.11.42.67:38529
I20240502 01:52:57.633394 13997 tablet_copy_service.cc:433] P d43426b5bc874e8482e7b3205a9ba970: ending tablet copy session 020e5f8ebb004e2ab6fddaece6663ece-0eb60206b8cb40d3b30cdc1f47d67f94 on tablet 0eb60206b8cb40d3b30cdc1f47d67f94 with peer 020e5f8ebb004e2ab6fddaece6663ece
I20240502 01:52:57.634697 16063 tablet_bootstrap.cc:492] T 6eb1311f3c054aec9a306922f25c6b89 P 1607e211ee8c48de9fa56e78a3b01418: Bootstrap starting.
I20240502 01:52:57.636560 16042 tablet_copy_client.cc:783] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P 39e5e5c1a5e74915800d96f720aee8c4: tablet copy: Starting download of 0 data blocks...
I20240502 01:52:57.636847 16042 tablet_copy_client.cc:647] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P 39e5e5c1a5e74915800d96f720aee8c4: tablet copy: Starting download of 1 WAL segments...
I20240502 01:52:57.642377 16042 tablet_copy_client.cc:523] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P 39e5e5c1a5e74915800d96f720aee8c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:52:57.643522 16037 tablet_bootstrap.cc:492] T aa387d2b1dcc42ddb34005f08dde2d8d P 95e7c90cdc634d64a66dad6805e1680c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:52:57.644284 16037 tablet_bootstrap.cc:492] T aa387d2b1dcc42ddb34005f08dde2d8d P 95e7c90cdc634d64a66dad6805e1680c: Bootstrap complete.
I20240502 01:52:57.644884 16037 ts_tablet_manager.cc:1367] T aa387d2b1dcc42ddb34005f08dde2d8d P 95e7c90cdc634d64a66dad6805e1680c: Time spent bootstrapping tablet: real 0.123s user 0.069s sys 0.010s
I20240502 01:52:57.653833 16037 raft_consensus.cc:348] T aa387d2b1dcc42ddb34005f08dde2d8d P 95e7c90cdc634d64a66dad6805e1680c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "95e7c90cdc634d64a66dad6805e1680c" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 43981 } attrs { promote: true } }
I20240502 01:52:57.654749 16037 raft_consensus.cc:729] T aa387d2b1dcc42ddb34005f08dde2d8d P 95e7c90cdc634d64a66dad6805e1680c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 95e7c90cdc634d64a66dad6805e1680c, State: Initialized, Role: LEARNER
I20240502 01:52:57.655555 16037 consensus_queue.cc:260] T aa387d2b1dcc42ddb34005f08dde2d8d P 95e7c90cdc634d64a66dad6805e1680c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "95e7c90cdc634d64a66dad6805e1680c" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 43981 } attrs { promote: true } }
I20240502 01:52:57.655756 16042 tablet_bootstrap.cc:492] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P 39e5e5c1a5e74915800d96f720aee8c4: Bootstrap starting.
I20240502 01:52:57.659930 16037 ts_tablet_manager.cc:1398] T aa387d2b1dcc42ddb34005f08dde2d8d P 95e7c90cdc634d64a66dad6805e1680c: Time spent starting tablet: real 0.015s user 0.007s sys 0.004s
I20240502 01:52:57.661235 16070 tablet_copy_client.cc:308] T 6ed541d2e5ba4bbab19869c10116142e P 39e5e5c1a5e74915800d96f720aee8c4: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.65:40383
I20240502 01:52:57.661882 13997 tablet_copy_service.cc:140] P d43426b5bc874e8482e7b3205a9ba970: Received BeginTabletCopySession request for tablet 6ed541d2e5ba4bbab19869c10116142e from peer 39e5e5c1a5e74915800d96f720aee8c4 ({username='slave'} at 127.11.42.71:39761)
I20240502 01:52:57.662094 13997 tablet_copy_service.cc:161] P d43426b5bc874e8482e7b3205a9ba970: Beginning new tablet copy session on tablet 6ed541d2e5ba4bbab19869c10116142e from peer 39e5e5c1a5e74915800d96f720aee8c4 at {username='slave'} at 127.11.42.71:39761: session id = 39e5e5c1a5e74915800d96f720aee8c4-6ed541d2e5ba4bbab19869c10116142e
I20240502 01:52:57.665217 14131 tablet_copy_service.cc:342] P 1478241a03934d029dc7030480423cab: Request end of tablet copy session 95e7c90cdc634d64a66dad6805e1680c-aa387d2b1dcc42ddb34005f08dde2d8d received from {username='slave'} at 127.11.42.69:46487
I20240502 01:52:57.665398 14131 tablet_copy_service.cc:433] P 1478241a03934d029dc7030480423cab: ending tablet copy session 95e7c90cdc634d64a66dad6805e1680c-aa387d2b1dcc42ddb34005f08dde2d8d on tablet aa387d2b1dcc42ddb34005f08dde2d8d with peer 95e7c90cdc634d64a66dad6805e1680c
I20240502 01:52:57.670044 13997 tablet_copy_source_session.cc:205] T 6ed541d2e5ba4bbab19869c10116142e P d43426b5bc874e8482e7b3205a9ba970: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:52:57.671329 16070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ed541d2e5ba4bbab19869c10116142e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:57.675014 16042 log.cc:824] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P 39e5e5c1a5e74915800d96f720aee8c4: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:57.678660 16070 tablet_copy_client.cc:783] T 6ed541d2e5ba4bbab19869c10116142e P 39e5e5c1a5e74915800d96f720aee8c4: tablet copy: Starting download of 0 data blocks...
I20240502 01:52:57.679045 16070 tablet_copy_client.cc:647] T 6ed541d2e5ba4bbab19869c10116142e P 39e5e5c1a5e74915800d96f720aee8c4: tablet copy: Starting download of 1 WAL segments...
I20240502 01:52:57.681239 16070 tablet_copy_client.cc:523] T 6ed541d2e5ba4bbab19869c10116142e P 39e5e5c1a5e74915800d96f720aee8c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:52:57.683516 16043 tablet_bootstrap.cc:492] T 50496c7daf804faa8d7bfaf128839e2c P 95e7c90cdc634d64a66dad6805e1680c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:52:57.683997 16043 tablet_bootstrap.cc:492] T 50496c7daf804faa8d7bfaf128839e2c P 95e7c90cdc634d64a66dad6805e1680c: Bootstrap complete.
I20240502 01:52:57.684309 16043 ts_tablet_manager.cc:1367] T 50496c7daf804faa8d7bfaf128839e2c P 95e7c90cdc634d64a66dad6805e1680c: Time spent bootstrapping tablet: real 0.133s user 0.045s sys 0.009s
I20240502 01:52:57.684882 16070 tablet_bootstrap.cc:492] T 6ed541d2e5ba4bbab19869c10116142e P 39e5e5c1a5e74915800d96f720aee8c4: Bootstrap starting.
I20240502 01:52:57.685509 16043 raft_consensus.cc:348] T 50496c7daf804faa8d7bfaf128839e2c P 95e7c90cdc634d64a66dad6805e1680c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "95e7c90cdc634d64a66dad6805e1680c" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 43981 } attrs { promote: true } }
I20240502 01:52:57.685889 16043 raft_consensus.cc:729] T 50496c7daf804faa8d7bfaf128839e2c P 95e7c90cdc634d64a66dad6805e1680c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 95e7c90cdc634d64a66dad6805e1680c, State: Initialized, Role: LEARNER
I20240502 01:52:57.693238 16043 consensus_queue.cc:260] T 50496c7daf804faa8d7bfaf128839e2c P 95e7c90cdc634d64a66dad6805e1680c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "95e7c90cdc634d64a66dad6805e1680c" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 43981 } attrs { promote: true } }
I20240502 01:52:57.696837 16043 ts_tablet_manager.cc:1398] T 50496c7daf804faa8d7bfaf128839e2c P 95e7c90cdc634d64a66dad6805e1680c: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240502 01:52:57.696933 16063 tablet_bootstrap.cc:492] T 6eb1311f3c054aec9a306922f25c6b89 P 1607e211ee8c48de9fa56e78a3b01418: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:52:57.697564 16063 tablet_bootstrap.cc:492] T 6eb1311f3c054aec9a306922f25c6b89 P 1607e211ee8c48de9fa56e78a3b01418: Bootstrap complete.
I20240502 01:52:57.697898 16063 ts_tablet_manager.cc:1367] T 6eb1311f3c054aec9a306922f25c6b89 P 1607e211ee8c48de9fa56e78a3b01418: Time spent bootstrapping tablet: real 0.063s user 0.026s sys 0.023s
I20240502 01:52:57.699023 16063 raft_consensus.cc:348] T 6eb1311f3c054aec9a306922f25c6b89 P 1607e211ee8c48de9fa56e78a3b01418 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "1607e211ee8c48de9fa56e78a3b01418" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 41779 } attrs { promote: true } }
I20240502 01:52:57.699345 16063 raft_consensus.cc:729] T 6eb1311f3c054aec9a306922f25c6b89 P 1607e211ee8c48de9fa56e78a3b01418 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1607e211ee8c48de9fa56e78a3b01418, State: Initialized, Role: LEARNER
I20240502 01:52:57.699661 16063 consensus_queue.cc:260] T 6eb1311f3c054aec9a306922f25c6b89 P 1607e211ee8c48de9fa56e78a3b01418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "1607e211ee8c48de9fa56e78a3b01418" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 41779 } attrs { promote: true } }
I20240502 01:52:57.700112 13997 tablet_copy_service.cc:342] P d43426b5bc874e8482e7b3205a9ba970: Request end of tablet copy session 95e7c90cdc634d64a66dad6805e1680c-50496c7daf804faa8d7bfaf128839e2c received from {username='slave'} at 127.11.42.69:32905
I20240502 01:52:57.700287 13997 tablet_copy_service.cc:433] P d43426b5bc874e8482e7b3205a9ba970: ending tablet copy session 95e7c90cdc634d64a66dad6805e1680c-50496c7daf804faa8d7bfaf128839e2c on tablet 50496c7daf804faa8d7bfaf128839e2c with peer 95e7c90cdc634d64a66dad6805e1680c
I20240502 01:52:57.702682 16063 ts_tablet_manager.cc:1398] T 6eb1311f3c054aec9a306922f25c6b89 P 1607e211ee8c48de9fa56e78a3b01418: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 01:52:57.705458 14131 tablet_copy_service.cc:342] P 1478241a03934d029dc7030480423cab: Request end of tablet copy session 1607e211ee8c48de9fa56e78a3b01418-6eb1311f3c054aec9a306922f25c6b89 received from {username='slave'} at 127.11.42.68:40023
I20240502 01:52:57.705600 14131 tablet_copy_service.cc:433] P 1478241a03934d029dc7030480423cab: ending tablet copy session 1607e211ee8c48de9fa56e78a3b01418-6eb1311f3c054aec9a306922f25c6b89 on tablet 6eb1311f3c054aec9a306922f25c6b89 with peer 1607e211ee8c48de9fa56e78a3b01418
I20240502 01:52:57.730237 16042 tablet_bootstrap.cc:492] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P 39e5e5c1a5e74915800d96f720aee8c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:52:57.731040 16042 tablet_bootstrap.cc:492] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P 39e5e5c1a5e74915800d96f720aee8c4: Bootstrap complete.
I20240502 01:52:57.731536 16042 ts_tablet_manager.cc:1367] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P 39e5e5c1a5e74915800d96f720aee8c4: Time spent bootstrapping tablet: real 0.079s user 0.045s sys 0.019s
I20240502 01:52:57.736968 16042 raft_consensus.cc:348] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P 39e5e5c1a5e74915800d96f720aee8c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "39e5e5c1a5e74915800d96f720aee8c4" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 37071 } attrs { promote: true } }
I20240502 01:52:57.737449 16042 raft_consensus.cc:729] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P 39e5e5c1a5e74915800d96f720aee8c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 39e5e5c1a5e74915800d96f720aee8c4, State: Initialized, Role: LEARNER
I20240502 01:52:57.738057 16042 consensus_queue.cc:260] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P 39e5e5c1a5e74915800d96f720aee8c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "39e5e5c1a5e74915800d96f720aee8c4" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 37071 } attrs { promote: true } }
I20240502 01:52:57.740028 16042 ts_tablet_manager.cc:1398] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P 39e5e5c1a5e74915800d96f720aee8c4: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240502 01:52:57.740979 13997 tablet_copy_service.cc:342] P d43426b5bc874e8482e7b3205a9ba970: Request end of tablet copy session 39e5e5c1a5e74915800d96f720aee8c4-67bc0cd4c3b94782ac1d9f150fcdbd03 received from {username='slave'} at 127.11.42.71:39761
I20240502 01:52:57.741163 13997 tablet_copy_service.cc:433] P d43426b5bc874e8482e7b3205a9ba970: ending tablet copy session 39e5e5c1a5e74915800d96f720aee8c4-67bc0cd4c3b94782ac1d9f150fcdbd03 on tablet 67bc0cd4c3b94782ac1d9f150fcdbd03 with peer 39e5e5c1a5e74915800d96f720aee8c4
I20240502 01:52:57.746817 16070 tablet_bootstrap.cc:492] T 6ed541d2e5ba4bbab19869c10116142e P 39e5e5c1a5e74915800d96f720aee8c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:52:57.747206 16070 tablet_bootstrap.cc:492] T 6ed541d2e5ba4bbab19869c10116142e P 39e5e5c1a5e74915800d96f720aee8c4: Bootstrap complete.
I20240502 01:52:57.747504 16070 ts_tablet_manager.cc:1367] T 6ed541d2e5ba4bbab19869c10116142e P 39e5e5c1a5e74915800d96f720aee8c4: Time spent bootstrapping tablet: real 0.063s user 0.050s sys 0.000s
I20240502 01:52:57.748485 16070 raft_consensus.cc:348] T 6ed541d2e5ba4bbab19869c10116142e P 39e5e5c1a5e74915800d96f720aee8c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "39e5e5c1a5e74915800d96f720aee8c4" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 37071 } attrs { promote: true } }
I20240502 01:52:57.748728 16070 raft_consensus.cc:729] T 6ed541d2e5ba4bbab19869c10116142e P 39e5e5c1a5e74915800d96f720aee8c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 39e5e5c1a5e74915800d96f720aee8c4, State: Initialized, Role: LEARNER
I20240502 01:52:57.748942 16070 consensus_queue.cc:260] T 6ed541d2e5ba4bbab19869c10116142e P 39e5e5c1a5e74915800d96f720aee8c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "39e5e5c1a5e74915800d96f720aee8c4" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 37071 } attrs { promote: true } }
I20240502 01:52:57.749531 16070 ts_tablet_manager.cc:1398] T 6ed541d2e5ba4bbab19869c10116142e P 39e5e5c1a5e74915800d96f720aee8c4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:52:57.750597 13997 tablet_copy_service.cc:342] P d43426b5bc874e8482e7b3205a9ba970: Request end of tablet copy session 39e5e5c1a5e74915800d96f720aee8c4-6ed541d2e5ba4bbab19869c10116142e received from {username='slave'} at 127.11.42.71:39761
I20240502 01:52:57.750737 13997 tablet_copy_service.cc:433] P d43426b5bc874e8482e7b3205a9ba970: ending tablet copy session 39e5e5c1a5e74915800d96f720aee8c4-6ed541d2e5ba4bbab19869c10116142e on tablet 6ed541d2e5ba4bbab19869c10116142e with peer 39e5e5c1a5e74915800d96f720aee8c4
I20240502 01:52:57.802803 15592 raft_consensus.cc:1203] T f9f4aab07f984168bdfec24f908c124d P 2ee638444633463b943c881866708482 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:52:57.857061 15731 raft_consensus.cc:1203] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P 39e5e5c1a5e74915800d96f720aee8c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:52:57.888659 15196 raft_consensus.cc:1203] T 5bc0ff324caa436196f22d90360c9247 P 020e5f8ebb004e2ab6fddaece6663ece [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:52:57.954349 15328 raft_consensus.cc:1203] T c0620ccc47634258a7962d91cc7bb060 P 1607e211ee8c48de9fa56e78a3b01418 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:52:58.013586 15459 raft_consensus.cc:1203] T aa387d2b1dcc42ddb34005f08dde2d8d P 95e7c90cdc634d64a66dad6805e1680c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:52:58.026367 15731 raft_consensus.cc:1203] T 6ed541d2e5ba4bbab19869c10116142e P 39e5e5c1a5e74915800d96f720aee8c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:52:58.039242 15328 raft_consensus.cc:1203] T 6eb1311f3c054aec9a306922f25c6b89 P 1607e211ee8c48de9fa56e78a3b01418 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:52:58.051584 15196 raft_consensus.cc:1203] T 0eb60206b8cb40d3b30cdc1f47d67f94 P 020e5f8ebb004e2ab6fddaece6663ece [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:52:58.077421 15459 raft_consensus.cc:1203] T 50496c7daf804faa8d7bfaf128839e2c P 95e7c90cdc634d64a66dad6805e1680c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:52:58.252010 16125 raft_consensus.cc:1050] T f9f4aab07f984168bdfec24f908c124d P 1478241a03934d029dc7030480423cab: attempting to promote NON_VOTER 2ee638444633463b943c881866708482 to VOTER
I20240502 01:52:58.252902 16125 consensus_queue.cc:237] T f9f4aab07f984168bdfec24f908c124d P 1478241a03934d029dc7030480423cab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "2ee638444633463b943c881866708482" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 45255 } attrs { promote: false } }
I20240502 01:52:58.254905 15592 raft_consensus.cc:1261] T f9f4aab07f984168bdfec24f908c124d P 2ee638444633463b943c881866708482 [term 1 LEARNER]: Refusing update from remote peer 1478241a03934d029dc7030480423cab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:52:58.255563 16124 consensus_queue.cc:1035] T f9f4aab07f984168bdfec24f908c124d P 1478241a03934d029dc7030480423cab [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ee638444633463b943c881866708482" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 45255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:52:58.258347 16125 raft_consensus.cc:2932] T f9f4aab07f984168bdfec24f908c124d P 1478241a03934d029dc7030480423cab [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2ee638444633463b943c881866708482 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "2ee638444633463b943c881866708482" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 45255 } attrs { promote: false } } }
I20240502 01:52:58.259002 15592 raft_consensus.cc:2932] T f9f4aab07f984168bdfec24f908c124d P 2ee638444633463b943c881866708482 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2ee638444633463b943c881866708482 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "2ee638444633463b943c881866708482" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 45255 } attrs { promote: false } } }
I20240502 01:52:58.263510 13834 catalog_manager.cc:5484] T f9f4aab07f984168bdfec24f908c124d P 1478241a03934d029dc7030480423cab reported cstate change: config changed from index 4 to 5, 2ee638444633463b943c881866708482 (127.11.42.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1478241a03934d029dc7030480423cab" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ee638444633463b943c881866708482" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 45255 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:52:58.344857 14111 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f9f4aab07f984168bdfec24f908c124d"
dest_uuid: "1478241a03934d029dc7030480423cab"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40364
I20240502 01:52:58.345115 14111 raft_consensus.cc:595] T f9f4aab07f984168bdfec24f908c124d P 1478241a03934d029dc7030480423cab [term 1 LEADER]: Received request to transfer leadership
I20240502 01:52:58.347903 16125 raft_consensus.cc:1050] T 5bc0ff324caa436196f22d90360c9247 P 1478241a03934d029dc7030480423cab: attempting to promote NON_VOTER 020e5f8ebb004e2ab6fddaece6663ece to VOTER
I20240502 01:52:58.348475 16125 consensus_queue.cc:237] T 5bc0ff324caa436196f22d90360c9247 P 1478241a03934d029dc7030480423cab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "020e5f8ebb004e2ab6fddaece6663ece" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 37505 } attrs { promote: false } }
I20240502 01:52:58.350566 15196 raft_consensus.cc:1261] T 5bc0ff324caa436196f22d90360c9247 P 020e5f8ebb004e2ab6fddaece6663ece [term 1 LEARNER]: Refusing update from remote peer 1478241a03934d029dc7030480423cab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:52:58.351145 16124 consensus_queue.cc:1035] T 5bc0ff324caa436196f22d90360c9247 P 1478241a03934d029dc7030480423cab [LEADER]: Connected to new peer: Peer: permanent_uuid: "020e5f8ebb004e2ab6fddaece6663ece" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 37505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:52:58.354872 16125 raft_consensus.cc:2932] T 5bc0ff324caa436196f22d90360c9247 P 1478241a03934d029dc7030480423cab [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 020e5f8ebb004e2ab6fddaece6663ece (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "020e5f8ebb004e2ab6fddaece6663ece" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 37505 } attrs { promote: false } } }
I20240502 01:52:58.356104 15196 raft_consensus.cc:2932] T 5bc0ff324caa436196f22d90360c9247 P 020e5f8ebb004e2ab6fddaece6663ece [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 020e5f8ebb004e2ab6fddaece6663ece (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "020e5f8ebb004e2ab6fddaece6663ece" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 37505 } attrs { promote: false } } }
I20240502 01:52:58.359249 13833 catalog_manager.cc:5484] T 5bc0ff324caa436196f22d90360c9247 P 1478241a03934d029dc7030480423cab reported cstate change: config changed from index 4 to 5, 020e5f8ebb004e2ab6fddaece6663ece (127.11.42.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1478241a03934d029dc7030480423cab" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "020e5f8ebb004e2ab6fddaece6663ece" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 37505 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:52:58.385000 16126 raft_consensus.cc:1050] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P d43426b5bc874e8482e7b3205a9ba970: attempting to promote NON_VOTER 39e5e5c1a5e74915800d96f720aee8c4 to VOTER
I20240502 01:52:58.385859 16126 consensus_queue.cc:237] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P d43426b5bc874e8482e7b3205a9ba970 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "39e5e5c1a5e74915800d96f720aee8c4" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 37071 } attrs { promote: false } }
I20240502 01:52:58.387682 15731 raft_consensus.cc:1261] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P 39e5e5c1a5e74915800d96f720aee8c4 [term 1 LEARNER]: Refusing update from remote peer d43426b5bc874e8482e7b3205a9ba970: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:52:58.388275 16127 consensus_queue.cc:1035] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P d43426b5bc874e8482e7b3205a9ba970 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39e5e5c1a5e74915800d96f720aee8c4" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 37071 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:52:58.391454 16126 raft_consensus.cc:2932] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 39e5e5c1a5e74915800d96f720aee8c4 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "39e5e5c1a5e74915800d96f720aee8c4" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 37071 } attrs { promote: false } } }
I20240502 01:52:58.392068 15731 raft_consensus.cc:2932] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P 39e5e5c1a5e74915800d96f720aee8c4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 39e5e5c1a5e74915800d96f720aee8c4 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "39e5e5c1a5e74915800d96f720aee8c4" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 37071 } attrs { promote: false } } }
I20240502 01:52:58.395831 13834 catalog_manager.cc:5484] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P d43426b5bc874e8482e7b3205a9ba970 reported cstate change: config changed from index 4 to 5, 39e5e5c1a5e74915800d96f720aee8c4 (127.11.42.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d43426b5bc874e8482e7b3205a9ba970" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39e5e5c1a5e74915800d96f720aee8c4" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 37071 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:52:58.411067 16125 raft_consensus.cc:1050] T aa387d2b1dcc42ddb34005f08dde2d8d P 1478241a03934d029dc7030480423cab: attempting to promote NON_VOTER 95e7c90cdc634d64a66dad6805e1680c to VOTER
I20240502 01:52:58.411604 16125 consensus_queue.cc:237] T aa387d2b1dcc42ddb34005f08dde2d8d P 1478241a03934d029dc7030480423cab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "95e7c90cdc634d64a66dad6805e1680c" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 43981 } attrs { promote: false } }
I20240502 01:52:58.413389 15459 raft_consensus.cc:1261] T aa387d2b1dcc42ddb34005f08dde2d8d P 95e7c90cdc634d64a66dad6805e1680c [term 1 LEARNER]: Refusing update from remote peer 1478241a03934d029dc7030480423cab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:52:58.413887 15866 consensus_queue.cc:1035] T aa387d2b1dcc42ddb34005f08dde2d8d P 1478241a03934d029dc7030480423cab [LEADER]: Connected to new peer: Peer: permanent_uuid: "95e7c90cdc634d64a66dad6805e1680c" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:52:58.416937 16125 raft_consensus.cc:2932] T aa387d2b1dcc42ddb34005f08dde2d8d P 1478241a03934d029dc7030480423cab [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 95e7c90cdc634d64a66dad6805e1680c (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "95e7c90cdc634d64a66dad6805e1680c" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 43981 } attrs { promote: false } } }
I20240502 01:52:58.417562 15459 raft_consensus.cc:2932] T aa387d2b1dcc42ddb34005f08dde2d8d P 95e7c90cdc634d64a66dad6805e1680c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 95e7c90cdc634d64a66dad6805e1680c (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "95e7c90cdc634d64a66dad6805e1680c" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 43981 } attrs { promote: false } } }
I20240502 01:52:58.420842 13833 catalog_manager.cc:5484] T aa387d2b1dcc42ddb34005f08dde2d8d P 1478241a03934d029dc7030480423cab reported cstate change: config changed from index 4 to 5, 95e7c90cdc634d64a66dad6805e1680c (127.11.42.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1478241a03934d029dc7030480423cab" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95e7c90cdc634d64a66dad6805e1680c" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 43981 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:52:58.439217 16125 raft_consensus.cc:1050] T 6eb1311f3c054aec9a306922f25c6b89 P 1478241a03934d029dc7030480423cab: attempting to promote NON_VOTER 1607e211ee8c48de9fa56e78a3b01418 to VOTER
I20240502 01:52:58.439688 16125 consensus_queue.cc:237] T 6eb1311f3c054aec9a306922f25c6b89 P 1478241a03934d029dc7030480423cab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "1607e211ee8c48de9fa56e78a3b01418" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 41779 } attrs { promote: false } }
I20240502 01:52:58.441458 15328 raft_consensus.cc:1261] T 6eb1311f3c054aec9a306922f25c6b89 P 1607e211ee8c48de9fa56e78a3b01418 [term 1 LEARNER]: Refusing update from remote peer 1478241a03934d029dc7030480423cab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:52:58.441956 16124 consensus_queue.cc:1035] T 6eb1311f3c054aec9a306922f25c6b89 P 1478241a03934d029dc7030480423cab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1607e211ee8c48de9fa56e78a3b01418" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 41779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:52:58.444998 16125 raft_consensus.cc:2932] T 6eb1311f3c054aec9a306922f25c6b89 P 1478241a03934d029dc7030480423cab [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1607e211ee8c48de9fa56e78a3b01418 (127.11.42.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "1607e211ee8c48de9fa56e78a3b01418" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 41779 } attrs { promote: false } } }
I20240502 01:52:58.445719 15328 raft_consensus.cc:2932] T 6eb1311f3c054aec9a306922f25c6b89 P 1607e211ee8c48de9fa56e78a3b01418 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1607e211ee8c48de9fa56e78a3b01418 (127.11.42.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "1607e211ee8c48de9fa56e78a3b01418" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 41779 } attrs { promote: false } } }
I20240502 01:52:58.448776 13834 catalog_manager.cc:5484] T 6eb1311f3c054aec9a306922f25c6b89 P 1478241a03934d029dc7030480423cab reported cstate change: config changed from index 4 to 5, 1607e211ee8c48de9fa56e78a3b01418 (127.11.42.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1478241a03934d029dc7030480423cab" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1607e211ee8c48de9fa56e78a3b01418" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 41779 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:52:58.465873 16126 raft_consensus.cc:1050] T 50496c7daf804faa8d7bfaf128839e2c P d43426b5bc874e8482e7b3205a9ba970: attempting to promote NON_VOTER 95e7c90cdc634d64a66dad6805e1680c to VOTER
I20240502 01:52:58.466457 16126 consensus_queue.cc:237] T 50496c7daf804faa8d7bfaf128839e2c P d43426b5bc874e8482e7b3205a9ba970 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "95e7c90cdc634d64a66dad6805e1680c" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 43981 } attrs { promote: false } }
I20240502 01:52:58.468096 15459 raft_consensus.cc:1261] T 50496c7daf804faa8d7bfaf128839e2c P 95e7c90cdc634d64a66dad6805e1680c [term 1 LEARNER]: Refusing update from remote peer d43426b5bc874e8482e7b3205a9ba970: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:52:58.468633 15924 consensus_queue.cc:1035] T 50496c7daf804faa8d7bfaf128839e2c P d43426b5bc874e8482e7b3205a9ba970 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95e7c90cdc634d64a66dad6805e1680c" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 43981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:52:58.471096 16126 raft_consensus.cc:2932] T 50496c7daf804faa8d7bfaf128839e2c P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 95e7c90cdc634d64a66dad6805e1680c (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "95e7c90cdc634d64a66dad6805e1680c" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 43981 } attrs { promote: false } } }
I20240502 01:52:58.471662 15459 raft_consensus.cc:2932] T 50496c7daf804faa8d7bfaf128839e2c P 95e7c90cdc634d64a66dad6805e1680c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 95e7c90cdc634d64a66dad6805e1680c (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "95e7c90cdc634d64a66dad6805e1680c" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 43981 } attrs { promote: false } } }
I20240502 01:52:58.476189 13833 catalog_manager.cc:5484] T 50496c7daf804faa8d7bfaf128839e2c P d43426b5bc874e8482e7b3205a9ba970 reported cstate change: config changed from index 4 to 5, 95e7c90cdc634d64a66dad6805e1680c (127.11.42.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d43426b5bc874e8482e7b3205a9ba970" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95e7c90cdc634d64a66dad6805e1680c" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 43981 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:52:58.505292 16127 raft_consensus.cc:1050] T c0620ccc47634258a7962d91cc7bb060 P d43426b5bc874e8482e7b3205a9ba970: attempting to promote NON_VOTER 1607e211ee8c48de9fa56e78a3b01418 to VOTER
I20240502 01:52:58.505924 16127 consensus_queue.cc:237] T c0620ccc47634258a7962d91cc7bb060 P d43426b5bc874e8482e7b3205a9ba970 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "1607e211ee8c48de9fa56e78a3b01418" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 41779 } attrs { promote: false } }
I20240502 01:52:58.507694 15328 raft_consensus.cc:1261] T c0620ccc47634258a7962d91cc7bb060 P 1607e211ee8c48de9fa56e78a3b01418 [term 1 LEARNER]: Refusing update from remote peer d43426b5bc874e8482e7b3205a9ba970: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:52:58.508152 15924 consensus_queue.cc:1035] T c0620ccc47634258a7962d91cc7bb060 P d43426b5bc874e8482e7b3205a9ba970 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1607e211ee8c48de9fa56e78a3b01418" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 41779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:52:58.510517 16127 raft_consensus.cc:2932] T c0620ccc47634258a7962d91cc7bb060 P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1607e211ee8c48de9fa56e78a3b01418 (127.11.42.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "1607e211ee8c48de9fa56e78a3b01418" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 41779 } attrs { promote: false } } }
I20240502 01:52:58.511063 15328 raft_consensus.cc:2932] T c0620ccc47634258a7962d91cc7bb060 P 1607e211ee8c48de9fa56e78a3b01418 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1607e211ee8c48de9fa56e78a3b01418 (127.11.42.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "1607e211ee8c48de9fa56e78a3b01418" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 41779 } attrs { promote: false } } }
I20240502 01:52:58.514957 13834 catalog_manager.cc:5484] T c0620ccc47634258a7962d91cc7bb060 P d43426b5bc874e8482e7b3205a9ba970 reported cstate change: config changed from index 4 to 5, 1607e211ee8c48de9fa56e78a3b01418 (127.11.42.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d43426b5bc874e8482e7b3205a9ba970" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1607e211ee8c48de9fa56e78a3b01418" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 41779 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:52:58.593153 13977 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "50496c7daf804faa8d7bfaf128839e2c"
dest_uuid: "d43426b5bc874e8482e7b3205a9ba970"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55892
I20240502 01:52:58.593407 13977 raft_consensus.cc:595] T 50496c7daf804faa8d7bfaf128839e2c P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:52:58.603220 16127 raft_consensus.cc:1050] T 6ed541d2e5ba4bbab19869c10116142e P d43426b5bc874e8482e7b3205a9ba970: attempting to promote NON_VOTER 39e5e5c1a5e74915800d96f720aee8c4 to VOTER
I20240502 01:52:58.603860 16127 consensus_queue.cc:237] T 6ed541d2e5ba4bbab19869c10116142e P d43426b5bc874e8482e7b3205a9ba970 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "39e5e5c1a5e74915800d96f720aee8c4" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 37071 } attrs { promote: false } }
I20240502 01:52:58.605736 15731 raft_consensus.cc:1261] T 6ed541d2e5ba4bbab19869c10116142e P 39e5e5c1a5e74915800d96f720aee8c4 [term 1 LEARNER]: Refusing update from remote peer d43426b5bc874e8482e7b3205a9ba970: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:52:58.606225 16126 consensus_queue.cc:1035] T 6ed541d2e5ba4bbab19869c10116142e P d43426b5bc874e8482e7b3205a9ba970 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39e5e5c1a5e74915800d96f720aee8c4" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 37071 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:52:58.608843 16127 raft_consensus.cc:2932] T 6ed541d2e5ba4bbab19869c10116142e P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 39e5e5c1a5e74915800d96f720aee8c4 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "39e5e5c1a5e74915800d96f720aee8c4" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 37071 } attrs { promote: false } } }
I20240502 01:52:58.609671 15731 raft_consensus.cc:2932] T 6ed541d2e5ba4bbab19869c10116142e P 39e5e5c1a5e74915800d96f720aee8c4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 39e5e5c1a5e74915800d96f720aee8c4 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "39e5e5c1a5e74915800d96f720aee8c4" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 37071 } attrs { promote: false } } }
I20240502 01:52:58.610060 14111 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6eb1311f3c054aec9a306922f25c6b89"
dest_uuid: "1478241a03934d029dc7030480423cab"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40384
I20240502 01:52:58.610210 14111 raft_consensus.cc:595] T 6eb1311f3c054aec9a306922f25c6b89 P 1478241a03934d029dc7030480423cab [term 1 LEADER]: Received request to transfer leadership
I20240502 01:52:58.614671 13833 catalog_manager.cc:5484] T 6ed541d2e5ba4bbab19869c10116142e P d43426b5bc874e8482e7b3205a9ba970 reported cstate change: config changed from index 4 to 5, 39e5e5c1a5e74915800d96f720aee8c4 (127.11.42.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d43426b5bc874e8482e7b3205a9ba970" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39e5e5c1a5e74915800d96f720aee8c4" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 37071 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:52:58.630920 14111 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aa387d2b1dcc42ddb34005f08dde2d8d"
dest_uuid: "1478241a03934d029dc7030480423cab"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40396
I20240502 01:52:58.631105 14111 raft_consensus.cc:595] T aa387d2b1dcc42ddb34005f08dde2d8d P 1478241a03934d029dc7030480423cab [term 1 LEADER]: Received request to transfer leadership
I20240502 01:52:58.645323 16126 raft_consensus.cc:1050] T 0eb60206b8cb40d3b30cdc1f47d67f94 P d43426b5bc874e8482e7b3205a9ba970: attempting to promote NON_VOTER 020e5f8ebb004e2ab6fddaece6663ece to VOTER
I20240502 01:52:58.646019 16126 consensus_queue.cc:237] T 0eb60206b8cb40d3b30cdc1f47d67f94 P d43426b5bc874e8482e7b3205a9ba970 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "020e5f8ebb004e2ab6fddaece6663ece" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 37505 } attrs { promote: false } }
I20240502 01:52:58.646569 13977 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c0620ccc47634258a7962d91cc7bb060"
dest_uuid: "d43426b5bc874e8482e7b3205a9ba970"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55922
I20240502 01:52:58.646765 13977 raft_consensus.cc:595] T c0620ccc47634258a7962d91cc7bb060 P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:52:58.648625 15196 raft_consensus.cc:1261] T 0eb60206b8cb40d3b30cdc1f47d67f94 P 020e5f8ebb004e2ab6fddaece6663ece [term 1 LEARNER]: Refusing update from remote peer d43426b5bc874e8482e7b3205a9ba970: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:52:58.649194 16127 consensus_queue.cc:1035] T 0eb60206b8cb40d3b30cdc1f47d67f94 P d43426b5bc874e8482e7b3205a9ba970 [LEADER]: Connected to new peer: Peer: permanent_uuid: "020e5f8ebb004e2ab6fddaece6663ece" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 37505 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:52:58.651798 16126 raft_consensus.cc:2932] T 0eb60206b8cb40d3b30cdc1f47d67f94 P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 020e5f8ebb004e2ab6fddaece6663ece (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "020e5f8ebb004e2ab6fddaece6663ece" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 37505 } attrs { promote: false } } }
I20240502 01:52:58.652474 15196 raft_consensus.cc:2932] T 0eb60206b8cb40d3b30cdc1f47d67f94 P 020e5f8ebb004e2ab6fddaece6663ece [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 020e5f8ebb004e2ab6fddaece6663ece (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "020e5f8ebb004e2ab6fddaece6663ece" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 37505 } attrs { promote: false } } }
I20240502 01:52:58.656883 13834 catalog_manager.cc:5484] T 0eb60206b8cb40d3b30cdc1f47d67f94 P d43426b5bc874e8482e7b3205a9ba970 reported cstate change: config changed from index 4 to 5, 020e5f8ebb004e2ab6fddaece6663ece (127.11.42.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d43426b5bc874e8482e7b3205a9ba970" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "020e5f8ebb004e2ab6fddaece6663ece" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 37505 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:52:58.666908 14111 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5bc0ff324caa436196f22d90360c9247"
dest_uuid: "1478241a03934d029dc7030480423cab"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40420
I20240502 01:52:58.667068 14111 raft_consensus.cc:595] T 5bc0ff324caa436196f22d90360c9247 P 1478241a03934d029dc7030480423cab [term 1 LEADER]: Received request to transfer leadership
I20240502 01:52:58.683321 14111 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f9f4aab07f984168bdfec24f908c124d"
dest_uuid: "1478241a03934d029dc7030480423cab"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40444
I20240502 01:52:58.683486 14111 raft_consensus.cc:595] T f9f4aab07f984168bdfec24f908c124d P 1478241a03934d029dc7030480423cab [term 1 LEADER]: Received request to transfer leadership
I20240502 01:52:58.698544 13977 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "67bc0cd4c3b94782ac1d9f150fcdbd03"
dest_uuid: "d43426b5bc874e8482e7b3205a9ba970"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55944
I20240502 01:52:58.698777 13977 raft_consensus.cc:595] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:52:58.713630 13977 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0eb60206b8cb40d3b30cdc1f47d67f94"
dest_uuid: "d43426b5bc874e8482e7b3205a9ba970"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55964
I20240502 01:52:58.713840 13977 raft_consensus.cc:595] T 0eb60206b8cb40d3b30cdc1f47d67f94 P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:52:58.815107 16126 raft_consensus.cc:979] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P d43426b5bc874e8482e7b3205a9ba970: : Instructing follower 39e5e5c1a5e74915800d96f720aee8c4 to start an election
I20240502 01:52:58.815297 16126 raft_consensus.cc:1067] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Signalling peer 39e5e5c1a5e74915800d96f720aee8c4 to start an election
I20240502 01:52:58.816234 15731 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "67bc0cd4c3b94782ac1d9f150fcdbd03"
dest_uuid: "39e5e5c1a5e74915800d96f720aee8c4"
from {username='slave'} at 127.11.42.65:50349
I20240502 01:52:58.816488 15731 raft_consensus.cc:482] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P 39e5e5c1a5e74915800d96f720aee8c4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:52:58.816681 15731 raft_consensus.cc:3037] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P 39e5e5c1a5e74915800d96f720aee8c4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:58.818748 15731 raft_consensus.cc:504] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P 39e5e5c1a5e74915800d96f720aee8c4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "39e5e5c1a5e74915800d96f720aee8c4" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 37071 } attrs { promote: false } }
I20240502 01:52:58.819991 15731 leader_election.cc:288] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P 39e5e5c1a5e74915800d96f720aee8c4 [CANDIDATE]: Term 2 election: Requested vote from peers d43426b5bc874e8482e7b3205a9ba970 (127.11.42.65:40383)
I20240502 01:52:58.820564 13977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67bc0cd4c3b94782ac1d9f150fcdbd03" candidate_uuid: "39e5e5c1a5e74915800d96f720aee8c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "d43426b5bc874e8482e7b3205a9ba970"
I20240502 01:52:58.820868 13977 raft_consensus.cc:3032] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:52:58.821038 13977 raft_consensus.cc:729] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d43426b5bc874e8482e7b3205a9ba970, State: Running, Role: LEADER
I20240502 01:52:58.821420 13977 consensus_queue.cc:260] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P d43426b5bc874e8482e7b3205a9ba970 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "39e5e5c1a5e74915800d96f720aee8c4" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 37071 } attrs { promote: false } }
I20240502 01:52:58.821825 13977 raft_consensus.cc:3037] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P d43426b5bc874e8482e7b3205a9ba970 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:58.823905 13977 raft_consensus.cc:2454] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P d43426b5bc874e8482e7b3205a9ba970 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39e5e5c1a5e74915800d96f720aee8c4 in term 2.
I20240502 01:52:58.824425 15664 leader_election.cc:302] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P 39e5e5c1a5e74915800d96f720aee8c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 39e5e5c1a5e74915800d96f720aee8c4, d43426b5bc874e8482e7b3205a9ba970; no voters:
I20240502 01:52:58.824749 16098 raft_consensus.cc:2781] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P 39e5e5c1a5e74915800d96f720aee8c4 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:52:58.825194 16098 raft_consensus.cc:686] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P 39e5e5c1a5e74915800d96f720aee8c4 [term 2 LEADER]: Becoming Leader. State: Replica: 39e5e5c1a5e74915800d96f720aee8c4, State: Running, Role: LEADER
I20240502 01:52:58.825634 16098 consensus_queue.cc:237] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P 39e5e5c1a5e74915800d96f720aee8c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "39e5e5c1a5e74915800d96f720aee8c4" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 37071 } attrs { promote: false } }
I20240502 01:52:58.829196 13833 catalog_manager.cc:5484] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P 39e5e5c1a5e74915800d96f720aee8c4 reported cstate change: term changed from 1 to 2, leader changed from d43426b5bc874e8482e7b3205a9ba970 (127.11.42.65) to 39e5e5c1a5e74915800d96f720aee8c4 (127.11.42.71). New cstate: current_term: 2 leader_uuid: "39e5e5c1a5e74915800d96f720aee8c4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39e5e5c1a5e74915800d96f720aee8c4" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 37071 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:52:58.833773 15866 raft_consensus.cc:979] T 6eb1311f3c054aec9a306922f25c6b89 P 1478241a03934d029dc7030480423cab: : Instructing follower 1607e211ee8c48de9fa56e78a3b01418 to start an election
I20240502 01:52:58.833966 15866 raft_consensus.cc:1067] T 6eb1311f3c054aec9a306922f25c6b89 P 1478241a03934d029dc7030480423cab [term 1 LEADER]: Signalling peer 1607e211ee8c48de9fa56e78a3b01418 to start an election
I20240502 01:52:58.834843 15328 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6eb1311f3c054aec9a306922f25c6b89"
dest_uuid: "1607e211ee8c48de9fa56e78a3b01418"
from {username='slave'} at 127.11.42.66:35205
I20240502 01:52:58.835074 15328 raft_consensus.cc:482] T 6eb1311f3c054aec9a306922f25c6b89 P 1607e211ee8c48de9fa56e78a3b01418 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:52:58.835239 15328 raft_consensus.cc:3037] T 6eb1311f3c054aec9a306922f25c6b89 P 1607e211ee8c48de9fa56e78a3b01418 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:58.837672 15328 raft_consensus.cc:504] T 6eb1311f3c054aec9a306922f25c6b89 P 1607e211ee8c48de9fa56e78a3b01418 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "1607e211ee8c48de9fa56e78a3b01418" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 41779 } attrs { promote: false } }
I20240502 01:52:58.839854 15328 leader_election.cc:288] T 6eb1311f3c054aec9a306922f25c6b89 P 1607e211ee8c48de9fa56e78a3b01418 [CANDIDATE]: Term 2 election: Requested vote from peers 1478241a03934d029dc7030480423cab (127.11.42.66:34613)
I20240502 01:52:58.839962 14111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eb1311f3c054aec9a306922f25c6b89" candidate_uuid: "1607e211ee8c48de9fa56e78a3b01418" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "1478241a03934d029dc7030480423cab"
I20240502 01:52:58.840309 14111 raft_consensus.cc:3032] T 6eb1311f3c054aec9a306922f25c6b89 P 1478241a03934d029dc7030480423cab [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:52:58.840430 14111 raft_consensus.cc:729] T 6eb1311f3c054aec9a306922f25c6b89 P 1478241a03934d029dc7030480423cab [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1478241a03934d029dc7030480423cab, State: Running, Role: LEADER
I20240502 01:52:58.840727 14111 consensus_queue.cc:260] T 6eb1311f3c054aec9a306922f25c6b89 P 1478241a03934d029dc7030480423cab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "1607e211ee8c48de9fa56e78a3b01418" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 41779 } attrs { promote: false } }
I20240502 01:52:58.841140 14111 raft_consensus.cc:3037] T 6eb1311f3c054aec9a306922f25c6b89 P 1478241a03934d029dc7030480423cab [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:52:58.841890 13819 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 67bc0cd4c3b94782ac1d9f150fcdbd03 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 39e5e5c1a5e74915800d96f720aee8c4 (127.11.42.71:37071) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240502 01:52:58.843483 14111 raft_consensus.cc:2454] T 6eb1311f3c054aec9a306922f25c6b89 P 1478241a03934d029dc7030480423cab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1607e211ee8c48de9fa56e78a3b01418 in term 2.
I20240502 01:52:58.843967 15263 leader_election.cc:302] T 6eb1311f3c054aec9a306922f25c6b89 P 1607e211ee8c48de9fa56e78a3b01418 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1478241a03934d029dc7030480423cab, 1607e211ee8c48de9fa56e78a3b01418; no voters:
I20240502 01:52:58.844240 16125 raft_consensus.cc:979] T f9f4aab07f984168bdfec24f908c124d P 1478241a03934d029dc7030480423cab: : Instructing follower 2ee638444633463b943c881866708482 to start an election
I20240502 01:52:58.844401 16068 raft_consensus.cc:2781] T 6eb1311f3c054aec9a306922f25c6b89 P 1607e211ee8c48de9fa56e78a3b01418 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:52:58.844417 16125 raft_consensus.cc:1067] T f9f4aab07f984168bdfec24f908c124d P 1478241a03934d029dc7030480423cab [term 1 LEADER]: Signalling peer 2ee638444633463b943c881866708482 to start an election
I20240502 01:52:58.844938 16068 raft_consensus.cc:686] T 6eb1311f3c054aec9a306922f25c6b89 P 1607e211ee8c48de9fa56e78a3b01418 [term 2 LEADER]: Becoming Leader. State: Replica: 1607e211ee8c48de9fa56e78a3b01418, State: Running, Role: LEADER
I20240502 01:52:58.845193 15592 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f9f4aab07f984168bdfec24f908c124d"
dest_uuid: "2ee638444633463b943c881866708482"
from {username='slave'} at 127.11.42.66:51395
I20240502 01:52:58.845489 15592 raft_consensus.cc:482] T f9f4aab07f984168bdfec24f908c124d P 2ee638444633463b943c881866708482 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:52:58.845446 16068 consensus_queue.cc:237] T 6eb1311f3c054aec9a306922f25c6b89 P 1607e211ee8c48de9fa56e78a3b01418 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "1607e211ee8c48de9fa56e78a3b01418" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 41779 } attrs { promote: false } }
I20240502 01:52:58.845590 15592 raft_consensus.cc:3037] T f9f4aab07f984168bdfec24f908c124d P 2ee638444633463b943c881866708482 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:58.847793 15592 raft_consensus.cc:504] T f9f4aab07f984168bdfec24f908c124d P 2ee638444633463b943c881866708482 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "2ee638444633463b943c881866708482" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 45255 } attrs { promote: false } }
I20240502 01:52:58.848802 15592 leader_election.cc:288] T f9f4aab07f984168bdfec24f908c124d P 2ee638444633463b943c881866708482 [CANDIDATE]: Term 2 election: Requested vote from peers 1478241a03934d029dc7030480423cab (127.11.42.66:34613)
I20240502 01:52:58.848779 13833 catalog_manager.cc:5484] T 6eb1311f3c054aec9a306922f25c6b89 P 1607e211ee8c48de9fa56e78a3b01418 reported cstate change: term changed from 1 to 2, leader changed from 1478241a03934d029dc7030480423cab (127.11.42.66) to 1607e211ee8c48de9fa56e78a3b01418 (127.11.42.68). New cstate: current_term: 2 leader_uuid: "1607e211ee8c48de9fa56e78a3b01418" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1607e211ee8c48de9fa56e78a3b01418" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 41779 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:52:58.849329 14111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9f4aab07f984168bdfec24f908c124d" candidate_uuid: "2ee638444633463b943c881866708482" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "1478241a03934d029dc7030480423cab"
I20240502 01:52:58.849467 14111 raft_consensus.cc:3032] T f9f4aab07f984168bdfec24f908c124d P 1478241a03934d029dc7030480423cab [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:52:58.849529 14111 raft_consensus.cc:729] T f9f4aab07f984168bdfec24f908c124d P 1478241a03934d029dc7030480423cab [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1478241a03934d029dc7030480423cab, State: Running, Role: LEADER
I20240502 01:52:58.849714 14111 consensus_queue.cc:260] T f9f4aab07f984168bdfec24f908c124d P 1478241a03934d029dc7030480423cab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "2ee638444633463b943c881866708482" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 45255 } attrs { promote: false } }
I20240502 01:52:58.850023 14111 raft_consensus.cc:3037] T f9f4aab07f984168bdfec24f908c124d P 1478241a03934d029dc7030480423cab [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:58.852084 14111 raft_consensus.cc:2454] T f9f4aab07f984168bdfec24f908c124d P 1478241a03934d029dc7030480423cab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ee638444633463b943c881866708482 in term 2.
I20240502 01:52:58.852562 15525 leader_election.cc:302] T f9f4aab07f984168bdfec24f908c124d P 2ee638444633463b943c881866708482 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1478241a03934d029dc7030480423cab, 2ee638444633463b943c881866708482; no voters:
I20240502 01:52:58.853159 16335 raft_consensus.cc:2781] T f9f4aab07f984168bdfec24f908c124d P 2ee638444633463b943c881866708482 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:52:58.853556 16335 raft_consensus.cc:686] T f9f4aab07f984168bdfec24f908c124d P 2ee638444633463b943c881866708482 [term 2 LEADER]: Becoming Leader. State: Replica: 2ee638444633463b943c881866708482, State: Running, Role: LEADER
I20240502 01:52:58.853885 16335 consensus_queue.cc:237] T f9f4aab07f984168bdfec24f908c124d P 2ee638444633463b943c881866708482 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "2ee638444633463b943c881866708482" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 45255 } attrs { promote: false } }
I20240502 01:52:58.856694 13833 catalog_manager.cc:5484] T f9f4aab07f984168bdfec24f908c124d P 2ee638444633463b943c881866708482 reported cstate change: term changed from 1 to 2, leader changed from 1478241a03934d029dc7030480423cab (127.11.42.66) to 2ee638444633463b943c881866708482 (127.11.42.70). New cstate: current_term: 2 leader_uuid: "2ee638444633463b943c881866708482" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ee638444633463b943c881866708482" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 45255 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:52:58.869642 16127 raft_consensus.cc:979] T 50496c7daf804faa8d7bfaf128839e2c P d43426b5bc874e8482e7b3205a9ba970: : Instructing follower 95e7c90cdc634d64a66dad6805e1680c to start an election
I20240502 01:52:58.869793 16127 raft_consensus.cc:1067] T 50496c7daf804faa8d7bfaf128839e2c P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Signalling peer 95e7c90cdc634d64a66dad6805e1680c to start an election
I20240502 01:52:58.870447 15459 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "50496c7daf804faa8d7bfaf128839e2c"
dest_uuid: "95e7c90cdc634d64a66dad6805e1680c"
from {username='slave'} at 127.11.42.65:53955
I20240502 01:52:58.870687 15459 raft_consensus.cc:482] T 50496c7daf804faa8d7bfaf128839e2c P 95e7c90cdc634d64a66dad6805e1680c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:52:58.870862 15459 raft_consensus.cc:3037] T 50496c7daf804faa8d7bfaf128839e2c P 95e7c90cdc634d64a66dad6805e1680c [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:58.872713 15459 raft_consensus.cc:504] T 50496c7daf804faa8d7bfaf128839e2c P 95e7c90cdc634d64a66dad6805e1680c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "95e7c90cdc634d64a66dad6805e1680c" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 43981 } attrs { promote: false } }
I20240502 01:52:58.873838 15459 leader_election.cc:288] T 50496c7daf804faa8d7bfaf128839e2c P 95e7c90cdc634d64a66dad6805e1680c [CANDIDATE]: Term 2 election: Requested vote from peers d43426b5bc874e8482e7b3205a9ba970 (127.11.42.65:40383)
I20240502 01:52:58.874305 13977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50496c7daf804faa8d7bfaf128839e2c" candidate_uuid: "95e7c90cdc634d64a66dad6805e1680c" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "d43426b5bc874e8482e7b3205a9ba970"
I20240502 01:52:58.874501 13977 raft_consensus.cc:3032] T 50496c7daf804faa8d7bfaf128839e2c P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:52:58.874588 13977 raft_consensus.cc:729] T 50496c7daf804faa8d7bfaf128839e2c P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d43426b5bc874e8482e7b3205a9ba970, State: Running, Role: LEADER
I20240502 01:52:58.874850 13977 consensus_queue.cc:260] T 50496c7daf804faa8d7bfaf128839e2c P d43426b5bc874e8482e7b3205a9ba970 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "95e7c90cdc634d64a66dad6805e1680c" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 43981 } attrs { promote: false } }
I20240502 01:52:58.875095 13977 raft_consensus.cc:3037] T 50496c7daf804faa8d7bfaf128839e2c P d43426b5bc874e8482e7b3205a9ba970 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:58.877162 13977 raft_consensus.cc:2454] T 50496c7daf804faa8d7bfaf128839e2c P d43426b5bc874e8482e7b3205a9ba970 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95e7c90cdc634d64a66dad6805e1680c in term 2.
I20240502 01:52:58.877228 16125 raft_consensus.cc:979] T aa387d2b1dcc42ddb34005f08dde2d8d P 1478241a03934d029dc7030480423cab: : Instructing follower 95e7c90cdc634d64a66dad6805e1680c to start an election
I20240502 01:52:58.877418 15866 raft_consensus.cc:1067] T aa387d2b1dcc42ddb34005f08dde2d8d P 1478241a03934d029dc7030480423cab [term 1 LEADER]: Signalling peer 95e7c90cdc634d64a66dad6805e1680c to start an election
I20240502 01:52:58.877676 15392 leader_election.cc:302] T 50496c7daf804faa8d7bfaf128839e2c P 95e7c90cdc634d64a66dad6805e1680c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 95e7c90cdc634d64a66dad6805e1680c, d43426b5bc874e8482e7b3205a9ba970; no voters:
I20240502 01:52:58.878074 16081 raft_consensus.cc:2781] T 50496c7daf804faa8d7bfaf128839e2c P 95e7c90cdc634d64a66dad6805e1680c [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:52:58.878096 15459 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aa387d2b1dcc42ddb34005f08dde2d8d"
dest_uuid: "95e7c90cdc634d64a66dad6805e1680c"
from {username='slave'} at 127.11.42.66:42165
I20240502 01:52:58.878297 15459 raft_consensus.cc:482] T aa387d2b1dcc42ddb34005f08dde2d8d P 95e7c90cdc634d64a66dad6805e1680c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:52:58.878418 15459 raft_consensus.cc:3037] T aa387d2b1dcc42ddb34005f08dde2d8d P 95e7c90cdc634d64a66dad6805e1680c [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:58.878536 16081 raft_consensus.cc:686] T 50496c7daf804faa8d7bfaf128839e2c P 95e7c90cdc634d64a66dad6805e1680c [term 2 LEADER]: Becoming Leader. State: Replica: 95e7c90cdc634d64a66dad6805e1680c, State: Running, Role: LEADER
I20240502 01:52:58.879081 16081 consensus_queue.cc:237] T 50496c7daf804faa8d7bfaf128839e2c P 95e7c90cdc634d64a66dad6805e1680c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "95e7c90cdc634d64a66dad6805e1680c" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 43981 } attrs { promote: false } }
I20240502 01:52:58.881157 15459 raft_consensus.cc:504] T aa387d2b1dcc42ddb34005f08dde2d8d P 95e7c90cdc634d64a66dad6805e1680c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "95e7c90cdc634d64a66dad6805e1680c" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 43981 } attrs { promote: false } }
I20240502 01:52:58.881623 15459 leader_election.cc:288] T aa387d2b1dcc42ddb34005f08dde2d8d P 95e7c90cdc634d64a66dad6805e1680c [CANDIDATE]: Term 2 election: Requested vote from peers 1478241a03934d029dc7030480423cab (127.11.42.66:34613)
I20240502 01:52:58.882174 14111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa387d2b1dcc42ddb34005f08dde2d8d" candidate_uuid: "95e7c90cdc634d64a66dad6805e1680c" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "1478241a03934d029dc7030480423cab"
I20240502 01:52:58.882372 14111 raft_consensus.cc:3032] T aa387d2b1dcc42ddb34005f08dde2d8d P 1478241a03934d029dc7030480423cab [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:52:58.882465 14111 raft_consensus.cc:729] T aa387d2b1dcc42ddb34005f08dde2d8d P 1478241a03934d029dc7030480423cab [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1478241a03934d029dc7030480423cab, State: Running, Role: LEADER
I20240502 01:52:58.882716 14111 consensus_queue.cc:260] T aa387d2b1dcc42ddb34005f08dde2d8d P 1478241a03934d029dc7030480423cab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "95e7c90cdc634d64a66dad6805e1680c" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 43981 } attrs { promote: false } }
I20240502 01:52:58.882977 14111 raft_consensus.cc:3037] T aa387d2b1dcc42ddb34005f08dde2d8d P 1478241a03934d029dc7030480423cab [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:58.882997 13833 catalog_manager.cc:5484] T 50496c7daf804faa8d7bfaf128839e2c P 95e7c90cdc634d64a66dad6805e1680c reported cstate change: term changed from 1 to 2, leader changed from d43426b5bc874e8482e7b3205a9ba970 (127.11.42.65) to 95e7c90cdc634d64a66dad6805e1680c (127.11.42.69). New cstate: current_term: 2 leader_uuid: "95e7c90cdc634d64a66dad6805e1680c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95e7c90cdc634d64a66dad6805e1680c" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 43981 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:52:58.884742 14111 raft_consensus.cc:2454] T aa387d2b1dcc42ddb34005f08dde2d8d P 1478241a03934d029dc7030480423cab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95e7c90cdc634d64a66dad6805e1680c in term 2.
I20240502 01:52:58.885166 15394 leader_election.cc:302] T aa387d2b1dcc42ddb34005f08dde2d8d P 95e7c90cdc634d64a66dad6805e1680c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1478241a03934d029dc7030480423cab, 95e7c90cdc634d64a66dad6805e1680c; no voters:
I20240502 01:52:58.885413 16081 raft_consensus.cc:2781] T aa387d2b1dcc42ddb34005f08dde2d8d P 95e7c90cdc634d64a66dad6805e1680c [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:52:58.885560 16081 raft_consensus.cc:686] T aa387d2b1dcc42ddb34005f08dde2d8d P 95e7c90cdc634d64a66dad6805e1680c [term 2 LEADER]: Becoming Leader. State: Replica: 95e7c90cdc634d64a66dad6805e1680c, State: Running, Role: LEADER
I20240502 01:52:58.885751 16081 consensus_queue.cc:237] T aa387d2b1dcc42ddb34005f08dde2d8d P 95e7c90cdc634d64a66dad6805e1680c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "95e7c90cdc634d64a66dad6805e1680c" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 43981 } attrs { promote: false } }
I20240502 01:52:58.888235 13833 catalog_manager.cc:5484] T aa387d2b1dcc42ddb34005f08dde2d8d P 95e7c90cdc634d64a66dad6805e1680c reported cstate change: term changed from 1 to 2, leader changed from 1478241a03934d029dc7030480423cab (127.11.42.66) to 95e7c90cdc634d64a66dad6805e1680c (127.11.42.69). New cstate: current_term: 2 leader_uuid: "95e7c90cdc634d64a66dad6805e1680c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95e7c90cdc634d64a66dad6805e1680c" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 43981 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:52:58.929533 13977 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6ed541d2e5ba4bbab19869c10116142e"
dest_uuid: "d43426b5bc874e8482e7b3205a9ba970"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55980
I20240502 01:52:58.929733 13977 raft_consensus.cc:595] T 6ed541d2e5ba4bbab19869c10116142e P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:52:58.963322 16125 raft_consensus.cc:979] T 5bc0ff324caa436196f22d90360c9247 P 1478241a03934d029dc7030480423cab: : Instructing follower 020e5f8ebb004e2ab6fddaece6663ece to start an election
I20240502 01:52:58.963485 16125 raft_consensus.cc:1067] T 5bc0ff324caa436196f22d90360c9247 P 1478241a03934d029dc7030480423cab [term 1 LEADER]: Signalling peer 020e5f8ebb004e2ab6fddaece6663ece to start an election
I20240502 01:52:58.964156 15196 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5bc0ff324caa436196f22d90360c9247"
dest_uuid: "020e5f8ebb004e2ab6fddaece6663ece"
from {username='slave'} at 127.11.42.66:52847
I20240502 01:52:58.964427 15196 raft_consensus.cc:482] T 5bc0ff324caa436196f22d90360c9247 P 020e5f8ebb004e2ab6fddaece6663ece [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:52:58.964618 15196 raft_consensus.cc:3037] T 5bc0ff324caa436196f22d90360c9247 P 020e5f8ebb004e2ab6fddaece6663ece [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:58.966980 15196 raft_consensus.cc:504] T 5bc0ff324caa436196f22d90360c9247 P 020e5f8ebb004e2ab6fddaece6663ece [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "020e5f8ebb004e2ab6fddaece6663ece" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 37505 } attrs { promote: false } }
I20240502 01:52:58.968312 15196 leader_election.cc:288] T 5bc0ff324caa436196f22d90360c9247 P 020e5f8ebb004e2ab6fddaece6663ece [CANDIDATE]: Term 2 election: Requested vote from peers 1478241a03934d029dc7030480423cab (127.11.42.66:34613)
I20240502 01:52:58.968981 14111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc0ff324caa436196f22d90360c9247" candidate_uuid: "020e5f8ebb004e2ab6fddaece6663ece" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "1478241a03934d029dc7030480423cab"
I20240502 01:52:58.969224 14111 raft_consensus.cc:3032] T 5bc0ff324caa436196f22d90360c9247 P 1478241a03934d029dc7030480423cab [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:52:58.969318 14111 raft_consensus.cc:729] T 5bc0ff324caa436196f22d90360c9247 P 1478241a03934d029dc7030480423cab [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1478241a03934d029dc7030480423cab, State: Running, Role: LEADER
I20240502 01:52:58.969589 14111 consensus_queue.cc:260] T 5bc0ff324caa436196f22d90360c9247 P 1478241a03934d029dc7030480423cab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "020e5f8ebb004e2ab6fddaece6663ece" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 37505 } attrs { promote: false } }
I20240502 01:52:58.969930 14111 raft_consensus.cc:3037] T 5bc0ff324caa436196f22d90360c9247 P 1478241a03934d029dc7030480423cab [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:58.971729 14111 raft_consensus.cc:2454] T 5bc0ff324caa436196f22d90360c9247 P 1478241a03934d029dc7030480423cab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 020e5f8ebb004e2ab6fddaece6663ece in term 2.
I20240502 01:52:58.972213 15131 leader_election.cc:302] T 5bc0ff324caa436196f22d90360c9247 P 020e5f8ebb004e2ab6fddaece6663ece [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 020e5f8ebb004e2ab6fddaece6663ece, 1478241a03934d029dc7030480423cab; no voters:
I20240502 01:52:58.972545 16029 raft_consensus.cc:2781] T 5bc0ff324caa436196f22d90360c9247 P 020e5f8ebb004e2ab6fddaece6663ece [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:52:58.972970 16029 raft_consensus.cc:686] T 5bc0ff324caa436196f22d90360c9247 P 020e5f8ebb004e2ab6fddaece6663ece [term 2 LEADER]: Becoming Leader. State: Replica: 020e5f8ebb004e2ab6fddaece6663ece, State: Running, Role: LEADER
I20240502 01:52:58.973479 16029 consensus_queue.cc:237] T 5bc0ff324caa436196f22d90360c9247 P 020e5f8ebb004e2ab6fddaece6663ece [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "020e5f8ebb004e2ab6fddaece6663ece" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 37505 } attrs { promote: false } }
I20240502 01:52:58.977082 13833 catalog_manager.cc:5484] T 5bc0ff324caa436196f22d90360c9247 P 020e5f8ebb004e2ab6fddaece6663ece reported cstate change: term changed from 1 to 2, leader changed from 1478241a03934d029dc7030480423cab (127.11.42.66) to 020e5f8ebb004e2ab6fddaece6663ece (127.11.42.67). New cstate: current_term: 2 leader_uuid: "020e5f8ebb004e2ab6fddaece6663ece" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "020e5f8ebb004e2ab6fddaece6663ece" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 37505 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:52:58.978152 13977 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c0620ccc47634258a7962d91cc7bb060"
dest_uuid: "d43426b5bc874e8482e7b3205a9ba970"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56012
I20240502 01:52:58.978349 13977 raft_consensus.cc:595] T c0620ccc47634258a7962d91cc7bb060 P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:52:59.020758 13977 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0eb60206b8cb40d3b30cdc1f47d67f94"
dest_uuid: "d43426b5bc874e8482e7b3205a9ba970"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56026
I20240502 01:52:59.020956 13977 raft_consensus.cc:595] T 0eb60206b8cb40d3b30cdc1f47d67f94 P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:52:59.067099 16126 raft_consensus.cc:979] T c0620ccc47634258a7962d91cc7bb060 P d43426b5bc874e8482e7b3205a9ba970: : Instructing follower 1607e211ee8c48de9fa56e78a3b01418 to start an election
I20240502 01:52:59.067286 16126 raft_consensus.cc:1067] T c0620ccc47634258a7962d91cc7bb060 P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Signalling peer 1607e211ee8c48de9fa56e78a3b01418 to start an election
I20240502 01:52:59.067870 15328 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c0620ccc47634258a7962d91cc7bb060"
dest_uuid: "1607e211ee8c48de9fa56e78a3b01418"
from {username='slave'} at 127.11.42.65:45385
I20240502 01:52:59.068073 15328 raft_consensus.cc:482] T c0620ccc47634258a7962d91cc7bb060 P 1607e211ee8c48de9fa56e78a3b01418 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:52:59.068187 15328 raft_consensus.cc:3037] T c0620ccc47634258a7962d91cc7bb060 P 1607e211ee8c48de9fa56e78a3b01418 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:59.070178 15328 raft_consensus.cc:504] T c0620ccc47634258a7962d91cc7bb060 P 1607e211ee8c48de9fa56e78a3b01418 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "1607e211ee8c48de9fa56e78a3b01418" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 41779 } attrs { promote: false } }
I20240502 01:52:59.070737 15328 leader_election.cc:288] T c0620ccc47634258a7962d91cc7bb060 P 1607e211ee8c48de9fa56e78a3b01418 [CANDIDATE]: Term 2 election: Requested vote from peers d43426b5bc874e8482e7b3205a9ba970 (127.11.42.65:40383)
I20240502 01:52:59.071208 13977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0620ccc47634258a7962d91cc7bb060" candidate_uuid: "1607e211ee8c48de9fa56e78a3b01418" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "d43426b5bc874e8482e7b3205a9ba970"
I20240502 01:52:59.071393 13977 raft_consensus.cc:3032] T c0620ccc47634258a7962d91cc7bb060 P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:52:59.071466 13977 raft_consensus.cc:729] T c0620ccc47634258a7962d91cc7bb060 P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d43426b5bc874e8482e7b3205a9ba970, State: Running, Role: LEADER
I20240502 01:52:59.071712 13977 consensus_queue.cc:260] T c0620ccc47634258a7962d91cc7bb060 P d43426b5bc874e8482e7b3205a9ba970 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "1607e211ee8c48de9fa56e78a3b01418" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 41779 } attrs { promote: false } }
I20240502 01:52:59.071964 13977 raft_consensus.cc:3037] T c0620ccc47634258a7962d91cc7bb060 P d43426b5bc874e8482e7b3205a9ba970 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:59.073779 13977 raft_consensus.cc:2454] T c0620ccc47634258a7962d91cc7bb060 P d43426b5bc874e8482e7b3205a9ba970 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1607e211ee8c48de9fa56e78a3b01418 in term 2.
I20240502 01:52:59.074151 15261 leader_election.cc:302] T c0620ccc47634258a7962d91cc7bb060 P 1607e211ee8c48de9fa56e78a3b01418 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1607e211ee8c48de9fa56e78a3b01418, d43426b5bc874e8482e7b3205a9ba970; no voters:
I20240502 01:52:59.074416 16068 raft_consensus.cc:2781] T c0620ccc47634258a7962d91cc7bb060 P 1607e211ee8c48de9fa56e78a3b01418 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:52:59.074573 16068 raft_consensus.cc:686] T c0620ccc47634258a7962d91cc7bb060 P 1607e211ee8c48de9fa56e78a3b01418 [term 2 LEADER]: Becoming Leader. State: Replica: 1607e211ee8c48de9fa56e78a3b01418, State: Running, Role: LEADER
I20240502 01:52:59.074810 16068 consensus_queue.cc:237] T c0620ccc47634258a7962d91cc7bb060 P 1607e211ee8c48de9fa56e78a3b01418 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "1607e211ee8c48de9fa56e78a3b01418" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 41779 } attrs { promote: false } }
I20240502 01:52:59.076745 13833 catalog_manager.cc:5484] T c0620ccc47634258a7962d91cc7bb060 P 1607e211ee8c48de9fa56e78a3b01418 reported cstate change: term changed from 1 to 2, leader changed from d43426b5bc874e8482e7b3205a9ba970 (127.11.42.65) to 1607e211ee8c48de9fa56e78a3b01418 (127.11.42.68). New cstate: current_term: 2 leader_uuid: "1607e211ee8c48de9fa56e78a3b01418" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1607e211ee8c48de9fa56e78a3b01418" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 41779 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:52:59.151787 16127 raft_consensus.cc:979] T 0eb60206b8cb40d3b30cdc1f47d67f94 P d43426b5bc874e8482e7b3205a9ba970: : Instructing follower 020e5f8ebb004e2ab6fddaece6663ece to start an election
I20240502 01:52:59.151997 16126 raft_consensus.cc:1067] T 0eb60206b8cb40d3b30cdc1f47d67f94 P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Signalling peer 020e5f8ebb004e2ab6fddaece6663ece to start an election
I20240502 01:52:59.152624 15196 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0eb60206b8cb40d3b30cdc1f47d67f94"
dest_uuid: "020e5f8ebb004e2ab6fddaece6663ece"
from {username='slave'} at 127.11.42.65:38695
I20240502 01:52:59.152840 15196 raft_consensus.cc:482] T 0eb60206b8cb40d3b30cdc1f47d67f94 P 020e5f8ebb004e2ab6fddaece6663ece [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:52:59.152940 15196 raft_consensus.cc:3037] T 0eb60206b8cb40d3b30cdc1f47d67f94 P 020e5f8ebb004e2ab6fddaece6663ece [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:59.155035 15196 raft_consensus.cc:504] T 0eb60206b8cb40d3b30cdc1f47d67f94 P 020e5f8ebb004e2ab6fddaece6663ece [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "020e5f8ebb004e2ab6fddaece6663ece" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 37505 } attrs { promote: false } }
I20240502 01:52:59.155627 15196 leader_election.cc:288] T 0eb60206b8cb40d3b30cdc1f47d67f94 P 020e5f8ebb004e2ab6fddaece6663ece [CANDIDATE]: Term 2 election: Requested vote from peers d43426b5bc874e8482e7b3205a9ba970 (127.11.42.65:40383)
I20240502 01:52:59.156087 13977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb60206b8cb40d3b30cdc1f47d67f94" candidate_uuid: "020e5f8ebb004e2ab6fddaece6663ece" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "d43426b5bc874e8482e7b3205a9ba970"
I20240502 01:52:59.156286 13977 raft_consensus.cc:3032] T 0eb60206b8cb40d3b30cdc1f47d67f94 P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:52:59.156358 13977 raft_consensus.cc:729] T 0eb60206b8cb40d3b30cdc1f47d67f94 P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d43426b5bc874e8482e7b3205a9ba970, State: Running, Role: LEADER
I20240502 01:52:59.156580 13977 consensus_queue.cc:260] T 0eb60206b8cb40d3b30cdc1f47d67f94 P d43426b5bc874e8482e7b3205a9ba970 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "020e5f8ebb004e2ab6fddaece6663ece" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 37505 } attrs { promote: false } }
I20240502 01:52:59.156822 13977 raft_consensus.cc:3037] T 0eb60206b8cb40d3b30cdc1f47d67f94 P d43426b5bc874e8482e7b3205a9ba970 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:59.158728 13977 raft_consensus.cc:2454] T 0eb60206b8cb40d3b30cdc1f47d67f94 P d43426b5bc874e8482e7b3205a9ba970 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 020e5f8ebb004e2ab6fddaece6663ece in term 2.
I20240502 01:52:59.159101 15129 leader_election.cc:302] T 0eb60206b8cb40d3b30cdc1f47d67f94 P 020e5f8ebb004e2ab6fddaece6663ece [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 020e5f8ebb004e2ab6fddaece6663ece, d43426b5bc874e8482e7b3205a9ba970; no voters:
I20240502 01:52:59.159368 16029 raft_consensus.cc:2781] T 0eb60206b8cb40d3b30cdc1f47d67f94 P 020e5f8ebb004e2ab6fddaece6663ece [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:52:59.159515 16029 raft_consensus.cc:686] T 0eb60206b8cb40d3b30cdc1f47d67f94 P 020e5f8ebb004e2ab6fddaece6663ece [term 2 LEADER]: Becoming Leader. State: Replica: 020e5f8ebb004e2ab6fddaece6663ece, State: Running, Role: LEADER
I20240502 01:52:59.159755 16029 consensus_queue.cc:237] T 0eb60206b8cb40d3b30cdc1f47d67f94 P 020e5f8ebb004e2ab6fddaece6663ece [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "020e5f8ebb004e2ab6fddaece6663ece" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 37505 } attrs { promote: false } }
I20240502 01:52:59.161646 13833 catalog_manager.cc:5484] T 0eb60206b8cb40d3b30cdc1f47d67f94 P 020e5f8ebb004e2ab6fddaece6663ece reported cstate change: term changed from 1 to 2, leader changed from d43426b5bc874e8482e7b3205a9ba970 (127.11.42.65) to 020e5f8ebb004e2ab6fddaece6663ece (127.11.42.67). New cstate: current_term: 2 leader_uuid: "020e5f8ebb004e2ab6fddaece6663ece" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "020e5f8ebb004e2ab6fddaece6663ece" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 37505 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:52:59.217074 16127 raft_consensus.cc:979] T 6ed541d2e5ba4bbab19869c10116142e P d43426b5bc874e8482e7b3205a9ba970: : Instructing follower 39e5e5c1a5e74915800d96f720aee8c4 to start an election
I20240502 01:52:59.217260 16127 raft_consensus.cc:1067] T 6ed541d2e5ba4bbab19869c10116142e P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Signalling peer 39e5e5c1a5e74915800d96f720aee8c4 to start an election
I20240502 01:52:59.217886 15731 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6ed541d2e5ba4bbab19869c10116142e"
dest_uuid: "39e5e5c1a5e74915800d96f720aee8c4"
from {username='slave'} at 127.11.42.65:50349
I20240502 01:52:59.218190 15731 raft_consensus.cc:482] T 6ed541d2e5ba4bbab19869c10116142e P 39e5e5c1a5e74915800d96f720aee8c4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:52:59.218338 15731 raft_consensus.cc:3037] T 6ed541d2e5ba4bbab19869c10116142e P 39e5e5c1a5e74915800d96f720aee8c4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:59.220175 15731 raft_consensus.cc:504] T 6ed541d2e5ba4bbab19869c10116142e P 39e5e5c1a5e74915800d96f720aee8c4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "39e5e5c1a5e74915800d96f720aee8c4" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 37071 } attrs { promote: false } }
I20240502 01:52:59.220619 15731 leader_election.cc:288] T 6ed541d2e5ba4bbab19869c10116142e P 39e5e5c1a5e74915800d96f720aee8c4 [CANDIDATE]: Term 2 election: Requested vote from peers d43426b5bc874e8482e7b3205a9ba970 (127.11.42.65:40383)
I20240502 01:52:59.221130 13977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ed541d2e5ba4bbab19869c10116142e" candidate_uuid: "39e5e5c1a5e74915800d96f720aee8c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "d43426b5bc874e8482e7b3205a9ba970"
I20240502 01:52:59.221330 13977 raft_consensus.cc:3032] T 6ed541d2e5ba4bbab19869c10116142e P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:52:59.221412 13977 raft_consensus.cc:729] T 6ed541d2e5ba4bbab19869c10116142e P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d43426b5bc874e8482e7b3205a9ba970, State: Running, Role: LEADER
I20240502 01:52:59.221680 13977 consensus_queue.cc:260] T 6ed541d2e5ba4bbab19869c10116142e P d43426b5bc874e8482e7b3205a9ba970 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "39e5e5c1a5e74915800d96f720aee8c4" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 37071 } attrs { promote: false } }
I20240502 01:52:59.221963 13977 raft_consensus.cc:3037] T 6ed541d2e5ba4bbab19869c10116142e P d43426b5bc874e8482e7b3205a9ba970 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:59.223994 13977 raft_consensus.cc:2454] T 6ed541d2e5ba4bbab19869c10116142e P d43426b5bc874e8482e7b3205a9ba970 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39e5e5c1a5e74915800d96f720aee8c4 in term 2.
I20240502 01:52:59.224396 15664 leader_election.cc:302] T 6ed541d2e5ba4bbab19869c10116142e P 39e5e5c1a5e74915800d96f720aee8c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 39e5e5c1a5e74915800d96f720aee8c4, d43426b5bc874e8482e7b3205a9ba970; no voters:
I20240502 01:52:59.224748 16098 raft_consensus.cc:2781] T 6ed541d2e5ba4bbab19869c10116142e P 39e5e5c1a5e74915800d96f720aee8c4 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:52:59.224922 16098 raft_consensus.cc:686] T 6ed541d2e5ba4bbab19869c10116142e P 39e5e5c1a5e74915800d96f720aee8c4 [term 2 LEADER]: Becoming Leader. State: Replica: 39e5e5c1a5e74915800d96f720aee8c4, State: Running, Role: LEADER
I20240502 01:52:59.225217 16098 consensus_queue.cc:237] T 6ed541d2e5ba4bbab19869c10116142e P 39e5e5c1a5e74915800d96f720aee8c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "39e5e5c1a5e74915800d96f720aee8c4" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 37071 } attrs { promote: false } }
I20240502 01:52:59.227979 13833 catalog_manager.cc:5484] T 6ed541d2e5ba4bbab19869c10116142e P 39e5e5c1a5e74915800d96f720aee8c4 reported cstate change: term changed from 1 to 2, leader changed from d43426b5bc874e8482e7b3205a9ba970 (127.11.42.65) to 39e5e5c1a5e74915800d96f720aee8c4 (127.11.42.71). New cstate: current_term: 2 leader_uuid: "39e5e5c1a5e74915800d96f720aee8c4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39e5e5c1a5e74915800d96f720aee8c4" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 37071 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:52:59.280709 14111 raft_consensus.cc:1261] T aa387d2b1dcc42ddb34005f08dde2d8d P 1478241a03934d029dc7030480423cab [term 2 FOLLOWER]: Refusing update from remote peer 95e7c90cdc634d64a66dad6805e1680c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:52:59.281525 16338 consensus_queue.cc:1035] T aa387d2b1dcc42ddb34005f08dde2d8d P 95e7c90cdc634d64a66dad6805e1680c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:52:59.295879 13977 raft_consensus.cc:1261] T 50496c7daf804faa8d7bfaf128839e2c P d43426b5bc874e8482e7b3205a9ba970 [term 2 FOLLOWER]: Refusing update from remote peer 95e7c90cdc634d64a66dad6805e1680c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:52:59.296708 16338 consensus_queue.cc:1035] T 50496c7daf804faa8d7bfaf128839e2c P 95e7c90cdc634d64a66dad6805e1680c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:52:59.364619 14111 raft_consensus.cc:1261] T 6eb1311f3c054aec9a306922f25c6b89 P 1478241a03934d029dc7030480423cab [term 2 FOLLOWER]: Refusing update from remote peer 1607e211ee8c48de9fa56e78a3b01418: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:52:59.365291 16068 consensus_queue.cc:1035] T 6eb1311f3c054aec9a306922f25c6b89 P 1607e211ee8c48de9fa56e78a3b01418 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:52:59.428282 13977 raft_consensus.cc:1261] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P d43426b5bc874e8482e7b3205a9ba970 [term 2 FOLLOWER]: Refusing update from remote peer 39e5e5c1a5e74915800d96f720aee8c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:52:59.428941 16098 consensus_queue.cc:1035] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P 39e5e5c1a5e74915800d96f720aee8c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:52:59.458586 14111 raft_consensus.cc:1261] T f9f4aab07f984168bdfec24f908c124d P 1478241a03934d029dc7030480423cab [term 2 FOLLOWER]: Refusing update from remote peer 2ee638444633463b943c881866708482: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:52:59.459203 16453 consensus_queue.cc:1035] T f9f4aab07f984168bdfec24f908c124d P 2ee638444633463b943c881866708482 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:52:59.486071 15328 consensus_queue.cc:237] T 6eb1311f3c054aec9a306922f25c6b89 P 1607e211ee8c48de9fa56e78a3b01418 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1607e211ee8c48de9fa56e78a3b01418" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 41779 } attrs { promote: false } }
I20240502 01:52:59.486938 16333 raft_consensus.cc:2932] T 6eb1311f3c054aec9a306922f25c6b89 P 1607e211ee8c48de9fa56e78a3b01418 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 1478241a03934d029dc7030480423cab (127.11.42.66) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1607e211ee8c48de9fa56e78a3b01418" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 41779 } attrs { promote: false } } }
I20240502 01:52:59.489584 13820 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6eb1311f3c054aec9a306922f25c6b89 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 01:52:59.490660 13833 catalog_manager.cc:5484] T 6eb1311f3c054aec9a306922f25c6b89 P 1607e211ee8c48de9fa56e78a3b01418 reported cstate change: config changed from index 5 to 7, VOTER 1478241a03934d029dc7030480423cab (127.11.42.66) evicted. New cstate: current_term: 2 leader_uuid: "1607e211ee8c48de9fa56e78a3b01418" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1607e211ee8c48de9fa56e78a3b01418" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 41779 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:52:59.493933 14090 tablet_service.cc:1514] Processing DeleteTablet for tablet 6eb1311f3c054aec9a306922f25c6b89 with delete_type TABLET_DATA_TOMBSTONED (TS 1478241a03934d029dc7030480423cab not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38844
I20240502 01:52:59.494695 16457 tablet_replica.cc:330] T 6eb1311f3c054aec9a306922f25c6b89 P 1478241a03934d029dc7030480423cab: stopping tablet replica
I20240502 01:52:59.495075 16457 raft_consensus.cc:2229] T 6eb1311f3c054aec9a306922f25c6b89 P 1478241a03934d029dc7030480423cab [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:52:59.495301 16457 raft_consensus.cc:2258] T 6eb1311f3c054aec9a306922f25c6b89 P 1478241a03934d029dc7030480423cab [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:59.496595 16457 ts_tablet_manager.cc:1875] T 6eb1311f3c054aec9a306922f25c6b89 P 1478241a03934d029dc7030480423cab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:52:59.498214 15731 consensus_queue.cc:237] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P 39e5e5c1a5e74915800d96f720aee8c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "39e5e5c1a5e74915800d96f720aee8c4" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 37071 } attrs { promote: false } }
I20240502 01:52:59.499073 16330 raft_consensus.cc:2932] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P 39e5e5c1a5e74915800d96f720aee8c4 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER d43426b5bc874e8482e7b3205a9ba970 (127.11.42.65) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "39e5e5c1a5e74915800d96f720aee8c4" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 37071 } attrs { promote: false } } }
I20240502 01:52:59.501451 16457 ts_tablet_manager.cc:1888] T 6eb1311f3c054aec9a306922f25c6b89 P 1478241a03934d029dc7030480423cab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 01:52:59.501590 16457 log.cc:1196] T 6eb1311f3c054aec9a306922f25c6b89 P 1478241a03934d029dc7030480423cab: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/wals/6eb1311f3c054aec9a306922f25c6b89
I20240502 01:52:59.502242 13819 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 67bc0cd4c3b94782ac1d9f150fcdbd03 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 01:52:59.503079 13821 catalog_manager.cc:4830] TS 1478241a03934d029dc7030480423cab (127.11.42.66:34613): tablet 6eb1311f3c054aec9a306922f25c6b89 (table rebalance_test_table_3 [id=fb174a9542bb4a20bf8666a08c8a114e]) successfully deleted
I20240502 01:52:59.503525 13833 catalog_manager.cc:5484] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P 39e5e5c1a5e74915800d96f720aee8c4 reported cstate change: config changed from index 5 to 7, VOTER d43426b5bc874e8482e7b3205a9ba970 (127.11.42.65) evicted. New cstate: current_term: 2 leader_uuid: "39e5e5c1a5e74915800d96f720aee8c4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "39e5e5c1a5e74915800d96f720aee8c4" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 37071 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:52:59.506542 13957 tablet_service.cc:1514] Processing DeleteTablet for tablet 67bc0cd4c3b94782ac1d9f150fcdbd03 with delete_type TABLET_DATA_TOMBSTONED (TS d43426b5bc874e8482e7b3205a9ba970 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33834
I20240502 01:52:59.507407 16458 tablet_replica.cc:330] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P d43426b5bc874e8482e7b3205a9ba970: stopping tablet replica
I20240502 01:52:59.507808 16458 raft_consensus.cc:2229] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P d43426b5bc874e8482e7b3205a9ba970 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:52:59.508054 16458 raft_consensus.cc:2258] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P d43426b5bc874e8482e7b3205a9ba970 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:59.508572 15592 consensus_queue.cc:237] T f9f4aab07f984168bdfec24f908c124d P 2ee638444633463b943c881866708482 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2ee638444633463b943c881866708482" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 45255 } attrs { promote: false } }
I20240502 01:52:59.509541 16458 ts_tablet_manager.cc:1875] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P d43426b5bc874e8482e7b3205a9ba970: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:52:59.509397 16455 raft_consensus.cc:2932] T f9f4aab07f984168bdfec24f908c124d P 2ee638444633463b943c881866708482 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 1478241a03934d029dc7030480423cab (127.11.42.66) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2ee638444633463b943c881866708482" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 45255 } attrs { promote: false } } }
I20240502 01:52:59.512004 13819 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f9f4aab07f984168bdfec24f908c124d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 01:52:59.514676 14111 raft_consensus.cc:1261] T 5bc0ff324caa436196f22d90360c9247 P 1478241a03934d029dc7030480423cab [term 2 FOLLOWER]: Refusing update from remote peer 020e5f8ebb004e2ab6fddaece6663ece: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:52:59.515269 16029 consensus_queue.cc:1035] T 5bc0ff324caa436196f22d90360c9247 P 020e5f8ebb004e2ab6fddaece6663ece [LEADER]: Connected to new peer: Peer: permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:52:59.515494 16458 ts_tablet_manager.cc:1888] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P d43426b5bc874e8482e7b3205a9ba970: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 01:52:59.515620 16458 log.cc:1196] T 67bc0cd4c3b94782ac1d9f150fcdbd03 P d43426b5bc874e8482e7b3205a9ba970: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/67bc0cd4c3b94782ac1d9f150fcdbd03
I20240502 01:52:59.516459 13819 catalog_manager.cc:4830] TS d43426b5bc874e8482e7b3205a9ba970 (127.11.42.65:40383): tablet 67bc0cd4c3b94782ac1d9f150fcdbd03 (table rebalance_test_table_0 [id=cd22e1a0d15d492fb0d2e238ceb715ea]) successfully deleted
I20240502 01:52:59.520475 13833 catalog_manager.cc:5484] T f9f4aab07f984168bdfec24f908c124d P 2ee638444633463b943c881866708482 reported cstate change: config changed from index 5 to 7, VOTER 1478241a03934d029dc7030480423cab (127.11.42.66) evicted. New cstate: current_term: 2 leader_uuid: "2ee638444633463b943c881866708482" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2ee638444633463b943c881866708482" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 45255 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:52:59.525339 14090 tablet_service.cc:1514] Processing DeleteTablet for tablet f9f4aab07f984168bdfec24f908c124d with delete_type TABLET_DATA_TOMBSTONED (TS 1478241a03934d029dc7030480423cab not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38844
I20240502 01:52:59.526578 15459 consensus_queue.cc:237] T 50496c7daf804faa8d7bfaf128839e2c P 95e7c90cdc634d64a66dad6805e1680c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "95e7c90cdc634d64a66dad6805e1680c" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 43981 } attrs { promote: false } }
I20240502 01:52:59.527412 16338 raft_consensus.cc:2932] T 50496c7daf804faa8d7bfaf128839e2c P 95e7c90cdc634d64a66dad6805e1680c [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER d43426b5bc874e8482e7b3205a9ba970 (127.11.42.65) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "95e7c90cdc634d64a66dad6805e1680c" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 43981 } attrs { promote: false } } }
I20240502 01:52:59.530474 13819 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 50496c7daf804faa8d7bfaf128839e2c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 01:52:59.530795 15459 consensus_queue.cc:237] T aa387d2b1dcc42ddb34005f08dde2d8d P 95e7c90cdc634d64a66dad6805e1680c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "95e7c90cdc634d64a66dad6805e1680c" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 43981 } attrs { promote: false } }
I20240502 01:52:59.530931 16457 tablet_replica.cc:330] T f9f4aab07f984168bdfec24f908c124d P 1478241a03934d029dc7030480423cab: stopping tablet replica
I20240502 01:52:59.531109 16457 raft_consensus.cc:2229] T f9f4aab07f984168bdfec24f908c124d P 1478241a03934d029dc7030480423cab [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:52:59.531291 16457 raft_consensus.cc:2258] T f9f4aab07f984168bdfec24f908c124d P 1478241a03934d029dc7030480423cab [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:59.531546 16081 raft_consensus.cc:2932] T aa387d2b1dcc42ddb34005f08dde2d8d P 95e7c90cdc634d64a66dad6805e1680c [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 1478241a03934d029dc7030480423cab (127.11.42.66) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "95e7c90cdc634d64a66dad6805e1680c" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 43981 } attrs { promote: false } } }
I20240502 01:52:59.531888 13833 catalog_manager.cc:5484] T 50496c7daf804faa8d7bfaf128839e2c P 95e7c90cdc634d64a66dad6805e1680c reported cstate change: config changed from index 5 to 7, VOTER d43426b5bc874e8482e7b3205a9ba970 (127.11.42.65) evicted. New cstate: current_term: 2 leader_uuid: "95e7c90cdc634d64a66dad6805e1680c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "95e7c90cdc634d64a66dad6805e1680c" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 43981 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:52:59.533941 13819 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet aa387d2b1dcc42ddb34005f08dde2d8d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 01:52:59.534613 16457 ts_tablet_manager.cc:1875] T f9f4aab07f984168bdfec24f908c124d P 1478241a03934d029dc7030480423cab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:52:59.535679 13957 tablet_service.cc:1514] Processing DeleteTablet for tablet 50496c7daf804faa8d7bfaf128839e2c with delete_type TABLET_DATA_TOMBSTONED (TS d43426b5bc874e8482e7b3205a9ba970 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33834
I20240502 01:52:59.535991 16458 tablet_replica.cc:330] T 50496c7daf804faa8d7bfaf128839e2c P d43426b5bc874e8482e7b3205a9ba970: stopping tablet replica
I20240502 01:52:59.536155 16458 raft_consensus.cc:2229] T 50496c7daf804faa8d7bfaf128839e2c P d43426b5bc874e8482e7b3205a9ba970 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:52:59.536343 16458 raft_consensus.cc:2258] T 50496c7daf804faa8d7bfaf128839e2c P d43426b5bc874e8482e7b3205a9ba970 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:59.536878 13833 catalog_manager.cc:5484] T aa387d2b1dcc42ddb34005f08dde2d8d P 95e7c90cdc634d64a66dad6805e1680c reported cstate change: config changed from index 5 to 7, VOTER 1478241a03934d029dc7030480423cab (127.11.42.66) evicted. New cstate: current_term: 2 leader_uuid: "95e7c90cdc634d64a66dad6805e1680c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "95e7c90cdc634d64a66dad6805e1680c" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 43981 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:52:59.537318 16458 ts_tablet_manager.cc:1875] T 50496c7daf804faa8d7bfaf128839e2c P d43426b5bc874e8482e7b3205a9ba970: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:52:59.540231 14090 tablet_service.cc:1514] Processing DeleteTablet for tablet aa387d2b1dcc42ddb34005f08dde2d8d with delete_type TABLET_DATA_TOMBSTONED (TS 1478241a03934d029dc7030480423cab not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38844
I20240502 01:52:59.541532 16457 ts_tablet_manager.cc:1888] T f9f4aab07f984168bdfec24f908c124d P 1478241a03934d029dc7030480423cab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 01:52:59.541632 16457 log.cc:1196] T f9f4aab07f984168bdfec24f908c124d P 1478241a03934d029dc7030480423cab: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/wals/f9f4aab07f984168bdfec24f908c124d
I20240502 01:52:59.542268 16457 tablet_replica.cc:330] T aa387d2b1dcc42ddb34005f08dde2d8d P 1478241a03934d029dc7030480423cab: stopping tablet replica
I20240502 01:52:59.542418 16457 raft_consensus.cc:2229] T aa387d2b1dcc42ddb34005f08dde2d8d P 1478241a03934d029dc7030480423cab [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:52:59.542565 16457 raft_consensus.cc:2258] T aa387d2b1dcc42ddb34005f08dde2d8d P 1478241a03934d029dc7030480423cab [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:59.542901 13821 catalog_manager.cc:4830] TS 1478241a03934d029dc7030480423cab (127.11.42.66:34613): tablet f9f4aab07f984168bdfec24f908c124d (table rebalance_test_table_1 [id=9069de6d20e94872b794e4757f1a1d1a]) successfully deleted
I20240502 01:52:59.543118 16458 ts_tablet_manager.cc:1888] T 50496c7daf804faa8d7bfaf128839e2c P d43426b5bc874e8482e7b3205a9ba970: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 01:52:59.543239 16458 log.cc:1196] T 50496c7daf804faa8d7bfaf128839e2c P d43426b5bc874e8482e7b3205a9ba970: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/50496c7daf804faa8d7bfaf128839e2c
I20240502 01:52:59.543967 13819 catalog_manager.cc:4830] TS d43426b5bc874e8482e7b3205a9ba970 (127.11.42.65:40383): tablet 50496c7daf804faa8d7bfaf128839e2c (table rebalance_test_table_4 [id=0c0532d94ab24438b74e82726a95b9e8]) successfully deleted
I20240502 01:52:59.544704 16457 ts_tablet_manager.cc:1875] T aa387d2b1dcc42ddb34005f08dde2d8d P 1478241a03934d029dc7030480423cab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:52:59.550258 16457 ts_tablet_manager.cc:1888] T aa387d2b1dcc42ddb34005f08dde2d8d P 1478241a03934d029dc7030480423cab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 01:52:59.550381 16457 log.cc:1196] T aa387d2b1dcc42ddb34005f08dde2d8d P 1478241a03934d029dc7030480423cab: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/wals/aa387d2b1dcc42ddb34005f08dde2d8d
I20240502 01:52:59.550922 13821 catalog_manager.cc:4830] TS 1478241a03934d029dc7030480423cab (127.11.42.66:34613): tablet aa387d2b1dcc42ddb34005f08dde2d8d (table rebalance_test_table_0 [id=cd22e1a0d15d492fb0d2e238ceb715ea]) successfully deleted
I20240502 01:52:59.559454 13977 raft_consensus.cc:1261] T 0eb60206b8cb40d3b30cdc1f47d67f94 P d43426b5bc874e8482e7b3205a9ba970 [term 2 FOLLOWER]: Refusing update from remote peer 020e5f8ebb004e2ab6fddaece6663ece: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:52:59.560030 16375 consensus_queue.cc:1035] T 0eb60206b8cb40d3b30cdc1f47d67f94 P 020e5f8ebb004e2ab6fddaece6663ece [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:52:59.597448 15196 consensus_queue.cc:237] T 5bc0ff324caa436196f22d90360c9247 P 020e5f8ebb004e2ab6fddaece6663ece [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "020e5f8ebb004e2ab6fddaece6663ece" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 37505 } attrs { promote: false } }
I20240502 01:52:59.598515 16466 raft_consensus.cc:2932] T 5bc0ff324caa436196f22d90360c9247 P 020e5f8ebb004e2ab6fddaece6663ece [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 1478241a03934d029dc7030480423cab (127.11.42.66) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "020e5f8ebb004e2ab6fddaece6663ece" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 37505 } attrs { promote: false } } }
I20240502 01:52:59.602216 13819 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5bc0ff324caa436196f22d90360c9247 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 01:52:59.603495 13834 catalog_manager.cc:5484] T 5bc0ff324caa436196f22d90360c9247 P 020e5f8ebb004e2ab6fddaece6663ece reported cstate change: config changed from index 5 to 7, VOTER 1478241a03934d029dc7030480423cab (127.11.42.66) evicted. New cstate: current_term: 2 leader_uuid: "020e5f8ebb004e2ab6fddaece6663ece" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "020e5f8ebb004e2ab6fddaece6663ece" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 37505 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:52:59.607241 14090 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bc0ff324caa436196f22d90360c9247 with delete_type TABLET_DATA_TOMBSTONED (TS 1478241a03934d029dc7030480423cab not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38844
I20240502 01:52:59.607690 16457 tablet_replica.cc:330] T 5bc0ff324caa436196f22d90360c9247 P 1478241a03934d029dc7030480423cab: stopping tablet replica
I20240502 01:52:59.607878 16457 raft_consensus.cc:2229] T 5bc0ff324caa436196f22d90360c9247 P 1478241a03934d029dc7030480423cab [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:52:59.608039 16457 raft_consensus.cc:2258] T 5bc0ff324caa436196f22d90360c9247 P 1478241a03934d029dc7030480423cab [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:59.608968 16457 ts_tablet_manager.cc:1875] T 5bc0ff324caa436196f22d90360c9247 P 1478241a03934d029dc7030480423cab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:52:59.614534 16457 ts_tablet_manager.cc:1888] T 5bc0ff324caa436196f22d90360c9247 P 1478241a03934d029dc7030480423cab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 01:52:59.614645 16457 log.cc:1196] T 5bc0ff324caa436196f22d90360c9247 P 1478241a03934d029dc7030480423cab: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/wals/5bc0ff324caa436196f22d90360c9247
I20240502 01:52:59.615270 13821 catalog_manager.cc:4830] TS 1478241a03934d029dc7030480423cab (127.11.42.66:34613): tablet 5bc0ff324caa436196f22d90360c9247 (table rebalance_test_table_3 [id=fb174a9542bb4a20bf8666a08c8a114e]) successfully deleted
I20240502 01:52:59.641341 13977 raft_consensus.cc:1261] T c0620ccc47634258a7962d91cc7bb060 P d43426b5bc874e8482e7b3205a9ba970 [term 2 FOLLOWER]: Refusing update from remote peer 1607e211ee8c48de9fa56e78a3b01418: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:52:59.641920 16333 consensus_queue.cc:1035] T c0620ccc47634258a7962d91cc7bb060 P 1607e211ee8c48de9fa56e78a3b01418 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:52:59.709966 15328 consensus_queue.cc:237] T c0620ccc47634258a7962d91cc7bb060 P 1607e211ee8c48de9fa56e78a3b01418 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1607e211ee8c48de9fa56e78a3b01418" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 41779 } attrs { promote: false } }
I20240502 01:52:59.710882 16523 raft_consensus.cc:2932] T c0620ccc47634258a7962d91cc7bb060 P 1607e211ee8c48de9fa56e78a3b01418 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER d43426b5bc874e8482e7b3205a9ba970 (127.11.42.65) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1607e211ee8c48de9fa56e78a3b01418" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 41779 } attrs { promote: false } } }
I20240502 01:52:59.713658 13820 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c0620ccc47634258a7962d91cc7bb060 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 01:52:59.715193 13833 catalog_manager.cc:5484] T c0620ccc47634258a7962d91cc7bb060 P 1607e211ee8c48de9fa56e78a3b01418 reported cstate change: config changed from index 5 to 7, VOTER d43426b5bc874e8482e7b3205a9ba970 (127.11.42.65) evicted. New cstate: current_term: 2 leader_uuid: "1607e211ee8c48de9fa56e78a3b01418" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1607e211ee8c48de9fa56e78a3b01418" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 41779 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:52:59.718598 13957 tablet_service.cc:1514] Processing DeleteTablet for tablet c0620ccc47634258a7962d91cc7bb060 with delete_type TABLET_DATA_TOMBSTONED (TS d43426b5bc874e8482e7b3205a9ba970 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33834
I20240502 01:52:59.718986 16458 tablet_replica.cc:330] T c0620ccc47634258a7962d91cc7bb060 P d43426b5bc874e8482e7b3205a9ba970: stopping tablet replica
I20240502 01:52:59.719203 16458 raft_consensus.cc:2229] T c0620ccc47634258a7962d91cc7bb060 P d43426b5bc874e8482e7b3205a9ba970 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:52:59.719416 16458 raft_consensus.cc:2258] T c0620ccc47634258a7962d91cc7bb060 P d43426b5bc874e8482e7b3205a9ba970 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:59.720207 16458 ts_tablet_manager.cc:1875] T c0620ccc47634258a7962d91cc7bb060 P d43426b5bc874e8482e7b3205a9ba970: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:52:59.724360 16458 ts_tablet_manager.cc:1888] T c0620ccc47634258a7962d91cc7bb060 P d43426b5bc874e8482e7b3205a9ba970: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 01:52:59.724471 16458 log.cc:1196] T c0620ccc47634258a7962d91cc7bb060 P d43426b5bc874e8482e7b3205a9ba970: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/c0620ccc47634258a7962d91cc7bb060
I20240502 01:52:59.725142 13819 catalog_manager.cc:4830] TS d43426b5bc874e8482e7b3205a9ba970 (127.11.42.65:40383): tablet c0620ccc47634258a7962d91cc7bb060 (table rebalance_test_table_2 [id=3c737a3ccdb1430496def9e37e6428d9]) successfully deleted
I20240502 01:52:59.780925 13977 raft_consensus.cc:1261] T 6ed541d2e5ba4bbab19869c10116142e P d43426b5bc874e8482e7b3205a9ba970 [term 2 FOLLOWER]: Refusing update from remote peer 39e5e5c1a5e74915800d96f720aee8c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:52:59.781525 16330 consensus_queue.cc:1035] T 6ed541d2e5ba4bbab19869c10116142e P 39e5e5c1a5e74915800d96f720aee8c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:52:59.795506 15196 consensus_queue.cc:237] T 0eb60206b8cb40d3b30cdc1f47d67f94 P 020e5f8ebb004e2ab6fddaece6663ece [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "020e5f8ebb004e2ab6fddaece6663ece" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 37505 } attrs { promote: false } }
I20240502 01:52:59.796329 16375 raft_consensus.cc:2932] T 0eb60206b8cb40d3b30cdc1f47d67f94 P 020e5f8ebb004e2ab6fddaece6663ece [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER d43426b5bc874e8482e7b3205a9ba970 (127.11.42.65) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "020e5f8ebb004e2ab6fddaece6663ece" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 37505 } attrs { promote: false } } }
I20240502 01:52:59.799415 13819 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0eb60206b8cb40d3b30cdc1f47d67f94 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 01:52:59.800668 13834 catalog_manager.cc:5484] T 0eb60206b8cb40d3b30cdc1f47d67f94 P 020e5f8ebb004e2ab6fddaece6663ece reported cstate change: config changed from index 5 to 7, VOTER d43426b5bc874e8482e7b3205a9ba970 (127.11.42.65) evicted. New cstate: current_term: 2 leader_uuid: "020e5f8ebb004e2ab6fddaece6663ece" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "020e5f8ebb004e2ab6fddaece6663ece" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 37505 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:52:59.803800 13957 tablet_service.cc:1514] Processing DeleteTablet for tablet 0eb60206b8cb40d3b30cdc1f47d67f94 with delete_type TABLET_DATA_TOMBSTONED (TS d43426b5bc874e8482e7b3205a9ba970 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33834
I20240502 01:52:59.804191 16458 tablet_replica.cc:330] T 0eb60206b8cb40d3b30cdc1f47d67f94 P d43426b5bc874e8482e7b3205a9ba970: stopping tablet replica
I20240502 01:52:59.804404 16458 raft_consensus.cc:2229] T 0eb60206b8cb40d3b30cdc1f47d67f94 P d43426b5bc874e8482e7b3205a9ba970 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:52:59.804590 16458 raft_consensus.cc:2258] T 0eb60206b8cb40d3b30cdc1f47d67f94 P d43426b5bc874e8482e7b3205a9ba970 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:59.805357 16458 ts_tablet_manager.cc:1875] T 0eb60206b8cb40d3b30cdc1f47d67f94 P d43426b5bc874e8482e7b3205a9ba970: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:52:59.809530 16458 ts_tablet_manager.cc:1888] T 0eb60206b8cb40d3b30cdc1f47d67f94 P d43426b5bc874e8482e7b3205a9ba970: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 01:52:59.809646 16458 log.cc:1196] T 0eb60206b8cb40d3b30cdc1f47d67f94 P d43426b5bc874e8482e7b3205a9ba970: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/0eb60206b8cb40d3b30cdc1f47d67f94
I20240502 01:52:59.810340 13819 catalog_manager.cc:4830] TS d43426b5bc874e8482e7b3205a9ba970 (127.11.42.65:40383): tablet 0eb60206b8cb40d3b30cdc1f47d67f94 (table rebalance_test_table_2 [id=3c737a3ccdb1430496def9e37e6428d9]) successfully deleted
I20240502 01:52:59.848824 15731 consensus_queue.cc:237] T 6ed541d2e5ba4bbab19869c10116142e P 39e5e5c1a5e74915800d96f720aee8c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "39e5e5c1a5e74915800d96f720aee8c4" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 37071 } attrs { promote: false } }
I20240502 01:52:59.849705 16098 raft_consensus.cc:2932] T 6ed541d2e5ba4bbab19869c10116142e P 39e5e5c1a5e74915800d96f720aee8c4 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER d43426b5bc874e8482e7b3205a9ba970 (127.11.42.65) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "39e5e5c1a5e74915800d96f720aee8c4" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 37071 } attrs { promote: false } } }
I20240502 01:52:59.852173 13819 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6ed541d2e5ba4bbab19869c10116142e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 01:52:59.853468 13834 catalog_manager.cc:5484] T 6ed541d2e5ba4bbab19869c10116142e P 39e5e5c1a5e74915800d96f720aee8c4 reported cstate change: config changed from index 5 to 7, VOTER d43426b5bc874e8482e7b3205a9ba970 (127.11.42.65) evicted. New cstate: current_term: 2 leader_uuid: "39e5e5c1a5e74915800d96f720aee8c4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "39e5e5c1a5e74915800d96f720aee8c4" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 37071 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:52:59.856600 13957 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ed541d2e5ba4bbab19869c10116142e with delete_type TABLET_DATA_TOMBSTONED (TS d43426b5bc874e8482e7b3205a9ba970 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33834
I20240502 01:52:59.856920 16458 tablet_replica.cc:330] T 6ed541d2e5ba4bbab19869c10116142e P d43426b5bc874e8482e7b3205a9ba970: stopping tablet replica
I20240502 01:52:59.857122 16458 raft_consensus.cc:2229] T 6ed541d2e5ba4bbab19869c10116142e P d43426b5bc874e8482e7b3205a9ba970 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:52:59.857303 16458 raft_consensus.cc:2258] T 6ed541d2e5ba4bbab19869c10116142e P d43426b5bc874e8482e7b3205a9ba970 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:59.858068 16458 ts_tablet_manager.cc:1875] T 6ed541d2e5ba4bbab19869c10116142e P d43426b5bc874e8482e7b3205a9ba970: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:52:59.862249 16458 ts_tablet_manager.cc:1888] T 6ed541d2e5ba4bbab19869c10116142e P d43426b5bc874e8482e7b3205a9ba970: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 01:52:59.862354 16458 log.cc:1196] T 6ed541d2e5ba4bbab19869c10116142e P d43426b5bc874e8482e7b3205a9ba970: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/6ed541d2e5ba4bbab19869c10116142e
I20240502 01:52:59.863040 13819 catalog_manager.cc:4830] TS d43426b5bc874e8482e7b3205a9ba970 (127.11.42.65:40383): tablet 6ed541d2e5ba4bbab19869c10116142e (table rebalance_test_table_4 [id=0c0532d94ab24438b74e82726a95b9e8]) successfully deleted
I20240502 01:53:00.006909 15176 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:00.020726 14090 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:00.022190 15308 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:00.022846 15572 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:00.034286 13957 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:00.038308 15711 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:00.065618 15439 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:00.227542 15176 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:00.238986 14090 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:00.257567 15572 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:00.268136 15308 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:00.280151 15711 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:00.286808 15439 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:00.324802 13957 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:00.965190 15176 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:00.994040 14090 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:01.003818 15308 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:01.040874 15572 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:01.044049 15711 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:01.050495 15439 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:01.057904 13957 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:01.232790 15176 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:01.233260 14090 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:01.272531 15308 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:01.273363 15572 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:01.276782 15711 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:01.291580 15439 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:01.301697 13957 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:01.642541 14090 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:01.650830 15176 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:01.653987 15711 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:01.661154 15572 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:01.676329 15439 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:01.678016 15308 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:01.717120 13957 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d00df36391f54eaf96213f38c5e602a1 | 127.11.42.126:40815 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.11.42.84:45633 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
020e5f8ebb004e2ab6fddaece6663ece | 127.11.42.67:37505 | HEALTHY | <none> | 2 | 0
1478241a03934d029dc7030480423cab | 127.11.42.66:34613 | HEALTHY | <none> | 3 | 0
1607e211ee8c48de9fa56e78a3b01418 | 127.11.42.68:41779 | HEALTHY | <none> | 2 | 0
2ee638444633463b943c881866708482 | 127.11.42.70:45255 | HEALTHY | <none> | 2 | 0
39e5e5c1a5e74915800d96f720aee8c4 | 127.11.42.71:37071 | HEALTHY | <none> | 2 | 0
95e7c90cdc634d64a66dad6805e1680c | 127.11.42.69:43981 | HEALTHY | <none> | 2 | 0
d43426b5bc874e8482e7b3205a9ba970 | 127.11.42.65:40383 | HEALTHY | <none> | 3 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.11.42.65 | experimental | 127.11.42.65:40383
local_ip_for_outbound_sockets | 127.11.42.66 | experimental | 127.11.42.66:34613
local_ip_for_outbound_sockets | 127.11.42.67 | experimental | 127.11.42.67:37505
local_ip_for_outbound_sockets | 127.11.42.68 | experimental | 127.11.42.68:41779
local_ip_for_outbound_sockets | 127.11.42.69 | experimental | 127.11.42.69:43981
local_ip_for_outbound_sockets | 127.11.42.70 | experimental | 127.11.42.70:45255
local_ip_for_outbound_sockets | 127.11.42.71 | experimental | 127.11.42.71:37071
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.11.42.65:40383
server_dump_info_path | /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.11.42.66:34613
server_dump_info_path | /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.11.42.67:37505
server_dump_info_path | /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.11.42.68:41779
server_dump_info_path | /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.11.42.69:43981
server_dump_info_path | /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.11.42.70:45255
server_dump_info_path | /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.11.42.71:37071
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.11.42.84:45633 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 01:53:01.759243 11433 log_verifier.cc:126] Checking tablet 0eb60206b8cb40d3b30cdc1f47d67f94
I20240502 01:53:01.774528 11433 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 0eb60206b8cb40d3b30cdc1f47d67f94
I20240502 01:53:01.774655 11433 log_verifier.cc:126] Checking tablet 2d8fb038394d4ad981d208e52b19672f
I20240502 01:53:01.785804 11433 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 2d8fb038394d4ad981d208e52b19672f
I20240502 01:53:01.785892 11433 log_verifier.cc:126] Checking tablet 50496c7daf804faa8d7bfaf128839e2c
I20240502 01:53:01.797888 11433 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 50496c7daf804faa8d7bfaf128839e2c
I20240502 01:53:01.797983 11433 log_verifier.cc:126] Checking tablet 563e1017b8774231bd29a1f8e628b4dc
I20240502 01:53:01.805143 11433 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 563e1017b8774231bd29a1f8e628b4dc
I20240502 01:53:01.805238 11433 log_verifier.cc:126] Checking tablet 5bc0ff324caa436196f22d90360c9247
I20240502 01:53:01.820500 11433 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 5bc0ff324caa436196f22d90360c9247
I20240502 01:53:01.820619 11433 log_verifier.cc:126] Checking tablet 62ad914a819c4c7a85856be3c1855270
I20240502 01:53:01.836010 11433 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 62ad914a819c4c7a85856be3c1855270
I20240502 01:53:01.836095 11433 log_verifier.cc:126] Checking tablet 65d44d1da4bf4366860fc5d1a1e0785c
I20240502 01:53:01.847587 11433 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 65d44d1da4bf4366860fc5d1a1e0785c
I20240502 01:53:01.847669 11433 log_verifier.cc:126] Checking tablet 67bc0cd4c3b94782ac1d9f150fcdbd03
I20240502 01:53:01.859091 11433 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 67bc0cd4c3b94782ac1d9f150fcdbd03
I20240502 01:53:01.859182 11433 log_verifier.cc:126] Checking tablet 6eb1311f3c054aec9a306922f25c6b89
I20240502 01:53:01.871187 11433 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 6eb1311f3c054aec9a306922f25c6b89
I20240502 01:53:01.871279 11433 log_verifier.cc:126] Checking tablet 6ed541d2e5ba4bbab19869c10116142e
I20240502 01:53:01.884810 11433 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 6ed541d2e5ba4bbab19869c10116142e
I20240502 01:53:01.884905 11433 log_verifier.cc:126] Checking tablet 86219a30419046dcb0384f5e539cdf15
I20240502 01:53:01.903771 11433 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 86219a30419046dcb0384f5e539cdf15
I20240502 01:53:01.903900 11433 log_verifier.cc:126] Checking tablet 954145d4672146ba8d26a25540de0325
I20240502 01:53:01.927296 11433 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 954145d4672146ba8d26a25540de0325
I20240502 01:53:01.927402 11433 log_verifier.cc:126] Checking tablet aa387d2b1dcc42ddb34005f08dde2d8d
I20240502 01:53:01.948261 11433 log_verifier.cc:177] Verified matching terms for 7 ops in tablet aa387d2b1dcc42ddb34005f08dde2d8d
I20240502 01:53:01.948400 11433 log_verifier.cc:126] Checking tablet c0620ccc47634258a7962d91cc7bb060
I20240502 01:53:01.967882 11433 log_verifier.cc:177] Verified matching terms for 7 ops in tablet c0620ccc47634258a7962d91cc7bb060
I20240502 01:53:01.967983 11433 log_verifier.cc:126] Checking tablet d22aa9aee27e43a79dd19934de05c43c
I20240502 01:53:01.986079 11433 log_verifier.cc:177] Verified matching terms for 3 ops in tablet d22aa9aee27e43a79dd19934de05c43c
I20240502 01:53:01.986196 11433 log_verifier.cc:126] Checking tablet f9f4aab07f984168bdfec24f908c124d
I20240502 01:53:02.008395 11433 log_verifier.cc:177] Verified matching terms for 7 ops in tablet f9f4aab07f984168bdfec24f908c124d
I20240502 01:53:02.011458 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.72:0
--local_ip_for_outbound_sockets=127.11.42.72
--webserver_interface=127.11.42.72
--webserver_port=0
--tserver_master_addrs=127.11.42.126:40815
--builtin_ntp_servers=127.11.42.84:45633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 01:53:02.394038 16740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:02.394351 16740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:02.394505 16740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:02.405484 16740 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:53:02.405633 16740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:02.405823 16740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.72
I20240502 01:53:02.415752 16740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:45633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.11.42.72
--webserver_port=0
--tserver_master_addrs=127.11.42.126:40815
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.72
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:02.417063 16740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:02.418884 16740 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:02.430696 16745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:02.431097 16746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:02.431522 16748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:02.431973 16740 server_base.cc:1034] running on GCE node
I20240502 01:53:02.432454 16740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:02.433432 16740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:02.434633 16740 hybrid_clock.cc:648] HybridClock initialized: now 1714614782434585 us; error 38 us; skew 500 ppm
I20240502 01:53:02.434942 16740 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:02.437186 16740 webserver.cc:460] Webserver started at http://127.11.42.72:43021/ using document root <none> and password file <none>
I20240502 01:53:02.437903 16740 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:02.438011 16740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:02.438320 16740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:02.440779 16740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "48a417d95b604326aa5a918ff2f806b7"
format_stamp: "Formatted at 2024-05-02 01:53:02 on dist-test-slave-xvk5"
I20240502 01:53:02.441555 16740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "48a417d95b604326aa5a918ff2f806b7"
format_stamp: "Formatted at 2024-05-02 01:53:02 on dist-test-slave-xvk5"
I20240502 01:53:02.446292 16740 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 01:53:02.449558 16754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:02.450786 16740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 01:53:02.450922 16740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "48a417d95b604326aa5a918ff2f806b7"
format_stamp: "Formatted at 2024-05-02 01:53:02 on dist-test-slave-xvk5"
I20240502 01:53:02.451108 16740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:02.476063 16740 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:02.476899 16740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:02.477231 16740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:02.478310 16740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:02.479789 16740 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:02.479912 16740 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:02.480019 16740 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:02.480095 16740 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:02.515810 16740 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.72:45917
I20240502 01:53:02.515897 16866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.72:45917 every 8 connection(s)
I20240502 01:53:02.517141 16740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 01:53:02.519047 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 16740
I20240502 01:53:02.519240 11433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 01:53:02.530799 16867 heartbeater.cc:347] Connected to a master server at 127.11.42.126:40815
I20240502 01:53:02.531098 16867 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:02.532044 16867 heartbeater.cc:511] Master 127.11.42.126:40815 requested a full tablet report, sending...
I20240502 01:53:02.533463 13828 ts_manager.cc:194] Registered new tserver with Master: 48a417d95b604326aa5a918ff2f806b7 (127.11.42.72:45917)
I20240502 01:53:02.534514 13828 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.72:57793
I20240502 01:53:03.004792 15176 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:03.023710 14090 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:03.026108 15308 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:03.062593 15572 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:03.071130 15711 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:03.091686 15439 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:03.114136 16802 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:03.127718 13957 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:03.299839 14090 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:03.308085 15176 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:03.311074 15572 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:03.318718 15308 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:03.335024 15711 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:03.336017 16802 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:03.359267 15439 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:03.372709 13957 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:03.439199 13977 consensus_queue.cc:237] T d22aa9aee27e43a79dd19934de05c43c P d43426b5bc874e8482e7b3205a9ba970 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "48a417d95b604326aa5a918ff2f806b7" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 45917 } attrs { promote: true } }
I20240502 01:53:03.441164 16962 raft_consensus.cc:2932] T d22aa9aee27e43a79dd19934de05c43c P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 48a417d95b604326aa5a918ff2f806b7 (127.11.42.72) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "48a417d95b604326aa5a918ff2f806b7" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 45917 } attrs { promote: true } } }
I20240502 01:53:03.447075 13832 catalog_manager.cc:5484] T d22aa9aee27e43a79dd19934de05c43c P d43426b5bc874e8482e7b3205a9ba970 reported cstate change: config changed from index -1 to 4, NON_VOTER 48a417d95b604326aa5a918ff2f806b7 (127.11.42.72) added. New cstate: current_term: 1 leader_uuid: "d43426b5bc874e8482e7b3205a9ba970" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48a417d95b604326aa5a918ff2f806b7" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 45917 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:53:03.449471 13913 consensus_peers.cc:485] T d22aa9aee27e43a79dd19934de05c43c P d43426b5bc874e8482e7b3205a9ba970 -> Peer 48a417d95b604326aa5a918ff2f806b7 (127.11.42.72:45917): Couldn't send request to peer 48a417d95b604326aa5a918ff2f806b7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d22aa9aee27e43a79dd19934de05c43c. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:03.457057 14110 consensus_queue.cc:237] T 2d8fb038394d4ad981d208e52b19672f P 1478241a03934d029dc7030480423cab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "48a417d95b604326aa5a918ff2f806b7" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 45917 } attrs { promote: true } }
I20240502 01:53:03.459072 16974 raft_consensus.cc:2932] T 2d8fb038394d4ad981d208e52b19672f P 1478241a03934d029dc7030480423cab [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 48a417d95b604326aa5a918ff2f806b7 (127.11.42.72) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "48a417d95b604326aa5a918ff2f806b7" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 45917 } attrs { promote: true } } }
I20240502 01:53:03.464304 13832 catalog_manager.cc:5484] T 2d8fb038394d4ad981d208e52b19672f P 1478241a03934d029dc7030480423cab reported cstate change: config changed from index -1 to 4, NON_VOTER 48a417d95b604326aa5a918ff2f806b7 (127.11.42.72) added. New cstate: current_term: 1 leader_uuid: "1478241a03934d029dc7030480423cab" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48a417d95b604326aa5a918ff2f806b7" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 45917 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:53:03.467336 14046 consensus_peers.cc:485] T 2d8fb038394d4ad981d208e52b19672f P 1478241a03934d029dc7030480423cab -> Peer 48a417d95b604326aa5a918ff2f806b7 (127.11.42.72:45917): Couldn't send request to peer 48a417d95b604326aa5a918ff2f806b7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2d8fb038394d4ad981d208e52b19672f. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:03.537348 16867 heartbeater.cc:503] Master 127.11.42.126:40815 was elected leader, sending a full tablet report...
I20240502 01:53:03.957152 17007 ts_tablet_manager.cc:902] T 2d8fb038394d4ad981d208e52b19672f P 48a417d95b604326aa5a918ff2f806b7: Initiating tablet copy from peer 1478241a03934d029dc7030480423cab (127.11.42.66:34613)
I20240502 01:53:03.958910 17007 tablet_copy_client.cc:308] T 2d8fb038394d4ad981d208e52b19672f P 48a417d95b604326aa5a918ff2f806b7: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.66:34613
I20240502 01:53:03.965092 14131 tablet_copy_service.cc:140] P 1478241a03934d029dc7030480423cab: Received BeginTabletCopySession request for tablet 2d8fb038394d4ad981d208e52b19672f from peer 48a417d95b604326aa5a918ff2f806b7 ({username='slave'} at 127.11.42.72:44767)
I20240502 01:53:03.965245 14131 tablet_copy_service.cc:161] P 1478241a03934d029dc7030480423cab: Beginning new tablet copy session on tablet 2d8fb038394d4ad981d208e52b19672f from peer 48a417d95b604326aa5a918ff2f806b7 at {username='slave'} at 127.11.42.72:44767: session id = 48a417d95b604326aa5a918ff2f806b7-2d8fb038394d4ad981d208e52b19672f
I20240502 01:53:03.967599 14131 tablet_copy_source_session.cc:205] T 2d8fb038394d4ad981d208e52b19672f P 1478241a03934d029dc7030480423cab: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:03.971772 17007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d8fb038394d4ad981d208e52b19672f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:03.981326 17007 tablet_copy_client.cc:783] T 2d8fb038394d4ad981d208e52b19672f P 48a417d95b604326aa5a918ff2f806b7: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:03.981637 17007 tablet_copy_client.cc:647] T 2d8fb038394d4ad981d208e52b19672f P 48a417d95b604326aa5a918ff2f806b7: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:03.983633 17007 tablet_copy_client.cc:523] T 2d8fb038394d4ad981d208e52b19672f P 48a417d95b604326aa5a918ff2f806b7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:03.986730 17007 tablet_bootstrap.cc:492] T 2d8fb038394d4ad981d208e52b19672f P 48a417d95b604326aa5a918ff2f806b7: Bootstrap starting.
I20240502 01:53:03.995815 17007 log.cc:824] T 2d8fb038394d4ad981d208e52b19672f P 48a417d95b604326aa5a918ff2f806b7: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:04.025089 17007 tablet_bootstrap.cc:492] T 2d8fb038394d4ad981d208e52b19672f P 48a417d95b604326aa5a918ff2f806b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:04.025528 17007 tablet_bootstrap.cc:492] T 2d8fb038394d4ad981d208e52b19672f P 48a417d95b604326aa5a918ff2f806b7: Bootstrap complete.
I20240502 01:53:04.025921 17007 ts_tablet_manager.cc:1367] T 2d8fb038394d4ad981d208e52b19672f P 48a417d95b604326aa5a918ff2f806b7: Time spent bootstrapping tablet: real 0.039s user 0.039s sys 0.001s
I20240502 01:53:04.030411 17007 raft_consensus.cc:348] T 2d8fb038394d4ad981d208e52b19672f P 48a417d95b604326aa5a918ff2f806b7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "48a417d95b604326aa5a918ff2f806b7" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 45917 } attrs { promote: true } }
I20240502 01:53:04.030802 17007 raft_consensus.cc:729] T 2d8fb038394d4ad981d208e52b19672f P 48a417d95b604326aa5a918ff2f806b7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 48a417d95b604326aa5a918ff2f806b7, State: Initialized, Role: LEARNER
I20240502 01:53:04.031252 17007 consensus_queue.cc:260] T 2d8fb038394d4ad981d208e52b19672f P 48a417d95b604326aa5a918ff2f806b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "48a417d95b604326aa5a918ff2f806b7" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 45917 } attrs { promote: true } }
I20240502 01:53:04.032806 17007 ts_tablet_manager.cc:1398] T 2d8fb038394d4ad981d208e52b19672f P 48a417d95b604326aa5a918ff2f806b7: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240502 01:53:04.033898 14131 tablet_copy_service.cc:342] P 1478241a03934d029dc7030480423cab: Request end of tablet copy session 48a417d95b604326aa5a918ff2f806b7-2d8fb038394d4ad981d208e52b19672f received from {username='slave'} at 127.11.42.72:44767
I20240502 01:53:04.034071 14131 tablet_copy_service.cc:433] P 1478241a03934d029dc7030480423cab: ending tablet copy session 48a417d95b604326aa5a918ff2f806b7-2d8fb038394d4ad981d208e52b19672f on tablet 2d8fb038394d4ad981d208e52b19672f with peer 48a417d95b604326aa5a918ff2f806b7
I20240502 01:53:04.048322 17007 ts_tablet_manager.cc:902] T d22aa9aee27e43a79dd19934de05c43c P 48a417d95b604326aa5a918ff2f806b7: Initiating tablet copy from peer d43426b5bc874e8482e7b3205a9ba970 (127.11.42.65:40383)
I20240502 01:53:04.049511 17007 tablet_copy_client.cc:308] T d22aa9aee27e43a79dd19934de05c43c P 48a417d95b604326aa5a918ff2f806b7: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.65:40383
I20240502 01:53:04.054507 13997 tablet_copy_service.cc:140] P d43426b5bc874e8482e7b3205a9ba970: Received BeginTabletCopySession request for tablet d22aa9aee27e43a79dd19934de05c43c from peer 48a417d95b604326aa5a918ff2f806b7 ({username='slave'} at 127.11.42.72:47821)
I20240502 01:53:04.054647 13997 tablet_copy_service.cc:161] P d43426b5bc874e8482e7b3205a9ba970: Beginning new tablet copy session on tablet d22aa9aee27e43a79dd19934de05c43c from peer 48a417d95b604326aa5a918ff2f806b7 at {username='slave'} at 127.11.42.72:47821: session id = 48a417d95b604326aa5a918ff2f806b7-d22aa9aee27e43a79dd19934de05c43c
I20240502 01:53:04.056612 13997 tablet_copy_source_session.cc:205] T d22aa9aee27e43a79dd19934de05c43c P d43426b5bc874e8482e7b3205a9ba970: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:04.057729 17007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d22aa9aee27e43a79dd19934de05c43c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:04.061985 17007 tablet_copy_client.cc:783] T d22aa9aee27e43a79dd19934de05c43c P 48a417d95b604326aa5a918ff2f806b7: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:04.062186 17007 tablet_copy_client.cc:647] T d22aa9aee27e43a79dd19934de05c43c P 48a417d95b604326aa5a918ff2f806b7: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:04.063743 17007 tablet_copy_client.cc:523] T d22aa9aee27e43a79dd19934de05c43c P 48a417d95b604326aa5a918ff2f806b7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:04.066186 17007 tablet_bootstrap.cc:492] T d22aa9aee27e43a79dd19934de05c43c P 48a417d95b604326aa5a918ff2f806b7: Bootstrap starting.
I20240502 01:53:04.096494 17007 tablet_bootstrap.cc:492] T d22aa9aee27e43a79dd19934de05c43c P 48a417d95b604326aa5a918ff2f806b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:04.096886 17007 tablet_bootstrap.cc:492] T d22aa9aee27e43a79dd19934de05c43c P 48a417d95b604326aa5a918ff2f806b7: Bootstrap complete.
I20240502 01:53:04.097163 17007 ts_tablet_manager.cc:1367] T d22aa9aee27e43a79dd19934de05c43c P 48a417d95b604326aa5a918ff2f806b7: Time spent bootstrapping tablet: real 0.031s user 0.031s sys 0.001s
I20240502 01:53:04.097961 17007 raft_consensus.cc:348] T d22aa9aee27e43a79dd19934de05c43c P 48a417d95b604326aa5a918ff2f806b7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "48a417d95b604326aa5a918ff2f806b7" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 45917 } attrs { promote: true } }
I20240502 01:53:04.098153 17007 raft_consensus.cc:729] T d22aa9aee27e43a79dd19934de05c43c P 48a417d95b604326aa5a918ff2f806b7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 48a417d95b604326aa5a918ff2f806b7, State: Initialized, Role: LEARNER
I20240502 01:53:04.098349 17007 consensus_queue.cc:260] T d22aa9aee27e43a79dd19934de05c43c P 48a417d95b604326aa5a918ff2f806b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "48a417d95b604326aa5a918ff2f806b7" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 45917 } attrs { promote: true } }
I20240502 01:53:04.098846 17007 ts_tablet_manager.cc:1398] T d22aa9aee27e43a79dd19934de05c43c P 48a417d95b604326aa5a918ff2f806b7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:53:04.099792 13997 tablet_copy_service.cc:342] P d43426b5bc874e8482e7b3205a9ba970: Request end of tablet copy session 48a417d95b604326aa5a918ff2f806b7-d22aa9aee27e43a79dd19934de05c43c received from {username='slave'} at 127.11.42.72:47821
I20240502 01:53:04.099952 13997 tablet_copy_service.cc:433] P d43426b5bc874e8482e7b3205a9ba970: ending tablet copy session 48a417d95b604326aa5a918ff2f806b7-d22aa9aee27e43a79dd19934de05c43c on tablet d22aa9aee27e43a79dd19934de05c43c with peer 48a417d95b604326aa5a918ff2f806b7
I20240502 01:53:04.388967 16822 raft_consensus.cc:1203] T 2d8fb038394d4ad981d208e52b19672f P 48a417d95b604326aa5a918ff2f806b7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:53:04.591652 16822 raft_consensus.cc:1203] T d22aa9aee27e43a79dd19934de05c43c P 48a417d95b604326aa5a918ff2f806b7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:53:04.790256 17042 raft_consensus.cc:1050] T 2d8fb038394d4ad981d208e52b19672f P 1478241a03934d029dc7030480423cab: attempting to promote NON_VOTER 48a417d95b604326aa5a918ff2f806b7 to VOTER
I20240502 01:53:04.790798 17042 consensus_queue.cc:237] T 2d8fb038394d4ad981d208e52b19672f P 1478241a03934d029dc7030480423cab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "48a417d95b604326aa5a918ff2f806b7" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 45917 } attrs { promote: false } }
I20240502 01:53:04.792654 16822 raft_consensus.cc:1261] T 2d8fb038394d4ad981d208e52b19672f P 48a417d95b604326aa5a918ff2f806b7 [term 1 LEARNER]: Refusing update from remote peer 1478241a03934d029dc7030480423cab: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:04.793205 17041 consensus_queue.cc:1035] T 2d8fb038394d4ad981d208e52b19672f P 1478241a03934d029dc7030480423cab [LEADER]: Connected to new peer: Peer: permanent_uuid: "48a417d95b604326aa5a918ff2f806b7" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 45917 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:04.796319 17042 raft_consensus.cc:2932] T 2d8fb038394d4ad981d208e52b19672f P 1478241a03934d029dc7030480423cab [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 48a417d95b604326aa5a918ff2f806b7 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "48a417d95b604326aa5a918ff2f806b7" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 45917 } attrs { promote: false } } }
I20240502 01:53:04.797080 16822 raft_consensus.cc:2932] T 2d8fb038394d4ad981d208e52b19672f P 48a417d95b604326aa5a918ff2f806b7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 48a417d95b604326aa5a918ff2f806b7 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "48a417d95b604326aa5a918ff2f806b7" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 45917 } attrs { promote: false } } }
I20240502 01:53:04.801278 13832 catalog_manager.cc:5484] T 2d8fb038394d4ad981d208e52b19672f P 1478241a03934d029dc7030480423cab reported cstate change: config changed from index 4 to 5, 48a417d95b604326aa5a918ff2f806b7 (127.11.42.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1478241a03934d029dc7030480423cab" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48a417d95b604326aa5a918ff2f806b7" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 45917 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:05.006261 14110 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2d8fb038394d4ad981d208e52b19672f"
dest_uuid: "1478241a03934d029dc7030480423cab"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40578
I20240502 01:53:05.006490 14110 raft_consensus.cc:595] T 2d8fb038394d4ad981d208e52b19672f P 1478241a03934d029dc7030480423cab [term 1 LEADER]: Received request to transfer leadership
I20240502 01:53:05.013559 17055 raft_consensus.cc:1050] T d22aa9aee27e43a79dd19934de05c43c P d43426b5bc874e8482e7b3205a9ba970: attempting to promote NON_VOTER 48a417d95b604326aa5a918ff2f806b7 to VOTER
I20240502 01:53:05.014268 17055 consensus_queue.cc:237] T d22aa9aee27e43a79dd19934de05c43c P d43426b5bc874e8482e7b3205a9ba970 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "48a417d95b604326aa5a918ff2f806b7" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 45917 } attrs { promote: false } }
I20240502 01:53:05.016124 16822 raft_consensus.cc:1261] T d22aa9aee27e43a79dd19934de05c43c P 48a417d95b604326aa5a918ff2f806b7 [term 1 LEARNER]: Refusing update from remote peer d43426b5bc874e8482e7b3205a9ba970: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:05.016791 17054 consensus_queue.cc:1035] T d22aa9aee27e43a79dd19934de05c43c P d43426b5bc874e8482e7b3205a9ba970 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48a417d95b604326aa5a918ff2f806b7" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 45917 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:05.019615 17055 raft_consensus.cc:2932] T d22aa9aee27e43a79dd19934de05c43c P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 48a417d95b604326aa5a918ff2f806b7 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "48a417d95b604326aa5a918ff2f806b7" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 45917 } attrs { promote: false } } }
I20240502 01:53:05.020166 16822 raft_consensus.cc:2932] T d22aa9aee27e43a79dd19934de05c43c P 48a417d95b604326aa5a918ff2f806b7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 48a417d95b604326aa5a918ff2f806b7 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "48a417d95b604326aa5a918ff2f806b7" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 45917 } attrs { promote: false } } }
I20240502 01:53:05.023547 13831 catalog_manager.cc:5484] T d22aa9aee27e43a79dd19934de05c43c P d43426b5bc874e8482e7b3205a9ba970 reported cstate change: config changed from index 4 to 5, 48a417d95b604326aa5a918ff2f806b7 (127.11.42.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d43426b5bc874e8482e7b3205a9ba970" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48a417d95b604326aa5a918ff2f806b7" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 45917 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:05.235034 14110 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2d8fb038394d4ad981d208e52b19672f"
dest_uuid: "1478241a03934d029dc7030480423cab"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40594
I20240502 01:53:05.235247 14110 raft_consensus.cc:595] T 2d8fb038394d4ad981d208e52b19672f P 1478241a03934d029dc7030480423cab [term 1 LEADER]: Received request to transfer leadership
I20240502 01:53:05.249123 17042 raft_consensus.cc:979] T 2d8fb038394d4ad981d208e52b19672f P 1478241a03934d029dc7030480423cab: : Instructing follower 48a417d95b604326aa5a918ff2f806b7 to start an election
I20240502 01:53:05.249300 17042 raft_consensus.cc:1067] T 2d8fb038394d4ad981d208e52b19672f P 1478241a03934d029dc7030480423cab [term 1 LEADER]: Signalling peer 48a417d95b604326aa5a918ff2f806b7 to start an election
I20240502 01:53:05.249990 16822 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2d8fb038394d4ad981d208e52b19672f"
dest_uuid: "48a417d95b604326aa5a918ff2f806b7"
from {username='slave'} at 127.11.42.66:47275
I20240502 01:53:05.250324 16822 raft_consensus.cc:482] T 2d8fb038394d4ad981d208e52b19672f P 48a417d95b604326aa5a918ff2f806b7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:53:05.250521 16822 raft_consensus.cc:3037] T 2d8fb038394d4ad981d208e52b19672f P 48a417d95b604326aa5a918ff2f806b7 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:05.253458 16822 raft_consensus.cc:504] T 2d8fb038394d4ad981d208e52b19672f P 48a417d95b604326aa5a918ff2f806b7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "48a417d95b604326aa5a918ff2f806b7" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 45917 } attrs { promote: false } }
I20240502 01:53:05.255019 16822 leader_election.cc:288] T 2d8fb038394d4ad981d208e52b19672f P 48a417d95b604326aa5a918ff2f806b7 [CANDIDATE]: Term 2 election: Requested vote from peers 1478241a03934d029dc7030480423cab (127.11.42.66:34613)
I20240502 01:53:05.255237 13977 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d22aa9aee27e43a79dd19934de05c43c"
dest_uuid: "d43426b5bc874e8482e7b3205a9ba970"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56220
I20240502 01:53:05.255465 13977 raft_consensus.cc:595] T d22aa9aee27e43a79dd19934de05c43c P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:53:05.255504 14110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d8fb038394d4ad981d208e52b19672f" candidate_uuid: "48a417d95b604326aa5a918ff2f806b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "1478241a03934d029dc7030480423cab"
I20240502 01:53:05.255718 14110 raft_consensus.cc:3032] T 2d8fb038394d4ad981d208e52b19672f P 1478241a03934d029dc7030480423cab [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:53:05.255796 14110 raft_consensus.cc:729] T 2d8fb038394d4ad981d208e52b19672f P 1478241a03934d029dc7030480423cab [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1478241a03934d029dc7030480423cab, State: Running, Role: LEADER
I20240502 01:53:05.256088 14110 consensus_queue.cc:260] T 2d8fb038394d4ad981d208e52b19672f P 1478241a03934d029dc7030480423cab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "48a417d95b604326aa5a918ff2f806b7" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 45917 } attrs { promote: false } }
I20240502 01:53:05.256428 14110 raft_consensus.cc:3037] T 2d8fb038394d4ad981d208e52b19672f P 1478241a03934d029dc7030480423cab [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:05.258405 14110 raft_consensus.cc:2454] T 2d8fb038394d4ad981d208e52b19672f P 1478241a03934d029dc7030480423cab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48a417d95b604326aa5a918ff2f806b7 in term 2.
I20240502 01:53:05.258937 16757 leader_election.cc:302] T 2d8fb038394d4ad981d208e52b19672f P 48a417d95b604326aa5a918ff2f806b7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1478241a03934d029dc7030480423cab, 48a417d95b604326aa5a918ff2f806b7; no voters:
I20240502 01:53:05.259290 17014 raft_consensus.cc:2781] T 2d8fb038394d4ad981d208e52b19672f P 48a417d95b604326aa5a918ff2f806b7 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:53:05.259688 17014 raft_consensus.cc:686] T 2d8fb038394d4ad981d208e52b19672f P 48a417d95b604326aa5a918ff2f806b7 [term 2 LEADER]: Becoming Leader. State: Replica: 48a417d95b604326aa5a918ff2f806b7, State: Running, Role: LEADER
I20240502 01:53:05.260082 17014 consensus_queue.cc:237] T 2d8fb038394d4ad981d208e52b19672f P 48a417d95b604326aa5a918ff2f806b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } } peers { permanent_uuid: "48a417d95b604326aa5a918ff2f806b7" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 45917 } attrs { promote: false } }
I20240502 01:53:05.263237 13832 catalog_manager.cc:5484] T 2d8fb038394d4ad981d208e52b19672f P 48a417d95b604326aa5a918ff2f806b7 reported cstate change: term changed from 1 to 2, leader changed from 1478241a03934d029dc7030480423cab (127.11.42.66) to 48a417d95b604326aa5a918ff2f806b7 (127.11.42.72). New cstate: current_term: 2 leader_uuid: "48a417d95b604326aa5a918ff2f806b7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48a417d95b604326aa5a918ff2f806b7" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 45917 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 01:53:05.272527 13822 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 2d8fb038394d4ad981d208e52b19672f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 48a417d95b604326aa5a918ff2f806b7 (127.11.42.72:45917) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240502 01:53:05.481673 13977 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d22aa9aee27e43a79dd19934de05c43c"
dest_uuid: "d43426b5bc874e8482e7b3205a9ba970"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56238
I20240502 01:53:05.481899 13977 raft_consensus.cc:595] T d22aa9aee27e43a79dd19934de05c43c P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:53:05.641484 17128 raft_consensus.cc:979] T d22aa9aee27e43a79dd19934de05c43c P d43426b5bc874e8482e7b3205a9ba970: : Instructing follower 48a417d95b604326aa5a918ff2f806b7 to start an election
I20240502 01:53:05.641650 17128 raft_consensus.cc:1067] T d22aa9aee27e43a79dd19934de05c43c P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Signalling peer 48a417d95b604326aa5a918ff2f806b7 to start an election
I20240502 01:53:05.642309 16822 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d22aa9aee27e43a79dd19934de05c43c"
dest_uuid: "48a417d95b604326aa5a918ff2f806b7"
from {username='slave'} at 127.11.42.65:40565
I20240502 01:53:05.642514 16822 raft_consensus.cc:482] T d22aa9aee27e43a79dd19934de05c43c P 48a417d95b604326aa5a918ff2f806b7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:53:05.642616 16822 raft_consensus.cc:3037] T d22aa9aee27e43a79dd19934de05c43c P 48a417d95b604326aa5a918ff2f806b7 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:05.644948 16822 raft_consensus.cc:504] T d22aa9aee27e43a79dd19934de05c43c P 48a417d95b604326aa5a918ff2f806b7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "48a417d95b604326aa5a918ff2f806b7" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 45917 } attrs { promote: false } }
I20240502 01:53:05.645749 16822 leader_election.cc:288] T d22aa9aee27e43a79dd19934de05c43c P 48a417d95b604326aa5a918ff2f806b7 [CANDIDATE]: Term 2 election: Requested vote from peers d43426b5bc874e8482e7b3205a9ba970 (127.11.42.65:40383)
I20240502 01:53:05.646345 13977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d22aa9aee27e43a79dd19934de05c43c" candidate_uuid: "48a417d95b604326aa5a918ff2f806b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "d43426b5bc874e8482e7b3205a9ba970"
I20240502 01:53:05.646595 13977 raft_consensus.cc:3032] T d22aa9aee27e43a79dd19934de05c43c P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:53:05.646700 13977 raft_consensus.cc:729] T d22aa9aee27e43a79dd19934de05c43c P d43426b5bc874e8482e7b3205a9ba970 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d43426b5bc874e8482e7b3205a9ba970, State: Running, Role: LEADER
I20240502 01:53:05.646983 13977 consensus_queue.cc:260] T d22aa9aee27e43a79dd19934de05c43c P d43426b5bc874e8482e7b3205a9ba970 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "48a417d95b604326aa5a918ff2f806b7" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 45917 } attrs { promote: false } }
I20240502 01:53:05.647322 13977 raft_consensus.cc:3037] T d22aa9aee27e43a79dd19934de05c43c P d43426b5bc874e8482e7b3205a9ba970 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:05.649278 13977 raft_consensus.cc:2454] T d22aa9aee27e43a79dd19934de05c43c P d43426b5bc874e8482e7b3205a9ba970 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48a417d95b604326aa5a918ff2f806b7 in term 2.
I20240502 01:53:05.649799 16755 leader_election.cc:302] T d22aa9aee27e43a79dd19934de05c43c P 48a417d95b604326aa5a918ff2f806b7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 48a417d95b604326aa5a918ff2f806b7, d43426b5bc874e8482e7b3205a9ba970; no voters:
I20240502 01:53:05.650121 17014 raft_consensus.cc:2781] T d22aa9aee27e43a79dd19934de05c43c P 48a417d95b604326aa5a918ff2f806b7 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:53:05.650338 17014 raft_consensus.cc:686] T d22aa9aee27e43a79dd19934de05c43c P 48a417d95b604326aa5a918ff2f806b7 [term 2 LEADER]: Becoming Leader. State: Replica: 48a417d95b604326aa5a918ff2f806b7, State: Running, Role: LEADER
I20240502 01:53:05.650624 17014 consensus_queue.cc:237] T d22aa9aee27e43a79dd19934de05c43c P 48a417d95b604326aa5a918ff2f806b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } } peers { permanent_uuid: "48a417d95b604326aa5a918ff2f806b7" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 45917 } attrs { promote: false } }
I20240502 01:53:05.652571 13832 catalog_manager.cc:5484] T d22aa9aee27e43a79dd19934de05c43c P 48a417d95b604326aa5a918ff2f806b7 reported cstate change: term changed from 1 to 2, leader changed from d43426b5bc874e8482e7b3205a9ba970 (127.11.42.65) to 48a417d95b604326aa5a918ff2f806b7 (127.11.42.72). New cstate: current_term: 2 leader_uuid: "48a417d95b604326aa5a918ff2f806b7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48a417d95b604326aa5a918ff2f806b7" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 45917 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:05.808522 14110 raft_consensus.cc:1261] T 2d8fb038394d4ad981d208e52b19672f P 1478241a03934d029dc7030480423cab [term 2 FOLLOWER]: Refusing update from remote peer 48a417d95b604326aa5a918ff2f806b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:53:05.809201 17014 consensus_queue.cc:1035] T 2d8fb038394d4ad981d208e52b19672f P 48a417d95b604326aa5a918ff2f806b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1478241a03934d029dc7030480423cab" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34613 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:53:05.897557 16822 consensus_queue.cc:237] T 2d8fb038394d4ad981d208e52b19672f P 48a417d95b604326aa5a918ff2f806b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "48a417d95b604326aa5a918ff2f806b7" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 45917 } attrs { promote: false } }
I20240502 01:53:05.898486 17109 raft_consensus.cc:2932] T 2d8fb038394d4ad981d208e52b19672f P 48a417d95b604326aa5a918ff2f806b7 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 1478241a03934d029dc7030480423cab (127.11.42.66) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "48a417d95b604326aa5a918ff2f806b7" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 45917 } attrs { promote: false } } }
I20240502 01:53:05.901279 13822 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2d8fb038394d4ad981d208e52b19672f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 01:53:05.902585 13832 catalog_manager.cc:5484] T 2d8fb038394d4ad981d208e52b19672f P 48a417d95b604326aa5a918ff2f806b7 reported cstate change: config changed from index 5 to 7, VOTER 1478241a03934d029dc7030480423cab (127.11.42.66) evicted. New cstate: current_term: 2 leader_uuid: "48a417d95b604326aa5a918ff2f806b7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "48a417d95b604326aa5a918ff2f806b7" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 45917 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:05.906157 14090 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d8fb038394d4ad981d208e52b19672f with delete_type TABLET_DATA_TOMBSTONED (TS 1478241a03934d029dc7030480423cab not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38844
I20240502 01:53:05.906875 17145 tablet_replica.cc:330] T 2d8fb038394d4ad981d208e52b19672f P 1478241a03934d029dc7030480423cab: stopping tablet replica
I20240502 01:53:05.907099 17145 raft_consensus.cc:2229] T 2d8fb038394d4ad981d208e52b19672f P 1478241a03934d029dc7030480423cab [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:05.907356 17145 raft_consensus.cc:2258] T 2d8fb038394d4ad981d208e52b19672f P 1478241a03934d029dc7030480423cab [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:05.909135 17145 ts_tablet_manager.cc:1875] T 2d8fb038394d4ad981d208e52b19672f P 1478241a03934d029dc7030480423cab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:05.914297 17145 ts_tablet_manager.cc:1888] T 2d8fb038394d4ad981d208e52b19672f P 1478241a03934d029dc7030480423cab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 01:53:05.914414 17145 log.cc:1196] T 2d8fb038394d4ad981d208e52b19672f P 1478241a03934d029dc7030480423cab: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/wals/2d8fb038394d4ad981d208e52b19672f
I20240502 01:53:05.915154 13821 catalog_manager.cc:4830] TS 1478241a03934d029dc7030480423cab (127.11.42.66:34613): tablet 2d8fb038394d4ad981d208e52b19672f (table rebalance_test_table_2 [id=3c737a3ccdb1430496def9e37e6428d9]) successfully deleted
I20240502 01:53:06.223548 13977 raft_consensus.cc:1261] T d22aa9aee27e43a79dd19934de05c43c P d43426b5bc874e8482e7b3205a9ba970 [term 2 FOLLOWER]: Refusing update from remote peer 48a417d95b604326aa5a918ff2f806b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:53:06.224224 17109 consensus_queue.cc:1035] T d22aa9aee27e43a79dd19934de05c43c P 48a417d95b604326aa5a918ff2f806b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d43426b5bc874e8482e7b3205a9ba970" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 40383 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:53:06.287292 16822 consensus_queue.cc:237] T d22aa9aee27e43a79dd19934de05c43c P 48a417d95b604326aa5a918ff2f806b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "48a417d95b604326aa5a918ff2f806b7" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 45917 } attrs { promote: false } }
I20240502 01:53:06.288242 17166 raft_consensus.cc:2932] T d22aa9aee27e43a79dd19934de05c43c P 48a417d95b604326aa5a918ff2f806b7 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER d43426b5bc874e8482e7b3205a9ba970 (127.11.42.65) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "48a417d95b604326aa5a918ff2f806b7" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 45917 } attrs { promote: false } } }
I20240502 01:53:06.291138 13822 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d22aa9aee27e43a79dd19934de05c43c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 01:53:06.292373 13832 catalog_manager.cc:5484] T d22aa9aee27e43a79dd19934de05c43c P 48a417d95b604326aa5a918ff2f806b7 reported cstate change: config changed from index 5 to 7, VOTER d43426b5bc874e8482e7b3205a9ba970 (127.11.42.65) evicted. New cstate: current_term: 2 leader_uuid: "48a417d95b604326aa5a918ff2f806b7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "48a417d95b604326aa5a918ff2f806b7" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 45917 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:06.295622 13957 tablet_service.cc:1514] Processing DeleteTablet for tablet d22aa9aee27e43a79dd19934de05c43c with delete_type TABLET_DATA_TOMBSTONED (TS d43426b5bc874e8482e7b3205a9ba970 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33834
I20240502 01:53:06.296226 17168 tablet_replica.cc:330] T d22aa9aee27e43a79dd19934de05c43c P d43426b5bc874e8482e7b3205a9ba970: stopping tablet replica
I20240502 01:53:06.296417 17168 raft_consensus.cc:2229] T d22aa9aee27e43a79dd19934de05c43c P d43426b5bc874e8482e7b3205a9ba970 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:06.296602 17168 raft_consensus.cc:2258] T d22aa9aee27e43a79dd19934de05c43c P d43426b5bc874e8482e7b3205a9ba970 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:06.298036 17168 ts_tablet_manager.cc:1875] T d22aa9aee27e43a79dd19934de05c43c P d43426b5bc874e8482e7b3205a9ba970: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:06.302261 17168 ts_tablet_manager.cc:1888] T d22aa9aee27e43a79dd19934de05c43c P d43426b5bc874e8482e7b3205a9ba970: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 01:53:06.302392 17168 log.cc:1196] T d22aa9aee27e43a79dd19934de05c43c P d43426b5bc874e8482e7b3205a9ba970: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/d22aa9aee27e43a79dd19934de05c43c
I20240502 01:53:06.303083 13819 catalog_manager.cc:4830] TS d43426b5bc874e8482e7b3205a9ba970 (127.11.42.65:40383): tablet d22aa9aee27e43a79dd19934de05c43c (table rebalance_test_table_3 [id=fb174a9542bb4a20bf8666a08c8a114e]) successfully deleted
I20240502 01:53:06.449188 15176 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:06.457495 14090 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:06.460283 15572 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:06.466121 15308 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:06.486184 16802 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:06.509657 15711 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:06.541158 13957 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:06.565541 15439 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:06.707399 15176 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:06.709059 14090 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:06.714259 15308 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:06.722605 15572 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:06.754065 16802 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:06.764597 15711 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:06.767195 15439 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:06.799659 13957 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:07.078821 15176 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:07.081234 15308 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:07.089290 14090 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:07.097487 16802 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:07.098567 15711 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:07.106477 15572 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:07.124327 15439 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:07.147387 13957 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d00df36391f54eaf96213f38c5e602a1 | 127.11.42.126:40815 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.11.42.84:45633 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
020e5f8ebb004e2ab6fddaece6663ece | 127.11.42.67:37505 | HEALTHY | <none> | 2 | 0
1478241a03934d029dc7030480423cab | 127.11.42.66:34613 | HEALTHY | <none> | 2 | 0
1607e211ee8c48de9fa56e78a3b01418 | 127.11.42.68:41779 | HEALTHY | <none> | 2 | 0
2ee638444633463b943c881866708482 | 127.11.42.70:45255 | HEALTHY | <none> | 2 | 0
39e5e5c1a5e74915800d96f720aee8c4 | 127.11.42.71:37071 | HEALTHY | <none> | 2 | 0
48a417d95b604326aa5a918ff2f806b7 | 127.11.42.72:45917 | HEALTHY | <none> | 2 | 0
95e7c90cdc634d64a66dad6805e1680c | 127.11.42.69:43981 | HEALTHY | <none> | 2 | 0
d43426b5bc874e8482e7b3205a9ba970 | 127.11.42.65:40383 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.11.42.65 | experimental | 127.11.42.65:40383
local_ip_for_outbound_sockets | 127.11.42.66 | experimental | 127.11.42.66:34613
local_ip_for_outbound_sockets | 127.11.42.67 | experimental | 127.11.42.67:37505
local_ip_for_outbound_sockets | 127.11.42.68 | experimental | 127.11.42.68:41779
local_ip_for_outbound_sockets | 127.11.42.69 | experimental | 127.11.42.69:43981
local_ip_for_outbound_sockets | 127.11.42.70 | experimental | 127.11.42.70:45255
local_ip_for_outbound_sockets | 127.11.42.71 | experimental | 127.11.42.71:37071
local_ip_for_outbound_sockets | 127.11.42.72 | experimental | 127.11.42.72:45917
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.11.42.65:40383
server_dump_info_path | /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.11.42.66:34613
server_dump_info_path | /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.11.42.67:37505
server_dump_info_path | /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.11.42.68:41779
server_dump_info_path | /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.11.42.69:43981
server_dump_info_path | /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.11.42.70:45255
server_dump_info_path | /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.11.42.71:37071
server_dump_info_path | /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.11.42.72:45917
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.11.42.84:45633 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 2
Maximum | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 01:53:07.205435 11433 log_verifier.cc:126] Checking tablet 0eb60206b8cb40d3b30cdc1f47d67f94
I20240502 01:53:07.215607 11433 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 0eb60206b8cb40d3b30cdc1f47d67f94
I20240502 01:53:07.215700 11433 log_verifier.cc:126] Checking tablet 2d8fb038394d4ad981d208e52b19672f
I20240502 01:53:07.230111 11433 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 2d8fb038394d4ad981d208e52b19672f
I20240502 01:53:07.230206 11433 log_verifier.cc:126] Checking tablet 50496c7daf804faa8d7bfaf128839e2c
I20240502 01:53:07.244727 11433 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 50496c7daf804faa8d7bfaf128839e2c
I20240502 01:53:07.244817 11433 log_verifier.cc:126] Checking tablet 563e1017b8774231bd29a1f8e628b4dc
I20240502 01:53:07.250140 11433 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 563e1017b8774231bd29a1f8e628b4dc
I20240502 01:53:07.250218 11433 log_verifier.cc:126] Checking tablet 5bc0ff324caa436196f22d90360c9247
I20240502 01:53:07.265182 11433 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 5bc0ff324caa436196f22d90360c9247
I20240502 01:53:07.265246 11433 log_verifier.cc:126] Checking tablet 62ad914a819c4c7a85856be3c1855270
I20240502 01:53:07.274236 11433 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 62ad914a819c4c7a85856be3c1855270
I20240502 01:53:07.274312 11433 log_verifier.cc:126] Checking tablet 65d44d1da4bf4366860fc5d1a1e0785c
I20240502 01:53:07.283273 11433 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 65d44d1da4bf4366860fc5d1a1e0785c
I20240502 01:53:07.283342 11433 log_verifier.cc:126] Checking tablet 67bc0cd4c3b94782ac1d9f150fcdbd03
I20240502 01:53:07.292351 11433 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 67bc0cd4c3b94782ac1d9f150fcdbd03
I20240502 01:53:07.292423 11433 log_verifier.cc:126] Checking tablet 6eb1311f3c054aec9a306922f25c6b89
I20240502 01:53:07.301841 11433 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 6eb1311f3c054aec9a306922f25c6b89
I20240502 01:53:07.301928 11433 log_verifier.cc:126] Checking tablet 6ed541d2e5ba4bbab19869c10116142e
I20240502 01:53:07.311650 11433 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 6ed541d2e5ba4bbab19869c10116142e
I20240502 01:53:07.311723 11433 log_verifier.cc:126] Checking tablet 86219a30419046dcb0384f5e539cdf15
I20240502 01:53:07.320537 11433 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 86219a30419046dcb0384f5e539cdf15
I20240502 01:53:07.320605 11433 log_verifier.cc:126] Checking tablet 954145d4672146ba8d26a25540de0325
I20240502 01:53:07.329217 11433 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 954145d4672146ba8d26a25540de0325
I20240502 01:53:07.329283 11433 log_verifier.cc:126] Checking tablet aa387d2b1dcc42ddb34005f08dde2d8d
I20240502 01:53:07.338661 11433 log_verifier.cc:177] Verified matching terms for 7 ops in tablet aa387d2b1dcc42ddb34005f08dde2d8d
I20240502 01:53:07.338734 11433 log_verifier.cc:126] Checking tablet c0620ccc47634258a7962d91cc7bb060
I20240502 01:53:07.348212 11433 log_verifier.cc:177] Verified matching terms for 7 ops in tablet c0620ccc47634258a7962d91cc7bb060
I20240502 01:53:07.348282 11433 log_verifier.cc:126] Checking tablet d22aa9aee27e43a79dd19934de05c43c
I20240502 01:53:07.360337 11433 log_verifier.cc:177] Verified matching terms for 7 ops in tablet d22aa9aee27e43a79dd19934de05c43c
I20240502 01:53:07.360412 11433 log_verifier.cc:126] Checking tablet f9f4aab07f984168bdfec24f908c124d
I20240502 01:53:07.369740 11433 log_verifier.cc:177] Verified matching terms for 7 ops in tablet f9f4aab07f984168bdfec24f908c124d
I20240502 01:53:07.370558 11433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu pid 13895
I20240502 01:53:07.371978 14019 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:07.513844 11433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu with pid 13895
I20240502 01:53:07.541059 11433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu pid 14028
I20240502 01:53:07.542124 14150 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:07.684835 11433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu with pid 14028
I20240502 01:53:07.705463 11433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu pid 15114
I20240502 01:53:07.711547 15236 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:07.838629 11433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu with pid 15114
W20240502 01:53:07.856379 13820 connection.cc:535] server connection from 127.11.42.67:54329 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:53:07.856585 11433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu pid 15246
I20240502 01:53:07.862015 15368 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:07.974800 11433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu with pid 15246
I20240502 01:53:07.993386 11433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu pid 15377
I20240502 01:53:07.999449 15498 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:08.112511 11433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu with pid 15377
I20240502 01:53:08.129981 11433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu pid 15508
I20240502 01:53:08.135298 15635 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:08.244956 11433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu with pid 15508
I20240502 01:53:08.262496 11433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu pid 15645
I20240502 01:53:08.268440 15771 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:08.379010 11433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu with pid 15645
I20240502 01:53:08.396574 11433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu pid 16740
I20240502 01:53:08.402222 16862 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:08.503542 11433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu with pid 16740
I20240502 01:53:08.520603 11433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu pid 13804
I20240502 01:53:08.521648 13866 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:08.672487 11433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu with pid 13804
2024-05-02T01:53:08Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/1 (25969 ms)
[----------] 1 test from RebalanceParamTest (25969 ms total)
[----------] 1 test from DDLDuringRebalancingTest
[ RUN ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/1
I20240502 01:53:08.732993 11433 test_util.cc:268] Using random seed: -580339668
I20240502 01:53:08.734750 11433 ts_itest-base.cc:112] Starting cluster with:
I20240502 01:53:08.734807 11433 ts_itest-base.cc:113] --------------
I20240502 01:53:08.734846 11433 ts_itest-base.cc:114] 8 tablet servers
I20240502 01:53:08.734880 11433 ts_itest-base.cc:115] 3 replicas per TS
I20240502 01:53:08.734925 11433 ts_itest-base.cc:116] --------------
2024-05-02T01:53:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T01:53:08Z Disabled control of system clock
I20240502 01:53:08.760344 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.42.126:39453
--webserver_interface=127.11.42.126
--webserver_port=0
--builtin_ntp_servers=127.11.42.84:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.42.126:39453
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 01:53:09.069993 17304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:09.070291 17304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:09.070416 17304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:09.077682 17304 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:53:09.077798 17304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 01:53:09.077877 17304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:09.077947 17304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 01:53:09.078009 17304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 01:53:09.086848 17304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.11.42.126:39453
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.42.126:39453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.11.42.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:09.088092 17304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:09.089840 17304 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:09.100327 17310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:09.100683 17309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:09.101111 17312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:09.101579 17304 server_base.cc:1034] running on GCE node
I20240502 01:53:09.102262 17304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:09.103596 17304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:09.104791 17304 hybrid_clock.cc:648] HybridClock initialized: now 1714614789104740 us; error 44 us; skew 500 ppm
I20240502 01:53:09.105142 17304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:09.107234 17304 webserver.cc:460] Webserver started at http://127.11.42.126:40687/ using document root <none> and password file <none>
I20240502 01:53:09.108021 17304 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:09.108143 17304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:09.108444 17304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:09.111057 17304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "b99aaf99a462480c9ce2bd089027d768"
format_stamp: "Formatted at 2024-05-02 01:53:09 on dist-test-slave-xvk5"
I20240502 01:53:09.111816 17304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "b99aaf99a462480c9ce2bd089027d768"
format_stamp: "Formatted at 2024-05-02 01:53:09 on dist-test-slave-xvk5"
I20240502 01:53:09.116627 17304 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20240502 01:53:09.119724 17318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:09.120970 17304 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:09.121161 17304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/wal
uuid: "b99aaf99a462480c9ce2bd089027d768"
format_stamp: "Formatted at 2024-05-02 01:53:09 on dist-test-slave-xvk5"
I20240502 01:53:09.121361 17304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:09.146080 17304 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:09.146869 17304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:09.147130 17304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:09.163941 17304 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.126:39453
I20240502 01:53:09.163990 17369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.126:39453 every 8 connection(s)
I20240502 01:53:09.165311 17304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 01:53:09.168854 17370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:09.174583 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 17304
I20240502 01:53:09.174784 11433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 01:53:09.178772 17370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Bootstrap starting.
I20240502 01:53:09.181686 17370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:09.182718 17370 log.cc:824] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:09.185212 17370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: No bootstrap required, opened a new log
I20240502 01:53:09.191664 17370 raft_consensus.cc:348] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b99aaf99a462480c9ce2bd089027d768" member_type: VOTER last_known_addr { host: "127.11.42.126" port: 39453 } }
I20240502 01:53:09.191954 17370 raft_consensus.cc:374] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:09.192061 17370 raft_consensus.cc:729] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b99aaf99a462480c9ce2bd089027d768, State: Initialized, Role: FOLLOWER
I20240502 01:53:09.192587 17370 consensus_queue.cc:260] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b99aaf99a462480c9ce2bd089027d768" member_type: VOTER last_known_addr { host: "127.11.42.126" port: 39453 } }
I20240502 01:53:09.192780 17370 raft_consensus.cc:388] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:53:09.192901 17370 raft_consensus.cc:482] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:53:09.193065 17370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:09.195160 17370 raft_consensus.cc:504] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b99aaf99a462480c9ce2bd089027d768" member_type: VOTER last_known_addr { host: "127.11.42.126" port: 39453 } }
I20240502 01:53:09.195601 17370 leader_election.cc:302] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b99aaf99a462480c9ce2bd089027d768; no voters:
I20240502 01:53:09.196161 17370 leader_election.cc:288] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:53:09.196326 17375 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:09.196894 17375 raft_consensus.cc:686] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768 [term 1 LEADER]: Becoming Leader. State: Replica: b99aaf99a462480c9ce2bd089027d768, State: Running, Role: LEADER
I20240502 01:53:09.197410 17375 consensus_queue.cc:237] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b99aaf99a462480c9ce2bd089027d768" member_type: VOTER last_known_addr { host: "127.11.42.126" port: 39453 } }
I20240502 01:53:09.198015 17370 sys_catalog.cc:564] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:53:09.200129 17376 sys_catalog.cc:455] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b99aaf99a462480c9ce2bd089027d768" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b99aaf99a462480c9ce2bd089027d768" member_type: VOTER last_known_addr { host: "127.11.42.126" port: 39453 } } }
I20240502 01:53:09.200510 17376 sys_catalog.cc:458] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768 [sys.catalog]: This master's current role is: LEADER
I20240502 01:53:09.200151 17377 sys_catalog.cc:455] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b99aaf99a462480c9ce2bd089027d768. Latest consensus state: current_term: 1 leader_uuid: "b99aaf99a462480c9ce2bd089027d768" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b99aaf99a462480c9ce2bd089027d768" member_type: VOTER last_known_addr { host: "127.11.42.126" port: 39453 } } }
I20240502 01:53:09.200851 17377 sys_catalog.cc:458] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768 [sys.catalog]: This master's current role is: LEADER
I20240502 01:53:09.201956 17383 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:53:09.206271 17383 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:53:09.214332 17383 catalog_manager.cc:1347] Generated new cluster ID: f9fdf54a0208476181ae0bf96add7998
I20240502 01:53:09.214437 17383 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:53:09.249321 17383 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:53:09.250952 17383 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:53:09.262476 17383 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Generated new TSK 0
I20240502 01:53:09.263428 17383 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:53:09.273857 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.65:0
--local_ip_for_outbound_sockets=127.11.42.65
--webserver_interface=127.11.42.65
--webserver_port=0
--tserver_master_addrs=127.11.42.126:39453
--builtin_ntp_servers=127.11.42.84:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 01:53:09.591713 17394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:09.591964 17394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:09.592056 17394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:09.599154 17394 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:53:09.599263 17394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:09.599401 17394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.65
I20240502 01:53:09.607959 17394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.11.42.65
--webserver_port=0
--tserver_master_addrs=127.11.42.126:39453
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.65
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:09.609117 17394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:09.610786 17394 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:09.622128 17400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:09.622515 17399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:09.622880 17402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:09.623406 17394 server_base.cc:1034] running on GCE node
I20240502 01:53:09.623888 17394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:09.624696 17394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:09.625878 17394 hybrid_clock.cc:648] HybridClock initialized: now 1714614789625831 us; error 35 us; skew 500 ppm
I20240502 01:53:09.626144 17394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:09.628173 17394 webserver.cc:460] Webserver started at http://127.11.42.65:41405/ using document root <none> and password file <none>
I20240502 01:53:09.628809 17394 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:09.628907 17394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:09.629201 17394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:09.631681 17394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "4da690c0fa13407a85c0689ce8de9ade"
format_stamp: "Formatted at 2024-05-02 01:53:09 on dist-test-slave-xvk5"
I20240502 01:53:09.632407 17394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "4da690c0fa13407a85c0689ce8de9ade"
format_stamp: "Formatted at 2024-05-02 01:53:09 on dist-test-slave-xvk5"
I20240502 01:53:09.637234 17394 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240502 01:53:09.640251 17408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:09.641506 17394 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:53:09.641646 17394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "4da690c0fa13407a85c0689ce8de9ade"
format_stamp: "Formatted at 2024-05-02 01:53:09 on dist-test-slave-xvk5"
I20240502 01:53:09.641821 17394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:09.662961 17394 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:09.663697 17394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:09.663942 17394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:09.664796 17394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:09.666221 17394 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:09.666323 17394 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:09.666424 17394 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:09.666491 17394 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:09.696050 17394 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.65:38321
I20240502 01:53:09.696112 17520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.65:38321 every 8 connection(s)
I20240502 01:53:09.697453 17394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 01:53:09.702004 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 17394
I20240502 01:53:09.702195 11433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 01:53:09.705245 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.66:0
--local_ip_for_outbound_sockets=127.11.42.66
--webserver_interface=127.11.42.66
--webserver_port=0
--tserver_master_addrs=127.11.42.126:39453
--builtin_ntp_servers=127.11.42.84:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:53:09.710031 17521 heartbeater.cc:347] Connected to a master server at 127.11.42.126:39453
I20240502 01:53:09.710328 17521 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:09.711258 17521 heartbeater.cc:511] Master 127.11.42.126:39453 requested a full tablet report, sending...
I20240502 01:53:09.713400 17335 ts_manager.cc:194] Registered new tserver with Master: 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:09.715376 17335 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.65:37805
W20240502 01:53:10.015389 17525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:10.015643 17525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:10.015738 17525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:10.022802 17525 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:53:10.022903 17525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:10.023017 17525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.66
I20240502 01:53:10.031733 17525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.11.42.66
--webserver_port=0
--tserver_master_addrs=127.11.42.126:39453
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.66
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:10.032871 17525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:10.034564 17525 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:10.045799 17530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:10.046252 17533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:10.046665 17531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:10.047191 17525 server_base.cc:1034] running on GCE node
I20240502 01:53:10.047732 17525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:10.048589 17525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:10.049799 17525 hybrid_clock.cc:648] HybridClock initialized: now 1714614790049744 us; error 46 us; skew 500 ppm
I20240502 01:53:10.050098 17525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:10.052235 17525 webserver.cc:460] Webserver started at http://127.11.42.66:38735/ using document root <none> and password file <none>
I20240502 01:53:10.052937 17525 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:10.053071 17525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:10.053380 17525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:10.055831 17525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "3fceccefba274b1dbbd8216af8b0bc29"
format_stamp: "Formatted at 2024-05-02 01:53:10 on dist-test-slave-xvk5"
I20240502 01:53:10.056568 17525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "3fceccefba274b1dbbd8216af8b0bc29"
format_stamp: "Formatted at 2024-05-02 01:53:10 on dist-test-slave-xvk5"
I20240502 01:53:10.061470 17525 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240502 01:53:10.064617 17539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:10.066007 17525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:53:10.066167 17525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "3fceccefba274b1dbbd8216af8b0bc29"
format_stamp: "Formatted at 2024-05-02 01:53:10 on dist-test-slave-xvk5"
I20240502 01:53:10.066361 17525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:10.084774 17525 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:10.085532 17525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:10.085778 17525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:10.086637 17525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:10.088117 17525 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:10.088215 17525 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:10.088325 17525 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:10.088395 17525 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:10.117165 17525 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.66:38531
I20240502 01:53:10.117221 17651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.66:38531 every 8 connection(s)
I20240502 01:53:10.118353 17525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 01:53:10.125216 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 17525
I20240502 01:53:10.125387 11433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 01:53:10.128929 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.67:0
--local_ip_for_outbound_sockets=127.11.42.67
--webserver_interface=127.11.42.67
--webserver_port=0
--tserver_master_addrs=127.11.42.126:39453
--builtin_ntp_servers=127.11.42.84:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:53:10.130770 17652 heartbeater.cc:347] Connected to a master server at 127.11.42.126:39453
I20240502 01:53:10.131070 17652 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:10.131970 17652 heartbeater.cc:511] Master 127.11.42.126:39453 requested a full tablet report, sending...
I20240502 01:53:10.133417 17335 ts_manager.cc:194] Registered new tserver with Master: 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:10.134352 17335 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.66:45283
W20240502 01:53:10.438227 17655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:10.438479 17655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:10.438602 17655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:10.446079 17655 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:53:10.446187 17655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:10.446346 17655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.67
I20240502 01:53:10.454905 17655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.11.42.67
--webserver_port=0
--tserver_master_addrs=127.11.42.126:39453
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.67
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:10.456071 17655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:10.457736 17655 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:10.468756 17660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:10.469247 17661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:10.469537 17663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:10.470037 17655 server_base.cc:1034] running on GCE node
I20240502 01:53:10.470587 17655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:10.471460 17655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:10.472649 17655 hybrid_clock.cc:648] HybridClock initialized: now 1714614790472607 us; error 39 us; skew 500 ppm
I20240502 01:53:10.472923 17655 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:10.474937 17655 webserver.cc:460] Webserver started at http://127.11.42.67:43503/ using document root <none> and password file <none>
I20240502 01:53:10.475600 17655 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:10.475704 17655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:10.475986 17655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:10.478399 17655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "dfa1152987644bd781b46c7e42a7a685"
format_stamp: "Formatted at 2024-05-02 01:53:10 on dist-test-slave-xvk5"
I20240502 01:53:10.479105 17655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "dfa1152987644bd781b46c7e42a7a685"
format_stamp: "Formatted at 2024-05-02 01:53:10 on dist-test-slave-xvk5"
I20240502 01:53:10.483763 17655 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 01:53:10.486791 17669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:10.487948 17655 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:10.488090 17655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "dfa1152987644bd781b46c7e42a7a685"
format_stamp: "Formatted at 2024-05-02 01:53:10 on dist-test-slave-xvk5"
I20240502 01:53:10.488258 17655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:10.507236 17655 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:10.507939 17655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:10.508174 17655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:10.509001 17655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:10.510366 17655 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:10.510463 17655 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:10.510565 17655 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:10.510633 17655 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:10.539168 17655 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.67:36115
I20240502 01:53:10.539243 17781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.67:36115 every 8 connection(s)
I20240502 01:53:10.540351 17655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 01:53:10.548683 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 17655
I20240502 01:53:10.548861 11433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 01:53:10.551995 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.68:0
--local_ip_for_outbound_sockets=127.11.42.68
--webserver_interface=127.11.42.68
--webserver_port=0
--tserver_master_addrs=127.11.42.126:39453
--builtin_ntp_servers=127.11.42.84:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:53:10.552245 17782 heartbeater.cc:347] Connected to a master server at 127.11.42.126:39453
I20240502 01:53:10.552533 17782 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:10.553494 17782 heartbeater.cc:511] Master 127.11.42.126:39453 requested a full tablet report, sending...
I20240502 01:53:10.554929 17335 ts_manager.cc:194] Registered new tserver with Master: dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:53:10.555691 17335 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.67:47713
I20240502 01:53:10.718057 17521 heartbeater.cc:503] Master 127.11.42.126:39453 was elected leader, sending a full tablet report...
W20240502 01:53:10.858537 17785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:10.858772 17785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:10.858861 17785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:10.866075 17785 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:53:10.866169 17785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:10.866269 17785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.68
I20240502 01:53:10.874619 17785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.11.42.68
--webserver_port=0
--tserver_master_addrs=127.11.42.126:39453
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.68
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:10.875634 17785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:10.877372 17785 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:10.888078 17790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:10.888505 17791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:10.888854 17793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:10.889381 17785 server_base.cc:1034] running on GCE node
I20240502 01:53:10.889892 17785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:10.890691 17785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:10.891888 17785 hybrid_clock.cc:648] HybridClock initialized: now 1714614790891845 us; error 38 us; skew 500 ppm
I20240502 01:53:10.892187 17785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:10.894194 17785 webserver.cc:460] Webserver started at http://127.11.42.68:45071/ using document root <none> and password file <none>
I20240502 01:53:10.894848 17785 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:10.894954 17785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:10.895231 17785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:10.897609 17785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "bb31bceba76d466db24b9341c04900c3"
format_stamp: "Formatted at 2024-05-02 01:53:10 on dist-test-slave-xvk5"
I20240502 01:53:10.898304 17785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "bb31bceba76d466db24b9341c04900c3"
format_stamp: "Formatted at 2024-05-02 01:53:10 on dist-test-slave-xvk5"
I20240502 01:53:10.903044 17785 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 01:53:10.906070 17799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:10.907243 17785 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20240502 01:53:10.907382 17785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "bb31bceba76d466db24b9341c04900c3"
format_stamp: "Formatted at 2024-05-02 01:53:10 on dist-test-slave-xvk5"
I20240502 01:53:10.907557 17785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:10.924237 17785 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:10.924908 17785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:10.925199 17785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:10.926028 17785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:10.927364 17785 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:10.927461 17785 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:10.927560 17785 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:10.927628 17785 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:10.956136 17785 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.68:44083
I20240502 01:53:10.956192 17911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.68:44083 every 8 connection(s)
I20240502 01:53:10.957324 17785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 01:53:10.961784 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 17785
I20240502 01:53:10.961957 11433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 01:53:10.965196 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.69:0
--local_ip_for_outbound_sockets=127.11.42.69
--webserver_interface=127.11.42.69
--webserver_port=0
--tserver_master_addrs=127.11.42.126:39453
--builtin_ntp_servers=127.11.42.84:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:53:10.969867 17912 heartbeater.cc:347] Connected to a master server at 127.11.42.126:39453
I20240502 01:53:10.970153 17912 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:10.971055 17912 heartbeater.cc:511] Master 127.11.42.126:39453 requested a full tablet report, sending...
I20240502 01:53:10.972492 17335 ts_manager.cc:194] Registered new tserver with Master: bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:53:10.973546 17335 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.68:60883
I20240502 01:53:11.138415 17652 heartbeater.cc:503] Master 127.11.42.126:39453 was elected leader, sending a full tablet report...
W20240502 01:53:11.272426 17915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:11.272665 17915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:11.272781 17915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:11.279800 17915 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:53:11.279897 17915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:11.280046 17915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.69
I20240502 01:53:11.288349 17915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.11.42.69
--webserver_port=0
--tserver_master_addrs=127.11.42.126:39453
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.69
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:11.289490 17915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:11.291066 17915 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:11.302250 17923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:11.302651 17921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:11.302968 17920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:11.303434 17915 server_base.cc:1034] running on GCE node
I20240502 01:53:11.303984 17915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:11.304776 17915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:11.305974 17915 hybrid_clock.cc:648] HybridClock initialized: now 1714614791305923 us; error 34 us; skew 500 ppm
I20240502 01:53:11.306252 17915 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:11.308283 17915 webserver.cc:460] Webserver started at http://127.11.42.69:37073/ using document root <none> and password file <none>
I20240502 01:53:11.308912 17915 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:11.309001 17915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:11.309298 17915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:11.311730 17915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "fdc2ac372492454c85387589a91f63af"
format_stamp: "Formatted at 2024-05-02 01:53:11 on dist-test-slave-xvk5"
I20240502 01:53:11.312476 17915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "fdc2ac372492454c85387589a91f63af"
format_stamp: "Formatted at 2024-05-02 01:53:11 on dist-test-slave-xvk5"
I20240502 01:53:11.317138 17915 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.003s
I20240502 01:53:11.320500 17929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:11.321712 17915 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:53:11.321857 17915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "fdc2ac372492454c85387589a91f63af"
format_stamp: "Formatted at 2024-05-02 01:53:11 on dist-test-slave-xvk5"
I20240502 01:53:11.322013 17915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:11.340618 17915 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:11.341437 17915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:11.341667 17915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:11.342454 17915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:11.343731 17915 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:11.343811 17915 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:11.343885 17915 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:11.343936 17915 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:11.371753 17915 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.69:34339
I20240502 01:53:11.371811 18041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.69:34339 every 8 connection(s)
I20240502 01:53:11.372853 17915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:53:11.374752 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 17915
I20240502 01:53:11.374924 11433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 01:53:11.378291 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.70:0
--local_ip_for_outbound_sockets=127.11.42.70
--webserver_interface=127.11.42.70
--webserver_port=0
--tserver_master_addrs=127.11.42.126:39453
--builtin_ntp_servers=127.11.42.84:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:53:11.385169 18042 heartbeater.cc:347] Connected to a master server at 127.11.42.126:39453
I20240502 01:53:11.385464 18042 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:11.386375 18042 heartbeater.cc:511] Master 127.11.42.126:39453 requested a full tablet report, sending...
I20240502 01:53:11.387696 17335 ts_manager.cc:194] Registered new tserver with Master: fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:53:11.388427 17335 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.69:33207
I20240502 01:53:11.558277 17782 heartbeater.cc:503] Master 127.11.42.126:39453 was elected leader, sending a full tablet report...
W20240502 01:53:11.691155 18045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:11.691421 18045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:11.691533 18045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:11.698901 18045 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:53:11.699002 18045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:11.699155 18045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.70
I20240502 01:53:11.707875 18045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.11.42.70
--webserver_port=0
--tserver_master_addrs=127.11.42.126:39453
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.70
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:11.709108 18045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:11.710716 18045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:11.722059 18050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:11.722622 18053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:11.722940 18051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:11.723431 18045 server_base.cc:1034] running on GCE node
I20240502 01:53:11.723938 18045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:11.724749 18045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:11.725937 18045 hybrid_clock.cc:648] HybridClock initialized: now 1714614791725899 us; error 28 us; skew 500 ppm
I20240502 01:53:11.726272 18045 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:11.733198 18045 webserver.cc:460] Webserver started at http://127.11.42.70:45687/ using document root <none> and password file <none>
I20240502 01:53:11.733904 18045 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:11.734010 18045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:11.734308 18045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:11.736660 18045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
format_stamp: "Formatted at 2024-05-02 01:53:11 on dist-test-slave-xvk5"
I20240502 01:53:11.737387 18045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
format_stamp: "Formatted at 2024-05-02 01:53:11 on dist-test-slave-xvk5"
I20240502 01:53:11.742026 18045 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 01:53:11.745150 18059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:11.746380 18045 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 01:53:11.746534 18045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
format_stamp: "Formatted at 2024-05-02 01:53:11 on dist-test-slave-xvk5"
I20240502 01:53:11.746694 18045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:11.764864 18045 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:11.765559 18045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:11.765770 18045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:11.766546 18045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:11.767838 18045 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:11.767918 18045 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:11.767993 18045 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:11.768044 18045 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:11.796697 18045 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.70:39155
I20240502 01:53:11.796756 18171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.70:39155 every 8 connection(s)
I20240502 01:53:11.797890 18045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 01:53:11.803479 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 18045
I20240502 01:53:11.803622 11433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 01:53:11.806618 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.71:0
--local_ip_for_outbound_sockets=127.11.42.71
--webserver_interface=127.11.42.71
--webserver_port=0
--tserver_master_addrs=127.11.42.126:39453
--builtin_ntp_servers=127.11.42.84:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:53:11.810460 18172 heartbeater.cc:347] Connected to a master server at 127.11.42.126:39453
I20240502 01:53:11.810732 18172 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:11.811512 18172 heartbeater.cc:511] Master 127.11.42.126:39453 requested a full tablet report, sending...
I20240502 01:53:11.812781 17335 ts_manager.cc:194] Registered new tserver with Master: 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:53:11.813701 17335 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.70:35485
I20240502 01:53:11.975762 17912 heartbeater.cc:503] Master 127.11.42.126:39453 was elected leader, sending a full tablet report...
W20240502 01:53:12.118453 18175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:12.118700 18175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:12.118822 18175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:12.125957 18175 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:53:12.126071 18175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:12.126209 18175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.71
I20240502 01:53:12.134716 18175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.11.42.71
--webserver_port=0
--tserver_master_addrs=127.11.42.126:39453
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.71
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:12.135847 18175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:12.137661 18175 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:12.149077 18181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:12.149475 18183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:12.149777 18180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:12.150274 18175 server_base.cc:1034] running on GCE node
I20240502 01:53:12.150780 18175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:12.151618 18175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:12.152786 18175 hybrid_clock.cc:648] HybridClock initialized: now 1714614792152740 us; error 39 us; skew 500 ppm
I20240502 01:53:12.153075 18175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:12.155027 18175 webserver.cc:460] Webserver started at http://127.11.42.71:36651/ using document root <none> and password file <none>
I20240502 01:53:12.155678 18175 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:12.155778 18175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:12.156069 18175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:12.158411 18175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "2f306549407b4bb281ccf2994035e4f1"
format_stamp: "Formatted at 2024-05-02 01:53:12 on dist-test-slave-xvk5"
I20240502 01:53:12.159108 18175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "2f306549407b4bb281ccf2994035e4f1"
format_stamp: "Formatted at 2024-05-02 01:53:12 on dist-test-slave-xvk5"
I20240502 01:53:12.163620 18175 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20240502 01:53:12.166709 18189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:12.167922 18175 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:12.168056 18175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "2f306549407b4bb281ccf2994035e4f1"
format_stamp: "Formatted at 2024-05-02 01:53:12 on dist-test-slave-xvk5"
I20240502 01:53:12.168219 18175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:12.204641 18175 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:12.205614 18175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:12.205915 18175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:12.206892 18175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:12.208343 18175 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:12.208433 18175 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:12.208541 18175 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:12.208612 18175 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:12.237174 18175 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.71:38623
I20240502 01:53:12.237221 18301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.71:38623 every 8 connection(s)
I20240502 01:53:12.238384 18175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 01:53:12.244810 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 18175
I20240502 01:53:12.244987 11433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 01:53:12.248471 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.72:0
--local_ip_for_outbound_sockets=127.11.42.72
--webserver_interface=127.11.42.72
--webserver_port=0
--tserver_master_addrs=127.11.42.126:39453
--builtin_ntp_servers=127.11.42.84:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:53:12.250610 18302 heartbeater.cc:347] Connected to a master server at 127.11.42.126:39453
I20240502 01:53:12.250897 18302 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:12.251821 18302 heartbeater.cc:511] Master 127.11.42.126:39453 requested a full tablet report, sending...
I20240502 01:53:12.253227 17335 ts_manager.cc:194] Registered new tserver with Master: 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:53:12.253983 17335 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.71:33877
I20240502 01:53:12.390568 18042 heartbeater.cc:503] Master 127.11.42.126:39453 was elected leader, sending a full tablet report...
W20240502 01:53:12.557036 18305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:12.557291 18305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:12.557385 18305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:12.564551 18305 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:53:12.564647 18305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:12.564760 18305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.72
I20240502 01:53:12.573156 18305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.11.42.72
--webserver_port=0
--tserver_master_addrs=127.11.42.126:39453
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.72
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:12.574254 18305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:12.575829 18305 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:12.587127 18310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:12.587587 18311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:12.587888 18313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:12.588227 18305 server_base.cc:1034] running on GCE node
I20240502 01:53:12.588809 18305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:12.589768 18305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:12.590996 18305 hybrid_clock.cc:648] HybridClock initialized: now 1714614792590911 us; error 33 us; skew 500 ppm
I20240502 01:53:12.591344 18305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:12.593333 18305 webserver.cc:460] Webserver started at http://127.11.42.72:46559/ using document root <none> and password file <none>
I20240502 01:53:12.593977 18305 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:12.594069 18305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:12.594316 18305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:12.596670 18305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "f19cc6cb1e76460aa6711e49a5775091"
format_stamp: "Formatted at 2024-05-02 01:53:12 on dist-test-slave-xvk5"
I20240502 01:53:12.597410 18305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "f19cc6cb1e76460aa6711e49a5775091"
format_stamp: "Formatted at 2024-05-02 01:53:12 on dist-test-slave-xvk5"
I20240502 01:53:12.602125 18305 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 01:53:12.605139 18319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:12.606456 18305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 01:53:12.606623 18305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "f19cc6cb1e76460aa6711e49a5775091"
format_stamp: "Formatted at 2024-05-02 01:53:12 on dist-test-slave-xvk5"
I20240502 01:53:12.606809 18305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:12.624811 18305 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:12.625600 18305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:12.625871 18305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:12.626746 18305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:12.628087 18305 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:12.628178 18305 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:12.628284 18305 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:12.628350 18305 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:12.657223 18305 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.72:42411
I20240502 01:53:12.657305 18431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.72:42411 every 8 connection(s)
I20240502 01:53:12.658432 18305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 01:53:12.666589 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 18305
I20240502 01:53:12.666754 11433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 01:53:12.670523 18432 heartbeater.cc:347] Connected to a master server at 127.11.42.126:39453
I20240502 01:53:12.670791 18432 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:12.671557 18432 heartbeater.cc:511] Master 127.11.42.126:39453 requested a full tablet report, sending...
I20240502 01:53:12.672816 17335 ts_manager.cc:194] Registered new tserver with Master: f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:53:12.673604 17335 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.72:42609
I20240502 01:53:12.684350 11433 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 01:53:12.699795 11433 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 01:53:12.699918 11433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu pid 17915
I20240502 01:53:12.707552 18037 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:12.787431 11433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu with pid 17915
I20240502 01:53:12.801090 11433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu pid 18045
I20240502 01:53:12.808609 18167 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:12.885591 18172 heartbeater.cc:503] Master 127.11.42.126:39453 was elected leader, sending a full tablet report...
I20240502 01:53:12.887051 11433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu with pid 18045
I20240502 01:53:12.900621 11433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu pid 18175
I20240502 01:53:12.908246 18297 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:12.985981 11433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu with pid 18175
I20240502 01:53:12.999763 11433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu pid 18305
I20240502 01:53:13.007503 18427 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:13.085090 11433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu with pid 18305
I20240502 01:53:16.850435 17335 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:16.886868 17586 tablet_service.cc:1467] Processing CreateTablet for tablet 8f7ccf4d86204a59bcced6ec414fe23b (DEFAULT_TABLE table=rebalance_test_table_0 [id=c58427a7a33942e1a0e24f2c6c4f0ea1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:16.888466 17586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f7ccf4d86204a59bcced6ec414fe23b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:16.889390 17717 tablet_service.cc:1467] Processing CreateTablet for tablet bc2c55fdea5c45caaff73a0937a31779 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c58427a7a33942e1a0e24f2c6c4f0ea1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:16.890195 17716 tablet_service.cc:1467] Processing CreateTablet for tablet 89213ec8d6f5415abde90e807a1d4684 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c58427a7a33942e1a0e24f2c6c4f0ea1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:16.890803 17717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc2c55fdea5c45caaff73a0937a31779. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:16.892408 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89213ec8d6f5415abde90e807a1d4684. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:16.895170 17847 tablet_service.cc:1467] Processing CreateTablet for tablet 8f7ccf4d86204a59bcced6ec414fe23b (DEFAULT_TABLE table=rebalance_test_table_0 [id=c58427a7a33942e1a0e24f2c6c4f0ea1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:16.895886 17587 tablet_service.cc:1467] Processing CreateTablet for tablet bc2c55fdea5c45caaff73a0937a31779 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c58427a7a33942e1a0e24f2c6c4f0ea1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:16.895217 17846 tablet_service.cc:1467] Processing CreateTablet for tablet 89213ec8d6f5415abde90e807a1d4684 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c58427a7a33942e1a0e24f2c6c4f0ea1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:16.896591 17587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc2c55fdea5c45caaff73a0937a31779. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:16.896961 17847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f7ccf4d86204a59bcced6ec414fe23b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:16.897432 17846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89213ec8d6f5415abde90e807a1d4684. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:16.908275 18461 tablet_bootstrap.cc:492] T 8f7ccf4d86204a59bcced6ec414fe23b P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:16.914777 17456 tablet_service.cc:1467] Processing CreateTablet for tablet bc2c55fdea5c45caaff73a0937a31779 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c58427a7a33942e1a0e24f2c6c4f0ea1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:16.915930 17454 tablet_service.cc:1467] Processing CreateTablet for tablet 89213ec8d6f5415abde90e807a1d4684 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c58427a7a33942e1a0e24f2c6c4f0ea1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:16.916718 18462 tablet_bootstrap.cc:492] T 89213ec8d6f5415abde90e807a1d4684 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:16.917318 17454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89213ec8d6f5415abde90e807a1d4684. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:16.916316 17455 tablet_service.cc:1467] Processing CreateTablet for tablet 8f7ccf4d86204a59bcced6ec414fe23b (DEFAULT_TABLE table=rebalance_test_table_0 [id=c58427a7a33942e1a0e24f2c6c4f0ea1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:16.917930 17455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f7ccf4d86204a59bcced6ec414fe23b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:16.919106 17456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc2c55fdea5c45caaff73a0937a31779. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:16.919266 18463 tablet_bootstrap.cc:492] T bc2c55fdea5c45caaff73a0937a31779 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:16.921212 18462 tablet_bootstrap.cc:654] T 89213ec8d6f5415abde90e807a1d4684 P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:16.921607 18461 tablet_bootstrap.cc:654] T 8f7ccf4d86204a59bcced6ec414fe23b P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:16.922559 18463 tablet_bootstrap.cc:654] T bc2c55fdea5c45caaff73a0937a31779 P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:16.923197 18462 log.cc:824] T 89213ec8d6f5415abde90e807a1d4684 P bb31bceba76d466db24b9341c04900c3: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:16.923890 18463 log.cc:824] T bc2c55fdea5c45caaff73a0937a31779 P dfa1152987644bd781b46c7e42a7a685: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:16.929344 18461 log.cc:824] T 8f7ccf4d86204a59bcced6ec414fe23b P 3fceccefba274b1dbbd8216af8b0bc29: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:16.929841 18463 tablet_bootstrap.cc:492] T bc2c55fdea5c45caaff73a0937a31779 P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:16.930166 18463 ts_tablet_manager.cc:1367] T bc2c55fdea5c45caaff73a0937a31779 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.004s
I20240502 01:53:16.932051 18461 tablet_bootstrap.cc:492] T 8f7ccf4d86204a59bcced6ec414fe23b P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:16.932394 18461 ts_tablet_manager.cc:1367] T 8f7ccf4d86204a59bcced6ec414fe23b P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.000s
I20240502 01:53:16.934454 18462 tablet_bootstrap.cc:492] T 89213ec8d6f5415abde90e807a1d4684 P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:16.935088 18462 ts_tablet_manager.cc:1367] T 89213ec8d6f5415abde90e807a1d4684 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.006s
I20240502 01:53:16.938203 18467 tablet_bootstrap.cc:492] T 8f7ccf4d86204a59bcced6ec414fe23b P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:16.941401 18461 raft_consensus.cc:348] T 8f7ccf4d86204a59bcced6ec414fe23b P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:16.941871 18461 raft_consensus.cc:374] T 8f7ccf4d86204a59bcced6ec414fe23b P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:16.941982 18461 raft_consensus.cc:729] T 8f7ccf4d86204a59bcced6ec414fe23b P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:16.942725 18461 consensus_queue.cc:260] T 8f7ccf4d86204a59bcced6ec414fe23b P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:16.943486 18467 tablet_bootstrap.cc:654] T 8f7ccf4d86204a59bcced6ec414fe23b P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:16.944571 18462 raft_consensus.cc:348] T 89213ec8d6f5415abde90e807a1d4684 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:16.944849 18467 log.cc:824] T 8f7ccf4d86204a59bcced6ec414fe23b P 4da690c0fa13407a85c0689ce8de9ade: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:16.945034 18462 raft_consensus.cc:374] T 89213ec8d6f5415abde90e807a1d4684 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:16.945194 18462 raft_consensus.cc:729] T 89213ec8d6f5415abde90e807a1d4684 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:16.945875 18462 consensus_queue.cc:260] T 89213ec8d6f5415abde90e807a1d4684 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:16.945959 18463 raft_consensus.cc:348] T bc2c55fdea5c45caaff73a0937a31779 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:16.946383 18463 raft_consensus.cc:374] T bc2c55fdea5c45caaff73a0937a31779 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:16.946537 18463 raft_consensus.cc:729] T bc2c55fdea5c45caaff73a0937a31779 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:16.947347 18463 consensus_queue.cc:260] T bc2c55fdea5c45caaff73a0937a31779 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:16.951076 18463 ts_tablet_manager.cc:1398] T bc2c55fdea5c45caaff73a0937a31779 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.021s user 0.011s sys 0.001s
I20240502 01:53:16.954336 18467 tablet_bootstrap.cc:492] T 8f7ccf4d86204a59bcced6ec414fe23b P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:16.954696 18467 ts_tablet_manager.cc:1367] T 8f7ccf4d86204a59bcced6ec414fe23b P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.004s
I20240502 01:53:16.957502 18463 tablet_bootstrap.cc:492] T 89213ec8d6f5415abde90e807a1d4684 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:16.959327 18463 tablet_bootstrap.cc:654] T 89213ec8d6f5415abde90e807a1d4684 P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:16.960780 18461 ts_tablet_manager.cc:1398] T 8f7ccf4d86204a59bcced6ec414fe23b P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.028s user 0.018s sys 0.007s
I20240502 01:53:16.961275 18461 tablet_bootstrap.cc:492] T bc2c55fdea5c45caaff73a0937a31779 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:16.963076 18461 tablet_bootstrap.cc:654] T bc2c55fdea5c45caaff73a0937a31779 P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:16.966477 18462 ts_tablet_manager.cc:1398] T 89213ec8d6f5415abde90e807a1d4684 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.031s user 0.012s sys 0.011s
I20240502 01:53:16.967018 18462 tablet_bootstrap.cc:492] T 8f7ccf4d86204a59bcced6ec414fe23b P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:16.968961 18462 tablet_bootstrap.cc:654] T 8f7ccf4d86204a59bcced6ec414fe23b P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:16.971156 18461 tablet_bootstrap.cc:492] T bc2c55fdea5c45caaff73a0937a31779 P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:16.971356 18461 ts_tablet_manager.cc:1367] T bc2c55fdea5c45caaff73a0937a31779 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.004s
I20240502 01:53:16.971398 18463 tablet_bootstrap.cc:492] T 89213ec8d6f5415abde90e807a1d4684 P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:16.971619 18463 ts_tablet_manager.cc:1367] T 89213ec8d6f5415abde90e807a1d4684 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.005s
I20240502 01:53:16.972383 18461 raft_consensus.cc:348] T bc2c55fdea5c45caaff73a0937a31779 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:16.972656 18461 raft_consensus.cc:374] T bc2c55fdea5c45caaff73a0937a31779 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:16.972620 18463 raft_consensus.cc:348] T 89213ec8d6f5415abde90e807a1d4684 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:16.972792 18461 raft_consensus.cc:729] T bc2c55fdea5c45caaff73a0937a31779 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:16.972818 18462 tablet_bootstrap.cc:492] T 8f7ccf4d86204a59bcced6ec414fe23b P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:16.972918 18463 raft_consensus.cc:374] T 89213ec8d6f5415abde90e807a1d4684 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:16.972947 18462 ts_tablet_manager.cc:1367] T 8f7ccf4d86204a59bcced6ec414fe23b P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240502 01:53:16.973091 18463 raft_consensus.cc:729] T 89213ec8d6f5415abde90e807a1d4684 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:16.973120 18461 consensus_queue.cc:260] T bc2c55fdea5c45caaff73a0937a31779 P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:16.973774 18461 ts_tablet_manager.cc:1398] T bc2c55fdea5c45caaff73a0937a31779 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:16.973913 18462 raft_consensus.cc:348] T 8f7ccf4d86204a59bcced6ec414fe23b P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:16.974133 18462 raft_consensus.cc:374] T 8f7ccf4d86204a59bcced6ec414fe23b P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:16.973431 18463 consensus_queue.cc:260] T 89213ec8d6f5415abde90e807a1d4684 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:16.974227 18462 raft_consensus.cc:729] T 8f7ccf4d86204a59bcced6ec414fe23b P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:16.974435 18462 consensus_queue.cc:260] T 8f7ccf4d86204a59bcced6ec414fe23b P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:16.974826 18463 ts_tablet_manager.cc:1398] T 89213ec8d6f5415abde90e807a1d4684 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:53:16.975214 18462 ts_tablet_manager.cc:1398] T 8f7ccf4d86204a59bcced6ec414fe23b P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 01:53:16.981283 18467 raft_consensus.cc:348] T 8f7ccf4d86204a59bcced6ec414fe23b P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:16.981695 18467 raft_consensus.cc:374] T 8f7ccf4d86204a59bcced6ec414fe23b P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:16.981791 18467 raft_consensus.cc:729] T 8f7ccf4d86204a59bcced6ec414fe23b P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:16.982468 18467 consensus_queue.cc:260] T 8f7ccf4d86204a59bcced6ec414fe23b P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:16.983953 18467 ts_tablet_manager.cc:1398] T 8f7ccf4d86204a59bcced6ec414fe23b P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.029s user 0.010s sys 0.000s
I20240502 01:53:16.984373 18467 tablet_bootstrap.cc:492] T bc2c55fdea5c45caaff73a0937a31779 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:16.985760 18467 tablet_bootstrap.cc:654] T bc2c55fdea5c45caaff73a0937a31779 P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:16.987686 18467 tablet_bootstrap.cc:492] T bc2c55fdea5c45caaff73a0937a31779 P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:16.987845 18467 ts_tablet_manager.cc:1367] T bc2c55fdea5c45caaff73a0937a31779 P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:53:16.988548 18467 raft_consensus.cc:348] T bc2c55fdea5c45caaff73a0937a31779 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:16.988690 18467 raft_consensus.cc:374] T bc2c55fdea5c45caaff73a0937a31779 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:16.988752 18467 raft_consensus.cc:729] T bc2c55fdea5c45caaff73a0937a31779 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:16.988910 18467 consensus_queue.cc:260] T bc2c55fdea5c45caaff73a0937a31779 P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:16.989367 18467 ts_tablet_manager.cc:1398] T bc2c55fdea5c45caaff73a0937a31779 P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 01:53:16.989590 18467 tablet_bootstrap.cc:492] T 89213ec8d6f5415abde90e807a1d4684 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:16.990689 18467 tablet_bootstrap.cc:654] T 89213ec8d6f5415abde90e807a1d4684 P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:16.993105 18467 tablet_bootstrap.cc:492] T 89213ec8d6f5415abde90e807a1d4684 P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:16.993280 18467 ts_tablet_manager.cc:1367] T 89213ec8d6f5415abde90e807a1d4684 P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:53:16.993760 18467 raft_consensus.cc:348] T 89213ec8d6f5415abde90e807a1d4684 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:16.993928 18467 raft_consensus.cc:374] T 89213ec8d6f5415abde90e807a1d4684 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:16.993994 18467 raft_consensus.cc:729] T 89213ec8d6f5415abde90e807a1d4684 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:16.994189 18467 consensus_queue.cc:260] T 89213ec8d6f5415abde90e807a1d4684 P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:16.994627 18467 ts_tablet_manager.cc:1398] T 89213ec8d6f5415abde90e807a1d4684 P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:53:17.021071 18470 raft_consensus.cc:482] T 8f7ccf4d86204a59bcced6ec414fe23b P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:17.021319 18470 raft_consensus.cc:504] T 8f7ccf4d86204a59bcced6ec414fe23b P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:17.022989 18470 leader_election.cc:288] T 8f7ccf4d86204a59bcced6ec414fe23b P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:17.030120 17607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7ccf4d86204a59bcced6ec414fe23b" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:53:17.030123 17476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7ccf4d86204a59bcced6ec414fe23b" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:17.030592 17476 raft_consensus.cc:2454] T 8f7ccf4d86204a59bcced6ec414fe23b P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 0.
I20240502 01:53:17.030597 17607 raft_consensus.cc:2454] T 8f7ccf4d86204a59bcced6ec414fe23b P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 0.
I20240502 01:53:17.031158 17802 leader_election.cc:302] T 8f7ccf4d86204a59bcced6ec414fe23b P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:53:17.031545 18470 raft_consensus.cc:2781] T 8f7ccf4d86204a59bcced6ec414fe23b P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:17.031678 18470 raft_consensus.cc:482] T 8f7ccf4d86204a59bcced6ec414fe23b P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:17.031782 18470 raft_consensus.cc:3037] T 8f7ccf4d86204a59bcced6ec414fe23b P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:17.033864 18470 raft_consensus.cc:504] T 8f7ccf4d86204a59bcced6ec414fe23b P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:17.034413 18470 leader_election.cc:288] T 8f7ccf4d86204a59bcced6ec414fe23b P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 election: Requested vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:17.034847 17607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7ccf4d86204a59bcced6ec414fe23b" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:53:17.034844 17476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7ccf4d86204a59bcced6ec414fe23b" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:53:17.035018 17476 raft_consensus.cc:3037] T 8f7ccf4d86204a59bcced6ec414fe23b P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:17.035025 17607 raft_consensus.cc:3037] T 8f7ccf4d86204a59bcced6ec414fe23b P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:17.037319 17607 raft_consensus.cc:2454] T 8f7ccf4d86204a59bcced6ec414fe23b P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 1.
I20240502 01:53:17.037416 17476 raft_consensus.cc:2454] T 8f7ccf4d86204a59bcced6ec414fe23b P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 1.
I20240502 01:53:17.037699 17800 leader_election.cc:302] T 8f7ccf4d86204a59bcced6ec414fe23b P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:53:17.037979 18470 raft_consensus.cc:2781] T 8f7ccf4d86204a59bcced6ec414fe23b P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:17.038388 18470 raft_consensus.cc:686] T 8f7ccf4d86204a59bcced6ec414fe23b P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Becoming Leader. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Running, Role: LEADER
I20240502 01:53:17.038889 18470 consensus_queue.cc:237] T 8f7ccf4d86204a59bcced6ec414fe23b P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:17.039196 18471 raft_consensus.cc:482] T bc2c55fdea5c45caaff73a0937a31779 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:17.039422 18471 raft_consensus.cc:504] T bc2c55fdea5c45caaff73a0937a31779 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:17.041074 18471 leader_election.cc:288] T bc2c55fdea5c45caaff73a0937a31779 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:17.042603 18475 raft_consensus.cc:482] T 89213ec8d6f5415abde90e807a1d4684 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:17.042764 18475 raft_consensus.cc:504] T 89213ec8d6f5415abde90e807a1d4684 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:17.043953 17335 catalog_manager.cc:5484] T 8f7ccf4d86204a59bcced6ec414fe23b P bb31bceba76d466db24b9341c04900c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb31bceba76d466db24b9341c04900c3 (127.11.42.68). New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:17.044508 18475 leader_election.cc:288] T 89213ec8d6f5415abde90e807a1d4684 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
W20240502 01:53:17.045918 17783 tablet.cc:2369] T 89213ec8d6f5415abde90e807a1d4684 P dfa1152987644bd781b46c7e42a7a685: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:53:17.049770 17476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc2c55fdea5c45caaff73a0937a31779" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:17.050011 17476 raft_consensus.cc:2454] T bc2c55fdea5c45caaff73a0937a31779 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:53:17.050485 17672 leader_election.cc:302] T bc2c55fdea5c45caaff73a0937a31779 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:17.050829 18471 raft_consensus.cc:2781] T bc2c55fdea5c45caaff73a0937a31779 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:17.050987 18471 raft_consensus.cc:482] T bc2c55fdea5c45caaff73a0937a31779 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:17.051090 18471 raft_consensus.cc:3037] T bc2c55fdea5c45caaff73a0937a31779 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:17.053665 17607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc2c55fdea5c45caaff73a0937a31779" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:53:17.053920 17607 raft_consensus.cc:2454] T bc2c55fdea5c45caaff73a0937a31779 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:53:17.053915 18471 raft_consensus.cc:504] T bc2c55fdea5c45caaff73a0937a31779 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:17.054584 18471 leader_election.cc:288] T bc2c55fdea5c45caaff73a0937a31779 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Requested vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:17.055171 17607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc2c55fdea5c45caaff73a0937a31779" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:53:17.055176 17476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc2c55fdea5c45caaff73a0937a31779" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:53:17.055377 17607 raft_consensus.cc:3037] T bc2c55fdea5c45caaff73a0937a31779 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:17.055400 17476 raft_consensus.cc:3037] T bc2c55fdea5c45caaff73a0937a31779 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:17.057987 17607 raft_consensus.cc:2454] T bc2c55fdea5c45caaff73a0937a31779 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:53:17.058198 17476 raft_consensus.cc:2454] T bc2c55fdea5c45caaff73a0937a31779 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:53:17.058495 17670 leader_election.cc:302] T bc2c55fdea5c45caaff73a0937a31779 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:17.058987 18471 raft_consensus.cc:2781] T bc2c55fdea5c45caaff73a0937a31779 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:17.059993 18471 raft_consensus.cc:686] T bc2c55fdea5c45caaff73a0937a31779 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Becoming Leader. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Running, Role: LEADER
I20240502 01:53:17.060519 18471 consensus_queue.cc:237] T bc2c55fdea5c45caaff73a0937a31779 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:17.062347 17867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89213ec8d6f5415abde90e807a1d4684" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:53:17.062775 17867 raft_consensus.cc:2454] T 89213ec8d6f5415abde90e807a1d4684 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 0.
I20240502 01:53:17.063354 17412 leader_election.cc:302] T 89213ec8d6f5415abde90e807a1d4684 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:53:17.063773 18475 raft_consensus.cc:2781] T 89213ec8d6f5415abde90e807a1d4684 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:17.063902 18475 raft_consensus.cc:482] T 89213ec8d6f5415abde90e807a1d4684 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:17.064003 18475 raft_consensus.cc:3037] T 89213ec8d6f5415abde90e807a1d4684 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:17.065972 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89213ec8d6f5415abde90e807a1d4684" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:53:17.066372 17737 raft_consensus.cc:2454] T 89213ec8d6f5415abde90e807a1d4684 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 0.
I20240502 01:53:17.066557 18475 raft_consensus.cc:504] T 89213ec8d6f5415abde90e807a1d4684 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:17.067203 18475 leader_election.cc:288] T 89213ec8d6f5415abde90e807a1d4684 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Requested vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:53:17.067584 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89213ec8d6f5415abde90e807a1d4684" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:53:17.067771 17737 raft_consensus.cc:3037] T 89213ec8d6f5415abde90e807a1d4684 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:17.068066 17867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89213ec8d6f5415abde90e807a1d4684" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:53:17.067945 17335 catalog_manager.cc:5484] T bc2c55fdea5c45caaff73a0937a31779 P dfa1152987644bd781b46c7e42a7a685 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfa1152987644bd781b46c7e42a7a685 (127.11.42.67). New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:17.068255 17867 raft_consensus.cc:3037] T 89213ec8d6f5415abde90e807a1d4684 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:17.070547 17737 raft_consensus.cc:2454] T 89213ec8d6f5415abde90e807a1d4684 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
I20240502 01:53:17.071007 17411 leader_election.cc:302] T 89213ec8d6f5415abde90e807a1d4684 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:17.071146 17867 raft_consensus.cc:2454] T 89213ec8d6f5415abde90e807a1d4684 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
I20240502 01:53:17.071333 18475 raft_consensus.cc:2781] T 89213ec8d6f5415abde90e807a1d4684 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:17.071767 18475 raft_consensus.cc:686] T 89213ec8d6f5415abde90e807a1d4684 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Becoming Leader. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Running, Role: LEADER
I20240502 01:53:17.072472 18475 consensus_queue.cc:237] T 89213ec8d6f5415abde90e807a1d4684 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:17.076893 17335 catalog_manager.cc:5484] T 89213ec8d6f5415abde90e807a1d4684 P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: term changed from 0 to 1, leader changed from <none> to 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65). New cstate: current_term: 1 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } }
W20240502 01:53:17.124089 17653 tablet.cc:2369] T 8f7ccf4d86204a59bcced6ec414fe23b P 3fceccefba274b1dbbd8216af8b0bc29: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:53:17.203877 17522 tablet.cc:2369] T 89213ec8d6f5415abde90e807a1d4684 P 4da690c0fa13407a85c0689ce8de9ade: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:53:17.212561 17913 tablet.cc:2369] T 89213ec8d6f5415abde90e807a1d4684 P bb31bceba76d466db24b9341c04900c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:53:17.435897 18490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:17.436218 18490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 01:53:17.439354 18471 consensus_queue.cc:1035] T bc2c55fdea5c45caaff73a0937a31779 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:17.444900 18490 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:53:17.446950 18471 consensus_queue.cc:1035] T bc2c55fdea5c45caaff73a0937a31779 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:17.479002 18475 consensus_queue.cc:1035] T 89213ec8d6f5415abde90e807a1d4684 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240502 01:53:17.489434 18488 consensus_queue.cc:1035] T 89213ec8d6f5415abde90e807a1d4684 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:17.526748 18470 consensus_queue.cc:1035] T 8f7ccf4d86204a59bcced6ec414fe23b P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:17.531298 18524 consensus_queue.cc:1035] T 8f7ccf4d86204a59bcced6ec414fe23b P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 243.235 ms
time per row: 0.121618 ms
I20240502 01:53:17.844476 17335 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:17.854424 17846 tablet_service.cc:1467] Processing CreateTablet for tablet 8e7f88b6c509477fb0900024e7de8f00 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e431affbb02a4744b67a522521a3ee04]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:17.854534 17586 tablet_service.cc:1467] Processing CreateTablet for tablet 8e7f88b6c509477fb0900024e7de8f00 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e431affbb02a4744b67a522521a3ee04]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:17.854907 17586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e7f88b6c509477fb0900024e7de8f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:17.854924 17846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e7f88b6c509477fb0900024e7de8f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:17.856046 17717 tablet_service.cc:1467] Processing CreateTablet for tablet 8e7f88b6c509477fb0900024e7de8f00 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e431affbb02a4744b67a522521a3ee04]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:17.856209 17716 tablet_service.cc:1467] Processing CreateTablet for tablet 5124abbb09db4be0955c6bc831af08fa (DEFAULT_TABLE table=rebalance_test_table_1 [id=e431affbb02a4744b67a522521a3ee04]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:17.856447 17717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e7f88b6c509477fb0900024e7de8f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:17.857003 17715 tablet_service.cc:1467] Processing CreateTablet for tablet 10ec78a08ab04678897a6892988e515f (DEFAULT_TABLE table=rebalance_test_table_1 [id=e431affbb02a4744b67a522521a3ee04]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:17.857496 17715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10ec78a08ab04678897a6892988e515f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:17.857825 17454 tablet_service.cc:1467] Processing CreateTablet for tablet 10ec78a08ab04678897a6892988e515f (DEFAULT_TABLE table=rebalance_test_table_1 [id=e431affbb02a4744b67a522521a3ee04]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:17.858238 17454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10ec78a08ab04678897a6892988e515f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:17.861408 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5124abbb09db4be0955c6bc831af08fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:17.861897 17587 tablet_service.cc:1467] Processing CreateTablet for tablet 5124abbb09db4be0955c6bc831af08fa (DEFAULT_TABLE table=rebalance_test_table_1 [id=e431affbb02a4744b67a522521a3ee04]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:17.862267 17587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5124abbb09db4be0955c6bc831af08fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:17.869061 18555 tablet_bootstrap.cc:492] T 8e7f88b6c509477fb0900024e7de8f00 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:17.869516 17585 tablet_service.cc:1467] Processing CreateTablet for tablet 10ec78a08ab04678897a6892988e515f (DEFAULT_TABLE table=rebalance_test_table_1 [id=e431affbb02a4744b67a522521a3ee04]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:17.869957 17585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10ec78a08ab04678897a6892988e515f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:17.870780 18555 tablet_bootstrap.cc:654] T 8e7f88b6c509477fb0900024e7de8f00 P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:17.874866 18555 tablet_bootstrap.cc:492] T 8e7f88b6c509477fb0900024e7de8f00 P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:17.875007 18555 ts_tablet_manager.cc:1367] T 8e7f88b6c509477fb0900024e7de8f00 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20240502 01:53:17.875804 18555 raft_consensus.cc:348] T 8e7f88b6c509477fb0900024e7de8f00 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:17.876009 18555 raft_consensus.cc:374] T 8e7f88b6c509477fb0900024e7de8f00 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:17.876091 18555 raft_consensus.cc:729] T 8e7f88b6c509477fb0900024e7de8f00 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:17.876368 18555 consensus_queue.cc:260] T 8e7f88b6c509477fb0900024e7de8f00 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:17.877102 18555 ts_tablet_manager.cc:1398] T 8e7f88b6c509477fb0900024e7de8f00 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:53:17.881409 17847 tablet_service.cc:1467] Processing CreateTablet for tablet 5124abbb09db4be0955c6bc831af08fa (DEFAULT_TABLE table=rebalance_test_table_1 [id=e431affbb02a4744b67a522521a3ee04]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:17.881705 17847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5124abbb09db4be0955c6bc831af08fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:17.886154 18558 tablet_bootstrap.cc:492] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:17.888705 18558 tablet_bootstrap.cc:654] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:17.889643 18555 tablet_bootstrap.cc:492] T 5124abbb09db4be0955c6bc831af08fa P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:17.891098 18555 tablet_bootstrap.cc:654] T 5124abbb09db4be0955c6bc831af08fa P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:17.895695 18555 tablet_bootstrap.cc:492] T 5124abbb09db4be0955c6bc831af08fa P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:17.895869 18555 ts_tablet_manager.cc:1367] T 5124abbb09db4be0955c6bc831af08fa P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:53:17.896613 18555 raft_consensus.cc:348] T 5124abbb09db4be0955c6bc831af08fa P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:17.896828 18555 raft_consensus.cc:374] T 5124abbb09db4be0955c6bc831af08fa P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:17.896905 18555 raft_consensus.cc:729] T 5124abbb09db4be0955c6bc831af08fa P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:17.897158 18555 consensus_queue.cc:260] T 5124abbb09db4be0955c6bc831af08fa P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:17.897491 18558 tablet_bootstrap.cc:492] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:17.897630 18558 ts_tablet_manager.cc:1367] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20240502 01:53:17.897686 18555 ts_tablet_manager.cc:1398] T 5124abbb09db4be0955c6bc831af08fa P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:53:17.898578 18558 raft_consensus.cc:348] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:17.898788 18558 raft_consensus.cc:374] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:17.898864 18558 raft_consensus.cc:729] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:17.899206 18558 consensus_queue.cc:260] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:17.899976 18558 ts_tablet_manager.cc:1398] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:17.900271 18558 tablet_bootstrap.cc:492] T 5124abbb09db4be0955c6bc831af08fa P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:17.901928 18558 tablet_bootstrap.cc:654] T 5124abbb09db4be0955c6bc831af08fa P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:17.904193 18558 tablet_bootstrap.cc:492] T 5124abbb09db4be0955c6bc831af08fa P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:17.904369 18558 ts_tablet_manager.cc:1367] T 5124abbb09db4be0955c6bc831af08fa P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:53:17.905095 18558 raft_consensus.cc:348] T 5124abbb09db4be0955c6bc831af08fa P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:17.905344 18558 raft_consensus.cc:374] T 5124abbb09db4be0955c6bc831af08fa P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:17.905467 18558 raft_consensus.cc:729] T 5124abbb09db4be0955c6bc831af08fa P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:17.905712 18558 consensus_queue.cc:260] T 5124abbb09db4be0955c6bc831af08fa P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:17.906260 18558 ts_tablet_manager.cc:1398] T 5124abbb09db4be0955c6bc831af08fa P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:17.906628 18558 tablet_bootstrap.cc:492] T 8e7f88b6c509477fb0900024e7de8f00 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:17.908028 18558 tablet_bootstrap.cc:654] T 8e7f88b6c509477fb0900024e7de8f00 P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:17.909417 18562 tablet_bootstrap.cc:492] T 10ec78a08ab04678897a6892988e515f P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:17.910023 18556 tablet_bootstrap.cc:492] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:17.910955 18562 tablet_bootstrap.cc:654] T 10ec78a08ab04678897a6892988e515f P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:17.911556 18556 tablet_bootstrap.cc:654] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:17.912608 18558 tablet_bootstrap.cc:492] T 8e7f88b6c509477fb0900024e7de8f00 P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:17.912734 18558 ts_tablet_manager.cc:1367] T 8e7f88b6c509477fb0900024e7de8f00 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:53:17.913309 18558 raft_consensus.cc:348] T 8e7f88b6c509477fb0900024e7de8f00 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:17.913452 18558 raft_consensus.cc:374] T 8e7f88b6c509477fb0900024e7de8f00 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:17.913537 18558 raft_consensus.cc:729] T 8e7f88b6c509477fb0900024e7de8f00 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:17.913723 18556 tablet_bootstrap.cc:492] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:17.913713 18558 consensus_queue.cc:260] T 8e7f88b6c509477fb0900024e7de8f00 P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:17.913839 18556 ts_tablet_manager.cc:1367] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 01:53:17.914132 18558 ts_tablet_manager.cc:1398] T 8e7f88b6c509477fb0900024e7de8f00 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:53:17.914582 18556 raft_consensus.cc:348] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:17.914779 18556 raft_consensus.cc:374] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:17.914876 18556 raft_consensus.cc:729] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:17.915131 18562 tablet_bootstrap.cc:492] T 10ec78a08ab04678897a6892988e515f P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:17.915095 18556 consensus_queue.cc:260] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:17.915279 18562 ts_tablet_manager.cc:1367] T 10ec78a08ab04678897a6892988e515f P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240502 01:53:17.915588 18556 ts_tablet_manager.cc:1398] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:53:17.915848 18556 tablet_bootstrap.cc:492] T 10ec78a08ab04678897a6892988e515f P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:17.916036 18562 raft_consensus.cc:348] T 10ec78a08ab04678897a6892988e515f P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:17.916247 18562 raft_consensus.cc:374] T 10ec78a08ab04678897a6892988e515f P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:17.916359 18562 raft_consensus.cc:729] T 10ec78a08ab04678897a6892988e515f P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:17.916625 18562 consensus_queue.cc:260] T 10ec78a08ab04678897a6892988e515f P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:17.917052 18556 tablet_bootstrap.cc:654] T 10ec78a08ab04678897a6892988e515f P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:17.917418 18562 ts_tablet_manager.cc:1398] T 10ec78a08ab04678897a6892988e515f P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:17.919121 18556 tablet_bootstrap.cc:492] T 10ec78a08ab04678897a6892988e515f P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:17.919281 18556 ts_tablet_manager.cc:1367] T 10ec78a08ab04678897a6892988e515f P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:53:17.920020 18556 raft_consensus.cc:348] T 10ec78a08ab04678897a6892988e515f P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:17.920226 18556 raft_consensus.cc:374] T 10ec78a08ab04678897a6892988e515f P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:17.920315 18556 raft_consensus.cc:729] T 10ec78a08ab04678897a6892988e515f P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:17.920567 18556 consensus_queue.cc:260] T 10ec78a08ab04678897a6892988e515f P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:17.921126 18556 ts_tablet_manager.cc:1398] T 10ec78a08ab04678897a6892988e515f P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:53:17.921454 18556 tablet_bootstrap.cc:492] T 5124abbb09db4be0955c6bc831af08fa P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:17.922518 18556 tablet_bootstrap.cc:654] T 5124abbb09db4be0955c6bc831af08fa P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:17.923962 18556 tablet_bootstrap.cc:492] T 5124abbb09db4be0955c6bc831af08fa P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:17.924089 18556 ts_tablet_manager.cc:1367] T 5124abbb09db4be0955c6bc831af08fa P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:17.924621 18556 raft_consensus.cc:348] T 5124abbb09db4be0955c6bc831af08fa P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:17.924764 18556 raft_consensus.cc:374] T 5124abbb09db4be0955c6bc831af08fa P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:17.924851 18556 raft_consensus.cc:729] T 5124abbb09db4be0955c6bc831af08fa P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:17.925076 18556 consensus_queue.cc:260] T 5124abbb09db4be0955c6bc831af08fa P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:17.925484 18556 ts_tablet_manager.cc:1398] T 5124abbb09db4be0955c6bc831af08fa P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:53:17.950126 18468 raft_consensus.cc:482] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:17.950301 18468 raft_consensus.cc:504] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:17.951864 18468 leader_election.cc:288] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:53:17.959162 17476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ec78a08ab04678897a6892988e515f" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:17.959409 17476 raft_consensus.cc:2454] T 10ec78a08ab04678897a6892988e515f P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:53:17.959863 17542 leader_election.cc:302] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, 4da690c0fa13407a85c0689ce8de9ade; no voters:
I20240502 01:53:17.960237 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ec78a08ab04678897a6892988e515f" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:53:17.960291 18468 raft_consensus.cc:2781] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:17.960412 18468 raft_consensus.cc:482] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:17.960489 18468 raft_consensus.cc:3037] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:17.960470 17737 raft_consensus.cc:2454] T 10ec78a08ab04678897a6892988e515f P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:53:17.963562 18468 raft_consensus.cc:504] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:17.964514 18468 leader_election.cc:288] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Requested vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:53:17.964685 17476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ec78a08ab04678897a6892988e515f" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:53:17.964725 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ec78a08ab04678897a6892988e515f" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:53:17.964907 17476 raft_consensus.cc:3037] T 10ec78a08ab04678897a6892988e515f P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:17.964936 17737 raft_consensus.cc:3037] T 10ec78a08ab04678897a6892988e515f P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:17.967630 17737 raft_consensus.cc:2454] T 10ec78a08ab04678897a6892988e515f P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:53:17.967643 17476 raft_consensus.cc:2454] T 10ec78a08ab04678897a6892988e515f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:53:17.967999 17542 leader_election.cc:302] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, 4da690c0fa13407a85c0689ce8de9ade; no voters:
I20240502 01:53:17.968335 18468 raft_consensus.cc:2781] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:17.968735 18468 raft_consensus.cc:686] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Becoming Leader. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Running, Role: LEADER
I20240502 01:53:17.969228 18468 consensus_queue.cc:237] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:17.972479 17335 catalog_manager.cc:5484] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66). New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:18.042431 18486 raft_consensus.cc:482] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:18.042596 18486 raft_consensus.cc:504] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:18.043385 18486 leader_election.cc:288] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:18.043968 17607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e7f88b6c509477fb0900024e7de8f00" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:53:18.044234 17607 raft_consensus.cc:2454] T 8e7f88b6c509477fb0900024e7de8f00 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:53:18.044723 17670 leader_election.cc:302] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:18.045094 18486 raft_consensus.cc:2781] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:18.045231 18486 raft_consensus.cc:482] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:18.045325 18486 raft_consensus.cc:3037] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:18.047371 18486 raft_consensus.cc:504] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:18.048044 18486 leader_election.cc:288] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Requested vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:18.048813 17607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e7f88b6c509477fb0900024e7de8f00" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:53:18.048981 17607 raft_consensus.cc:3037] T 8e7f88b6c509477fb0900024e7de8f00 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:18.050928 17607 raft_consensus.cc:2454] T 8e7f88b6c509477fb0900024e7de8f00 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:53:18.051362 17670 leader_election.cc:302] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:18.051620 18486 raft_consensus.cc:2781] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:18.051638 17867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e7f88b6c509477fb0900024e7de8f00" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:53:18.051781 18486 raft_consensus.cc:686] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Becoming Leader. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Running, Role: LEADER
I20240502 01:53:18.051748 17866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e7f88b6c509477fb0900024e7de8f00" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:53:18.051913 17867 raft_consensus.cc:2454] T 8e7f88b6c509477fb0900024e7de8f00 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:53:18.052060 18486 consensus_queue.cc:237] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:18.055016 17335 catalog_manager.cc:5484] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfa1152987644bd781b46c7e42a7a685 (127.11.42.67). New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } }
I20240502 01:53:18.072898 18470 raft_consensus.cc:482] T 5124abbb09db4be0955c6bc831af08fa P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:18.073065 18470 raft_consensus.cc:504] T 5124abbb09db4be0955c6bc831af08fa P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:18.073678 18470 leader_election.cc:288] T 5124abbb09db4be0955c6bc831af08fa P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:53:18.074100 17607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5124abbb09db4be0955c6bc831af08fa" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:53:18.074345 17607 raft_consensus.cc:2454] T 5124abbb09db4be0955c6bc831af08fa P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 0.
I20240502 01:53:18.074743 17800 leader_election.cc:302] T 5124abbb09db4be0955c6bc831af08fa P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:53:18.075012 18470 raft_consensus.cc:2781] T 5124abbb09db4be0955c6bc831af08fa P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:18.075152 18470 raft_consensus.cc:482] T 5124abbb09db4be0955c6bc831af08fa P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:18.075295 18470 raft_consensus.cc:3037] T 5124abbb09db4be0955c6bc831af08fa P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:18.077852 18470 raft_consensus.cc:504] T 5124abbb09db4be0955c6bc831af08fa P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:18.078408 18470 leader_election.cc:288] T 5124abbb09db4be0955c6bc831af08fa P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 election: Requested vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:53:18.078816 17607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5124abbb09db4be0955c6bc831af08fa" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:53:18.078986 17607 raft_consensus.cc:3037] T 5124abbb09db4be0955c6bc831af08fa P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:18.080261 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5124abbb09db4be0955c6bc831af08fa" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:53:18.080405 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5124abbb09db4be0955c6bc831af08fa" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:53:18.080531 17737 raft_consensus.cc:2454] T 5124abbb09db4be0955c6bc831af08fa P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 0.
I20240502 01:53:18.081260 17607 raft_consensus.cc:2454] T 5124abbb09db4be0955c6bc831af08fa P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 1.
I20240502 01:53:18.081574 17800 leader_election.cc:302] T 5124abbb09db4be0955c6bc831af08fa P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, bb31bceba76d466db24b9341c04900c3; no voters: dfa1152987644bd781b46c7e42a7a685
I20240502 01:53:18.081869 18470 raft_consensus.cc:2781] T 5124abbb09db4be0955c6bc831af08fa P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:18.081993 18470 raft_consensus.cc:686] T 5124abbb09db4be0955c6bc831af08fa P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Becoming Leader. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Running, Role: LEADER
I20240502 01:53:18.082221 18470 consensus_queue.cc:237] T 5124abbb09db4be0955c6bc831af08fa P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:18.084595 17335 catalog_manager.cc:5484] T 5124abbb09db4be0955c6bc831af08fa P bb31bceba76d466db24b9341c04900c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb31bceba76d466db24b9341c04900c3 (127.11.42.68). New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } }
W20240502 01:53:18.433511 18576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:18.433775 18576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:18.440939 18576 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:53:18.443048 18468 consensus_queue.cc:1035] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:18.452728 18468 consensus_queue.cc:1035] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:18.452888 17866 raft_consensus.cc:3037] T 8e7f88b6c509477fb0900024e7de8f00 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:18.456349 18486 consensus_queue.cc:1035] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:18.474879 18486 consensus_queue.cc:1035] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
I20240502 01:53:18.530361 18470 consensus_queue.cc:1035] T 5124abbb09db4be0955c6bc831af08fa P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:18.534590 17736 raft_consensus.cc:3037] T 5124abbb09db4be0955c6bc831af08fa P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:18.537981 18524 consensus_queue.cc:1035] T 5124abbb09db4be0955c6bc831af08fa P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 238.219 ms
time per row: 0.119109 ms
I20240502 01:53:18.841418 17335 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:18.850770 17586 tablet_service.cc:1467] Processing CreateTablet for tablet 5ad48f64a0f54050a5b6112ebb551228 (DEFAULT_TABLE table=rebalance_test_table_2 [id=5af2852a0ad343e9aa110a94b3402a8a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:18.850977 17847 tablet_service.cc:1467] Processing CreateTablet for tablet 5ad48f64a0f54050a5b6112ebb551228 (DEFAULT_TABLE table=rebalance_test_table_2 [id=5af2852a0ad343e9aa110a94b3402a8a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:18.851258 17586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ad48f64a0f54050a5b6112ebb551228. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:18.851157 17717 tablet_service.cc:1467] Processing CreateTablet for tablet 4cbb5336819548ebb1239ee9016910ec (DEFAULT_TABLE table=rebalance_test_table_2 [id=5af2852a0ad343e9aa110a94b3402a8a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:18.851446 17847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ad48f64a0f54050a5b6112ebb551228. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:18.851574 17717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cbb5336819548ebb1239ee9016910ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:18.851871 17454 tablet_service.cc:1467] Processing CreateTablet for tablet 5ad48f64a0f54050a5b6112ebb551228 (DEFAULT_TABLE table=rebalance_test_table_2 [id=5af2852a0ad343e9aa110a94b3402a8a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:18.852274 17454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ad48f64a0f54050a5b6112ebb551228. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:18.854013 17587 tablet_service.cc:1467] Processing CreateTablet for tablet 4a4cc6012ea54b5791d0df2ffe36f7eb (DEFAULT_TABLE table=rebalance_test_table_2 [id=5af2852a0ad343e9aa110a94b3402a8a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:18.854348 17587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a4cc6012ea54b5791d0df2ffe36f7eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:18.858814 17455 tablet_service.cc:1467] Processing CreateTablet for tablet 4cbb5336819548ebb1239ee9016910ec (DEFAULT_TABLE table=rebalance_test_table_2 [id=5af2852a0ad343e9aa110a94b3402a8a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:18.859230 17455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cbb5336819548ebb1239ee9016910ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:18.861404 17456 tablet_service.cc:1467] Processing CreateTablet for tablet 4a4cc6012ea54b5791d0df2ffe36f7eb (DEFAULT_TABLE table=rebalance_test_table_2 [id=5af2852a0ad343e9aa110a94b3402a8a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:18.861763 17456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a4cc6012ea54b5791d0df2ffe36f7eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:18.863276 18645 tablet_bootstrap.cc:492] T 5ad48f64a0f54050a5b6112ebb551228 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:18.865394 17846 tablet_service.cc:1467] Processing CreateTablet for tablet 4cbb5336819548ebb1239ee9016910ec (DEFAULT_TABLE table=rebalance_test_table_2 [id=5af2852a0ad343e9aa110a94b3402a8a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:18.864985 18645 tablet_bootstrap.cc:654] T 5ad48f64a0f54050a5b6112ebb551228 P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:18.865787 17846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cbb5336819548ebb1239ee9016910ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:18.866683 18644 tablet_bootstrap.cc:492] T 4cbb5336819548ebb1239ee9016910ec P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:18.868379 18644 tablet_bootstrap.cc:654] T 4cbb5336819548ebb1239ee9016910ec P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:18.869503 17845 tablet_service.cc:1467] Processing CreateTablet for tablet 4a4cc6012ea54b5791d0df2ffe36f7eb (DEFAULT_TABLE table=rebalance_test_table_2 [id=5af2852a0ad343e9aa110a94b3402a8a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:18.869927 17845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a4cc6012ea54b5791d0df2ffe36f7eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:18.870679 18644 tablet_bootstrap.cc:492] T 4cbb5336819548ebb1239ee9016910ec P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:18.870821 18644 ts_tablet_manager.cc:1367] T 4cbb5336819548ebb1239ee9016910ec P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:53:18.871611 18644 raft_consensus.cc:348] T 4cbb5336819548ebb1239ee9016910ec P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:18.871841 18644 raft_consensus.cc:374] T 4cbb5336819548ebb1239ee9016910ec P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:18.871919 18644 raft_consensus.cc:729] T 4cbb5336819548ebb1239ee9016910ec P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:18.872197 18644 consensus_queue.cc:260] T 4cbb5336819548ebb1239ee9016910ec P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:18.873581 18644 ts_tablet_manager.cc:1398] T 4cbb5336819548ebb1239ee9016910ec P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:18.874722 18647 tablet_bootstrap.cc:492] T 4a4cc6012ea54b5791d0df2ffe36f7eb P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:18.876492 18647 tablet_bootstrap.cc:654] T 4a4cc6012ea54b5791d0df2ffe36f7eb P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:18.876935 18643 tablet_bootstrap.cc:492] T 5ad48f64a0f54050a5b6112ebb551228 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:18.879026 18643 tablet_bootstrap.cc:654] T 5ad48f64a0f54050a5b6112ebb551228 P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:18.881340 18647 tablet_bootstrap.cc:492] T 4a4cc6012ea54b5791d0df2ffe36f7eb P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:18.881547 18647 ts_tablet_manager.cc:1367] T 4a4cc6012ea54b5791d0df2ffe36f7eb P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240502 01:53:18.882352 18647 raft_consensus.cc:348] T 4a4cc6012ea54b5791d0df2ffe36f7eb P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:18.882627 18647 raft_consensus.cc:374] T 4a4cc6012ea54b5791d0df2ffe36f7eb P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:18.882711 18643 tablet_bootstrap.cc:492] T 5ad48f64a0f54050a5b6112ebb551228 P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:18.882812 18647 raft_consensus.cc:729] T 4a4cc6012ea54b5791d0df2ffe36f7eb P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:18.883098 18647 consensus_queue.cc:260] T 4a4cc6012ea54b5791d0df2ffe36f7eb P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:18.883777 18647 ts_tablet_manager.cc:1398] T 4a4cc6012ea54b5791d0df2ffe36f7eb P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:18.883864 18643 ts_tablet_manager.cc:1367] T 5ad48f64a0f54050a5b6112ebb551228 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20240502 01:53:18.884142 18647 tablet_bootstrap.cc:492] T 5ad48f64a0f54050a5b6112ebb551228 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:18.885704 18647 tablet_bootstrap.cc:654] T 5ad48f64a0f54050a5b6112ebb551228 P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:18.886984 18643 raft_consensus.cc:348] T 5ad48f64a0f54050a5b6112ebb551228 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:18.887197 18643 raft_consensus.cc:374] T 5ad48f64a0f54050a5b6112ebb551228 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:18.887324 18643 raft_consensus.cc:729] T 5ad48f64a0f54050a5b6112ebb551228 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:18.887550 18643 consensus_queue.cc:260] T 5ad48f64a0f54050a5b6112ebb551228 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:18.887843 18645 tablet_bootstrap.cc:492] T 5ad48f64a0f54050a5b6112ebb551228 P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:18.887986 18645 ts_tablet_manager.cc:1367] T 5ad48f64a0f54050a5b6112ebb551228 P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.010s
I20240502 01:53:18.888260 18643 ts_tablet_manager.cc:1398] T 5ad48f64a0f54050a5b6112ebb551228 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 01:53:18.888533 18643 tablet_bootstrap.cc:492] T 4a4cc6012ea54b5791d0df2ffe36f7eb P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:18.888753 18645 raft_consensus.cc:348] T 5ad48f64a0f54050a5b6112ebb551228 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:18.888954 18645 raft_consensus.cc:374] T 5ad48f64a0f54050a5b6112ebb551228 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:18.889070 18645 raft_consensus.cc:729] T 5ad48f64a0f54050a5b6112ebb551228 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:18.889304 18645 consensus_queue.cc:260] T 5ad48f64a0f54050a5b6112ebb551228 P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:18.889637 18647 tablet_bootstrap.cc:492] T 5ad48f64a0f54050a5b6112ebb551228 P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:18.889914 18645 ts_tablet_manager.cc:1398] T 5ad48f64a0f54050a5b6112ebb551228 P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:18.890164 18647 ts_tablet_manager.cc:1367] T 5ad48f64a0f54050a5b6112ebb551228 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 01:53:18.890185 18645 tablet_bootstrap.cc:492] T 4a4cc6012ea54b5791d0df2ffe36f7eb P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:18.890952 18647 raft_consensus.cc:348] T 5ad48f64a0f54050a5b6112ebb551228 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:18.891206 18647 raft_consensus.cc:374] T 5ad48f64a0f54050a5b6112ebb551228 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:18.891296 18647 raft_consensus.cc:729] T 5ad48f64a0f54050a5b6112ebb551228 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:18.891517 18647 consensus_queue.cc:260] T 5ad48f64a0f54050a5b6112ebb551228 P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:18.891777 18645 tablet_bootstrap.cc:654] T 4a4cc6012ea54b5791d0df2ffe36f7eb P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:18.892068 18647 ts_tablet_manager.cc:1398] T 5ad48f64a0f54050a5b6112ebb551228 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:18.890024 18643 tablet_bootstrap.cc:654] T 4a4cc6012ea54b5791d0df2ffe36f7eb P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:18.893661 18645 tablet_bootstrap.cc:492] T 4a4cc6012ea54b5791d0df2ffe36f7eb P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:18.893827 18645 ts_tablet_manager.cc:1367] T 4a4cc6012ea54b5791d0df2ffe36f7eb P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240502 01:53:18.894543 18645 raft_consensus.cc:348] T 4a4cc6012ea54b5791d0df2ffe36f7eb P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:18.894764 18645 raft_consensus.cc:374] T 4a4cc6012ea54b5791d0df2ffe36f7eb P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:18.894829 18645 raft_consensus.cc:729] T 4a4cc6012ea54b5791d0df2ffe36f7eb P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:18.895081 18645 consensus_queue.cc:260] T 4a4cc6012ea54b5791d0df2ffe36f7eb P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:18.895555 18645 ts_tablet_manager.cc:1398] T 4a4cc6012ea54b5791d0df2ffe36f7eb P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:53:18.895831 18645 tablet_bootstrap.cc:492] T 4cbb5336819548ebb1239ee9016910ec P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:18.897189 18643 tablet_bootstrap.cc:492] T 4a4cc6012ea54b5791d0df2ffe36f7eb P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:18.897313 18645 tablet_bootstrap.cc:654] T 4cbb5336819548ebb1239ee9016910ec P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:18.897362 18643 ts_tablet_manager.cc:1367] T 4a4cc6012ea54b5791d0df2ffe36f7eb P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240502 01:53:18.898133 18643 raft_consensus.cc:348] T 4a4cc6012ea54b5791d0df2ffe36f7eb P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:18.898344 18643 raft_consensus.cc:374] T 4a4cc6012ea54b5791d0df2ffe36f7eb P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:18.898612 18643 raft_consensus.cc:729] T 4a4cc6012ea54b5791d0df2ffe36f7eb P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:18.898902 18645 tablet_bootstrap.cc:492] T 4cbb5336819548ebb1239ee9016910ec P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:18.899024 18645 ts_tablet_manager.cc:1367] T 4cbb5336819548ebb1239ee9016910ec P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:18.899044 18643 consensus_queue.cc:260] T 4a4cc6012ea54b5791d0df2ffe36f7eb P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:18.899636 18643 ts_tablet_manager.cc:1398] T 4a4cc6012ea54b5791d0df2ffe36f7eb P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:18.899891 18643 tablet_bootstrap.cc:492] T 4cbb5336819548ebb1239ee9016910ec P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:18.900060 18645 raft_consensus.cc:348] T 4cbb5336819548ebb1239ee9016910ec P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:18.900200 18645 raft_consensus.cc:374] T 4cbb5336819548ebb1239ee9016910ec P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:18.900305 18645 raft_consensus.cc:729] T 4cbb5336819548ebb1239ee9016910ec P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:18.900482 18645 consensus_queue.cc:260] T 4cbb5336819548ebb1239ee9016910ec P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:18.900946 18645 ts_tablet_manager.cc:1398] T 4cbb5336819548ebb1239ee9016910ec P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:18.901319 18643 tablet_bootstrap.cc:654] T 4cbb5336819548ebb1239ee9016910ec P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:18.903537 18643 tablet_bootstrap.cc:492] T 4cbb5336819548ebb1239ee9016910ec P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:18.903678 18643 ts_tablet_manager.cc:1367] T 4cbb5336819548ebb1239ee9016910ec P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:53:18.904403 18643 raft_consensus.cc:348] T 4cbb5336819548ebb1239ee9016910ec P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:18.904596 18643 raft_consensus.cc:374] T 4cbb5336819548ebb1239ee9016910ec P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:18.904678 18643 raft_consensus.cc:729] T 4cbb5336819548ebb1239ee9016910ec P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:18.904855 18643 consensus_queue.cc:260] T 4cbb5336819548ebb1239ee9016910ec P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:18.905257 18643 ts_tablet_manager.cc:1398] T 4cbb5336819548ebb1239ee9016910ec P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 01:53:18.923434 18524 raft_consensus.cc:482] T 4a4cc6012ea54b5791d0df2ffe36f7eb P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:18.923573 18524 raft_consensus.cc:504] T 4a4cc6012ea54b5791d0df2ffe36f7eb P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:18.924131 18524 leader_election.cc:288] T 4a4cc6012ea54b5791d0df2ffe36f7eb P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:18.924481 17476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4cc6012ea54b5791d0df2ffe36f7eb" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:18.924571 17606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4cc6012ea54b5791d0df2ffe36f7eb" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:53:18.924680 17476 raft_consensus.cc:2454] T 4a4cc6012ea54b5791d0df2ffe36f7eb P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 0.
I20240502 01:53:18.924803 17606 raft_consensus.cc:2454] T 4a4cc6012ea54b5791d0df2ffe36f7eb P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 0.
I20240502 01:53:18.925105 17802 leader_election.cc:302] T 4a4cc6012ea54b5791d0df2ffe36f7eb P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:53:18.925365 18524 raft_consensus.cc:2781] T 4a4cc6012ea54b5791d0df2ffe36f7eb P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:18.925473 18524 raft_consensus.cc:482] T 4a4cc6012ea54b5791d0df2ffe36f7eb P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:18.925561 18524 raft_consensus.cc:3037] T 4a4cc6012ea54b5791d0df2ffe36f7eb P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:18.927410 18524 raft_consensus.cc:504] T 4a4cc6012ea54b5791d0df2ffe36f7eb P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:18.927917 18524 leader_election.cc:288] T 4a4cc6012ea54b5791d0df2ffe36f7eb P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 election: Requested vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:18.928278 17476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4cc6012ea54b5791d0df2ffe36f7eb" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:53:18.928455 17476 raft_consensus.cc:3037] T 4a4cc6012ea54b5791d0df2ffe36f7eb P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:18.928421 17606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4cc6012ea54b5791d0df2ffe36f7eb" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:53:18.928576 17606 raft_consensus.cc:3037] T 4a4cc6012ea54b5791d0df2ffe36f7eb P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:18.930281 17476 raft_consensus.cc:2454] T 4a4cc6012ea54b5791d0df2ffe36f7eb P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 1.
I20240502 01:53:18.930459 17606 raft_consensus.cc:2454] T 4a4cc6012ea54b5791d0df2ffe36f7eb P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 1.
I20240502 01:53:18.930631 17802 leader_election.cc:302] T 4a4cc6012ea54b5791d0df2ffe36f7eb P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:53:18.931046 18524 raft_consensus.cc:2781] T 4a4cc6012ea54b5791d0df2ffe36f7eb P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:18.931191 18524 raft_consensus.cc:686] T 4a4cc6012ea54b5791d0df2ffe36f7eb P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Becoming Leader. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Running, Role: LEADER
I20240502 01:53:18.931455 18524 consensus_queue.cc:237] T 4a4cc6012ea54b5791d0df2ffe36f7eb P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:18.933859 17333 catalog_manager.cc:5484] T 4a4cc6012ea54b5791d0df2ffe36f7eb P bb31bceba76d466db24b9341c04900c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb31bceba76d466db24b9341c04900c3 (127.11.42.68). New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:18.984480 18475 raft_consensus.cc:482] T 4cbb5336819548ebb1239ee9016910ec P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:18.984637 18475 raft_consensus.cc:504] T 4cbb5336819548ebb1239ee9016910ec P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:18.985308 18475 leader_election.cc:288] T 4cbb5336819548ebb1239ee9016910ec P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:53:18.985744 17866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cbb5336819548ebb1239ee9016910ec" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:53:18.985735 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cbb5336819548ebb1239ee9016910ec" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:53:18.985940 17866 raft_consensus.cc:2454] T 4cbb5336819548ebb1239ee9016910ec P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 0.
I20240502 01:53:18.985944 17737 raft_consensus.cc:2454] T 4cbb5336819548ebb1239ee9016910ec P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 0.
I20240502 01:53:18.986335 17411 leader_election.cc:302] T 4cbb5336819548ebb1239ee9016910ec P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:18.986591 18475 raft_consensus.cc:2781] T 4cbb5336819548ebb1239ee9016910ec P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:18.986706 18475 raft_consensus.cc:482] T 4cbb5336819548ebb1239ee9016910ec P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:18.986800 18475 raft_consensus.cc:3037] T 4cbb5336819548ebb1239ee9016910ec P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:18.988734 18475 raft_consensus.cc:504] T 4cbb5336819548ebb1239ee9016910ec P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:18.989377 18475 leader_election.cc:288] T 4cbb5336819548ebb1239ee9016910ec P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Requested vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:53:18.989714 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cbb5336819548ebb1239ee9016910ec" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:53:18.989796 17866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cbb5336819548ebb1239ee9016910ec" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:53:18.989871 17737 raft_consensus.cc:3037] T 4cbb5336819548ebb1239ee9016910ec P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:18.989967 17866 raft_consensus.cc:3037] T 4cbb5336819548ebb1239ee9016910ec P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:18.991679 17866 raft_consensus.cc:2454] T 4cbb5336819548ebb1239ee9016910ec P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
I20240502 01:53:18.991709 17737 raft_consensus.cc:2454] T 4cbb5336819548ebb1239ee9016910ec P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
I20240502 01:53:18.992051 17412 leader_election.cc:302] T 4cbb5336819548ebb1239ee9016910ec P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:53:18.992338 18475 raft_consensus.cc:2781] T 4cbb5336819548ebb1239ee9016910ec P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:18.992669 18475 raft_consensus.cc:686] T 4cbb5336819548ebb1239ee9016910ec P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Becoming Leader. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Running, Role: LEADER
I20240502 01:53:18.992986 18475 consensus_queue.cc:237] T 4cbb5336819548ebb1239ee9016910ec P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:18.995503 17333 catalog_manager.cc:5484] T 4cbb5336819548ebb1239ee9016910ec P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: term changed from 0 to 1, leader changed from <none> to 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65). New cstate: current_term: 1 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:19.002785 18524 raft_consensus.cc:482] T 5ad48f64a0f54050a5b6112ebb551228 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:19.002925 18524 raft_consensus.cc:504] T 5ad48f64a0f54050a5b6112ebb551228 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:19.003527 18524 leader_election.cc:288] T 5ad48f64a0f54050a5b6112ebb551228 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:19.003937 17476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad48f64a0f54050a5b6112ebb551228" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:19.004117 17476 raft_consensus.cc:2454] T 5ad48f64a0f54050a5b6112ebb551228 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 0.
I20240502 01:53:19.004065 17606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad48f64a0f54050a5b6112ebb551228" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:53:19.004330 17606 raft_consensus.cc:2454] T 5ad48f64a0f54050a5b6112ebb551228 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 0.
I20240502 01:53:19.004439 17802 leader_election.cc:302] T 5ad48f64a0f54050a5b6112ebb551228 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:53:19.004719 18524 raft_consensus.cc:2781] T 5ad48f64a0f54050a5b6112ebb551228 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:19.004823 18524 raft_consensus.cc:482] T 5ad48f64a0f54050a5b6112ebb551228 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:19.004920 18524 raft_consensus.cc:3037] T 5ad48f64a0f54050a5b6112ebb551228 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:19.006665 18524 raft_consensus.cc:504] T 5ad48f64a0f54050a5b6112ebb551228 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:19.007234 18524 leader_election.cc:288] T 5ad48f64a0f54050a5b6112ebb551228 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 election: Requested vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:19.007547 17476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad48f64a0f54050a5b6112ebb551228" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:53:19.007597 17606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad48f64a0f54050a5b6112ebb551228" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:53:19.007699 17476 raft_consensus.cc:3037] T 5ad48f64a0f54050a5b6112ebb551228 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:19.007759 17606 raft_consensus.cc:3037] T 5ad48f64a0f54050a5b6112ebb551228 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:19.009505 17476 raft_consensus.cc:2454] T 5ad48f64a0f54050a5b6112ebb551228 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 1.
I20240502 01:53:19.009567 17606 raft_consensus.cc:2454] T 5ad48f64a0f54050a5b6112ebb551228 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 1.
I20240502 01:53:19.009856 17802 leader_election.cc:302] T 5ad48f64a0f54050a5b6112ebb551228 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:53:19.010115 18524 raft_consensus.cc:2781] T 5ad48f64a0f54050a5b6112ebb551228 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:19.010278 18524 raft_consensus.cc:686] T 5ad48f64a0f54050a5b6112ebb551228 P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Becoming Leader. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Running, Role: LEADER
I20240502 01:53:19.010535 18524 consensus_queue.cc:237] T 5ad48f64a0f54050a5b6112ebb551228 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:19.012943 17333 catalog_manager.cc:5484] T 5ad48f64a0f54050a5b6112ebb551228 P bb31bceba76d466db24b9341c04900c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb31bceba76d466db24b9341c04900c3 (127.11.42.68). New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } }
I20240502 01:53:19.339810 18524 consensus_queue.cc:1035] T 4a4cc6012ea54b5791d0df2ffe36f7eb P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:19.345468 18524 consensus_queue.cc:1035] T 4a4cc6012ea54b5791d0df2ffe36f7eb P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:19.364634 18660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:19.364892 18660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:19.372035 18660 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 01:53:19.444456 18475 consensus_queue.cc:1035] T 4cbb5336819548ebb1239ee9016910ec P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:19.451159 18475 consensus_queue.cc:1035] T 4cbb5336819548ebb1239ee9016910ec P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:19.491768 18524 consensus_queue.cc:1035] T 5ad48f64a0f54050a5b6112ebb551228 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:19.558455 18706 consensus_queue.cc:1035] T 5ad48f64a0f54050a5b6112ebb551228 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:19.627709 18701 mvcc.cc:205] Tried to move back new op lower bound from 7023062219046547456 to 7023062216749412352. Current Snapshot: MvccSnapshot[applied={T|T < 7023062219046547456 or (T in {7023062219046547456})}]
INSERT report
rows total: 2000
time total: 226.572 ms
time per row: 0.113286 ms
-----------------------------------------------------
Suppressions used:
count bytes template
4 1192 libcrypto.so.1.1
-----------------------------------------------------
I20240502 01:53:20.598979 17335 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:20.607443 17455 tablet_service.cc:1467] Processing CreateTablet for tablet 9642b9b37a784cf881bfc4971d8c9cb6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b996bd60793c4d3c83bf34f4989da909]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:20.607900 17455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9642b9b37a784cf881bfc4971d8c9cb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:20.609328 17587 tablet_service.cc:1467] Processing CreateTablet for tablet 9642b9b37a784cf881bfc4971d8c9cb6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b996bd60793c4d3c83bf34f4989da909]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:20.609390 17586 tablet_service.cc:1467] Processing CreateTablet for tablet f0b3862e504d4b43a60477e5efef29b1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b996bd60793c4d3c83bf34f4989da909]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:20.609373 17847 tablet_service.cc:1467] Processing CreateTablet for tablet 03bd47783e884a70bfea1dfab4f2e563 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b996bd60793c4d3c83bf34f4989da909]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:20.609767 17587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9642b9b37a784cf881bfc4971d8c9cb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:20.609807 17847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03bd47783e884a70bfea1dfab4f2e563. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:20.609995 17586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0b3862e504d4b43a60477e5efef29b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:20.613391 17456 tablet_service.cc:1467] Processing CreateTablet for tablet 03bd47783e884a70bfea1dfab4f2e563 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b996bd60793c4d3c83bf34f4989da909]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:20.613739 17456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03bd47783e884a70bfea1dfab4f2e563. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:20.613783 17716 tablet_service.cc:1467] Processing CreateTablet for tablet 03bd47783e884a70bfea1dfab4f2e563 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b996bd60793c4d3c83bf34f4989da909]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:20.613936 17717 tablet_service.cc:1467] Processing CreateTablet for tablet 9642b9b37a784cf881bfc4971d8c9cb6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b996bd60793c4d3c83bf34f4989da909]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:20.613999 17715 tablet_service.cc:1467] Processing CreateTablet for tablet f0b3862e504d4b43a60477e5efef29b1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b996bd60793c4d3c83bf34f4989da909]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:20.614197 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03bd47783e884a70bfea1dfab4f2e563. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:20.614413 17715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0b3862e504d4b43a60477e5efef29b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:20.614626 17717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9642b9b37a784cf881bfc4971d8c9cb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:20.617102 17845 tablet_service.cc:1467] Processing CreateTablet for tablet f0b3862e504d4b43a60477e5efef29b1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b996bd60793c4d3c83bf34f4989da909]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:20.617446 17845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0b3862e504d4b43a60477e5efef29b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:20.632076 18731 tablet_bootstrap.cc:492] T 9642b9b37a784cf881bfc4971d8c9cb6 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:20.632648 18728 tablet_bootstrap.cc:492] T f0b3862e504d4b43a60477e5efef29b1 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:20.634059 18731 tablet_bootstrap.cc:654] T 9642b9b37a784cf881bfc4971d8c9cb6 P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:20.634436 18728 tablet_bootstrap.cc:654] T f0b3862e504d4b43a60477e5efef29b1 P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:20.634742 18730 tablet_bootstrap.cc:492] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:20.635442 18729 tablet_bootstrap.cc:492] T 9642b9b37a784cf881bfc4971d8c9cb6 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:20.636847 18729 tablet_bootstrap.cc:654] T 9642b9b37a784cf881bfc4971d8c9cb6 P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:20.636854 18730 tablet_bootstrap.cc:654] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:20.637856 18731 tablet_bootstrap.cc:492] T 9642b9b37a784cf881bfc4971d8c9cb6 P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:20.637864 18728 tablet_bootstrap.cc:492] T f0b3862e504d4b43a60477e5efef29b1 P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:20.638001 18728 ts_tablet_manager.cc:1367] T f0b3862e504d4b43a60477e5efef29b1 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 01:53:20.638073 18731 ts_tablet_manager.cc:1367] T 9642b9b37a784cf881bfc4971d8c9cb6 P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 01:53:20.638908 18731 raft_consensus.cc:348] T 9642b9b37a784cf881bfc4971d8c9cb6 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:20.639179 18731 raft_consensus.cc:374] T 9642b9b37a784cf881bfc4971d8c9cb6 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:20.639298 18731 raft_consensus.cc:729] T 9642b9b37a784cf881bfc4971d8c9cb6 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:20.639393 18728 raft_consensus.cc:348] T f0b3862e504d4b43a60477e5efef29b1 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:20.639693 18728 raft_consensus.cc:374] T f0b3862e504d4b43a60477e5efef29b1 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:20.639631 18731 consensus_queue.cc:260] T 9642b9b37a784cf881bfc4971d8c9cb6 P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:20.639827 18728 raft_consensus.cc:729] T f0b3862e504d4b43a60477e5efef29b1 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:20.640393 18731 ts_tablet_manager.cc:1398] T 9642b9b37a784cf881bfc4971d8c9cb6 P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:20.641101 18731 tablet_bootstrap.cc:492] T 03bd47783e884a70bfea1dfab4f2e563 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:20.642980 18731 tablet_bootstrap.cc:654] T 03bd47783e884a70bfea1dfab4f2e563 P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:20.644891 18729 tablet_bootstrap.cc:492] T 9642b9b37a784cf881bfc4971d8c9cb6 P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:20.644992 18730 tablet_bootstrap.cc:492] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:20.645123 18729 ts_tablet_manager.cc:1367] T 9642b9b37a784cf881bfc4971d8c9cb6 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20240502 01:53:20.645382 18730 ts_tablet_manager.cc:1367] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.001s
I20240502 01:53:20.646031 18729 raft_consensus.cc:348] T 9642b9b37a784cf881bfc4971d8c9cb6 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:20.646188 18730 raft_consensus.cc:348] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:20.646315 18729 raft_consensus.cc:374] T 9642b9b37a784cf881bfc4971d8c9cb6 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:20.646462 18729 raft_consensus.cc:729] T 9642b9b37a784cf881bfc4971d8c9cb6 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:20.646466 18730 raft_consensus.cc:374] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:20.646570 18730 raft_consensus.cc:729] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:20.646847 18730 consensus_queue.cc:260] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:20.646991 18729 consensus_queue.cc:260] T 9642b9b37a784cf881bfc4971d8c9cb6 P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:20.647461 18730 ts_tablet_manager.cc:1398] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:53:20.647742 18729 ts_tablet_manager.cc:1398] T 9642b9b37a784cf881bfc4971d8c9cb6 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:20.648073 18729 tablet_bootstrap.cc:492] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:20.649178 18730 tablet_bootstrap.cc:492] T f0b3862e504d4b43a60477e5efef29b1 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:20.649967 18729 tablet_bootstrap.cc:654] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:20.651232 18730 tablet_bootstrap.cc:654] T f0b3862e504d4b43a60477e5efef29b1 P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:20.656229 18730 tablet_bootstrap.cc:492] T f0b3862e504d4b43a60477e5efef29b1 P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:20.656462 18730 ts_tablet_manager.cc:1367] T f0b3862e504d4b43a60477e5efef29b1 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20240502 01:53:20.657259 18730 raft_consensus.cc:348] T f0b3862e504d4b43a60477e5efef29b1 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:20.657568 18730 raft_consensus.cc:374] T f0b3862e504d4b43a60477e5efef29b1 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:20.657687 18730 raft_consensus.cc:729] T f0b3862e504d4b43a60477e5efef29b1 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:20.657949 18730 consensus_queue.cc:260] T f0b3862e504d4b43a60477e5efef29b1 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:20.658012 18728 consensus_queue.cc:260] T f0b3862e504d4b43a60477e5efef29b1 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:20.658987 18728 ts_tablet_manager.cc:1398] T f0b3862e504d4b43a60477e5efef29b1 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20240502 01:53:20.659353 18729 tablet_bootstrap.cc:492] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:20.659560 18729 ts_tablet_manager.cc:1367] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.010s
I20240502 01:53:20.660960 18728 tablet_bootstrap.cc:492] T 9642b9b37a784cf881bfc4971d8c9cb6 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:20.661834 18730 ts_tablet_manager.cc:1398] T f0b3862e504d4b43a60477e5efef29b1 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20240502 01:53:20.663770 18729 raft_consensus.cc:348] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:20.664978 18729 raft_consensus.cc:374] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:20.665123 18729 raft_consensus.cc:729] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:20.665958 18729 consensus_queue.cc:260] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:20.666596 18728 tablet_bootstrap.cc:654] T 9642b9b37a784cf881bfc4971d8c9cb6 P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:20.666632 18729 ts_tablet_manager.cc:1398] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 01:53:20.667147 18731 tablet_bootstrap.cc:492] T 03bd47783e884a70bfea1dfab4f2e563 P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:20.667330 18731 ts_tablet_manager.cc:1367] T 03bd47783e884a70bfea1dfab4f2e563 P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.026s user 0.010s sys 0.010s
I20240502 01:53:20.667982 18731 raft_consensus.cc:348] T 03bd47783e884a70bfea1dfab4f2e563 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:20.668246 18731 raft_consensus.cc:374] T 03bd47783e884a70bfea1dfab4f2e563 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:20.668349 18731 raft_consensus.cc:729] T 03bd47783e884a70bfea1dfab4f2e563 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:20.668573 18728 tablet_bootstrap.cc:492] T 9642b9b37a784cf881bfc4971d8c9cb6 P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:20.668757 18728 ts_tablet_manager.cc:1367] T 9642b9b37a784cf881bfc4971d8c9cb6 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240502 01:53:20.669178 18731 consensus_queue.cc:260] T 03bd47783e884a70bfea1dfab4f2e563 P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:20.669543 18728 raft_consensus.cc:348] T 9642b9b37a784cf881bfc4971d8c9cb6 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:20.669759 18731 ts_tablet_manager.cc:1398] T 03bd47783e884a70bfea1dfab4f2e563 P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:20.669811 18728 raft_consensus.cc:374] T 9642b9b37a784cf881bfc4971d8c9cb6 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:20.669888 18728 raft_consensus.cc:729] T 9642b9b37a784cf881bfc4971d8c9cb6 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:20.670150 18728 consensus_queue.cc:260] T 9642b9b37a784cf881bfc4971d8c9cb6 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:20.670691 18728 ts_tablet_manager.cc:1398] T 9642b9b37a784cf881bfc4971d8c9cb6 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:20.670997 18728 tablet_bootstrap.cc:492] T 03bd47783e884a70bfea1dfab4f2e563 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:20.672478 18728 tablet_bootstrap.cc:654] T 03bd47783e884a70bfea1dfab4f2e563 P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:20.675012 18728 tablet_bootstrap.cc:492] T 03bd47783e884a70bfea1dfab4f2e563 P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:20.675149 18728 ts_tablet_manager.cc:1367] T 03bd47783e884a70bfea1dfab4f2e563 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 01:53:20.675873 18728 raft_consensus.cc:348] T 03bd47783e884a70bfea1dfab4f2e563 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:20.676015 18728 raft_consensus.cc:374] T 03bd47783e884a70bfea1dfab4f2e563 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:20.676080 18728 raft_consensus.cc:729] T 03bd47783e884a70bfea1dfab4f2e563 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:20.676224 18728 consensus_queue.cc:260] T 03bd47783e884a70bfea1dfab4f2e563 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:20.676615 18728 ts_tablet_manager.cc:1398] T 03bd47783e884a70bfea1dfab4f2e563 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:53:20.684051 18737 raft_consensus.cc:482] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:20.684183 18737 raft_consensus.cc:504] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:20.684798 18737 leader_election.cc:288] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:20.685153 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03bd47783e884a70bfea1dfab4f2e563" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:53:20.685261 17476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03bd47783e884a70bfea1dfab4f2e563" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:20.685392 17737 raft_consensus.cc:2454] T 03bd47783e884a70bfea1dfab4f2e563 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 0.
I20240502 01:53:20.685498 17476 raft_consensus.cc:2454] T 03bd47783e884a70bfea1dfab4f2e563 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 0.
I20240502 01:53:20.685798 17802 leader_election.cc:302] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb31bceba76d466db24b9341c04900c3, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:20.686069 18737 raft_consensus.cc:2781] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:20.686185 18737 raft_consensus.cc:482] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:20.686295 18737 raft_consensus.cc:3037] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:20.688316 18737 raft_consensus.cc:504] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:20.688804 18737 leader_election.cc:288] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 election: Requested vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:20.689205 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03bd47783e884a70bfea1dfab4f2e563" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:53:20.689303 17476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03bd47783e884a70bfea1dfab4f2e563" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:53:20.689378 17737 raft_consensus.cc:3037] T 03bd47783e884a70bfea1dfab4f2e563 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:20.689483 17476 raft_consensus.cc:3037] T 03bd47783e884a70bfea1dfab4f2e563 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:20.691134 17737 raft_consensus.cc:2454] T 03bd47783e884a70bfea1dfab4f2e563 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 1.
I20240502 01:53:20.691277 17476 raft_consensus.cc:2454] T 03bd47783e884a70bfea1dfab4f2e563 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 1.
I20240502 01:53:20.691505 17802 leader_election.cc:302] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb31bceba76d466db24b9341c04900c3, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:20.692059 18737 raft_consensus.cc:2781] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:20.692214 18737 raft_consensus.cc:686] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Becoming Leader. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Running, Role: LEADER
I20240502 01:53:20.692704 18737 consensus_queue.cc:237] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:20.695724 17331 catalog_manager.cc:5484] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb31bceba76d466db24b9341c04900c3 (127.11.42.68). New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:20.740731 18570 raft_consensus.cc:482] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:20.740880 18570 raft_consensus.cc:504] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:20.741639 18570 leader_election.cc:288] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:53:20.742026 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b3862e504d4b43a60477e5efef29b1" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:53:20.742298 17737 raft_consensus.cc:2454] T f0b3862e504d4b43a60477e5efef29b1 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:53:20.742749 17542 leader_election.cc:302] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:20.743007 18570 raft_consensus.cc:2781] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:20.743142 18570 raft_consensus.cc:482] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:20.743259 18570 raft_consensus.cc:3037] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:20.745847 18570 raft_consensus.cc:504] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:20.746409 18570 leader_election.cc:288] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Requested vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:53:20.746847 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b3862e504d4b43a60477e5efef29b1" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:53:20.747061 17737 raft_consensus.cc:3037] T f0b3862e504d4b43a60477e5efef29b1 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:20.748837 17866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b3862e504d4b43a60477e5efef29b1" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:53:20.748960 17867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b3862e504d4b43a60477e5efef29b1" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:53:20.749076 17866 raft_consensus.cc:2454] T f0b3862e504d4b43a60477e5efef29b1 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:53:20.749547 17737 raft_consensus.cc:2454] T f0b3862e504d4b43a60477e5efef29b1 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:53:20.749845 17542 leader_election.cc:302] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, dfa1152987644bd781b46c7e42a7a685; no voters: bb31bceba76d466db24b9341c04900c3
I20240502 01:53:20.750128 18570 raft_consensus.cc:2781] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:20.750468 18570 raft_consensus.cc:686] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Becoming Leader. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Running, Role: LEADER
I20240502 01:53:20.750761 18570 consensus_queue.cc:237] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:20.752233 18689 raft_consensus.cc:482] T 9642b9b37a784cf881bfc4971d8c9cb6 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:20.752409 18689 raft_consensus.cc:504] T 9642b9b37a784cf881bfc4971d8c9cb6 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:20.753154 18689 leader_election.cc:288] T 9642b9b37a784cf881bfc4971d8c9cb6 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:20.753430 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9642b9b37a784cf881bfc4971d8c9cb6" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:53:20.753613 17737 raft_consensus.cc:2454] T 9642b9b37a784cf881bfc4971d8c9cb6 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 0.
I20240502 01:53:20.753563 17331 catalog_manager.cc:5484] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66). New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } }
I20240502 01:53:20.754050 17411 leader_election.cc:302] T 9642b9b37a784cf881bfc4971d8c9cb6 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:20.754359 18689 raft_consensus.cc:2781] T 9642b9b37a784cf881bfc4971d8c9cb6 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:20.754438 18689 raft_consensus.cc:482] T 9642b9b37a784cf881bfc4971d8c9cb6 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:20.754491 18689 raft_consensus.cc:3037] T 9642b9b37a784cf881bfc4971d8c9cb6 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:20.756558 18689 raft_consensus.cc:504] T 9642b9b37a784cf881bfc4971d8c9cb6 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:20.757371 18689 leader_election.cc:288] T 9642b9b37a784cf881bfc4971d8c9cb6 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Requested vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:20.757483 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9642b9b37a784cf881bfc4971d8c9cb6" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:53:20.757681 17737 raft_consensus.cc:3037] T 9642b9b37a784cf881bfc4971d8c9cb6 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:20.760033 17737 raft_consensus.cc:2454] T 9642b9b37a784cf881bfc4971d8c9cb6 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
I20240502 01:53:20.760397 17411 leader_election.cc:302] T 9642b9b37a784cf881bfc4971d8c9cb6 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:20.760699 18689 raft_consensus.cc:2781] T 9642b9b37a784cf881bfc4971d8c9cb6 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:20.761085 18689 raft_consensus.cc:686] T 9642b9b37a784cf881bfc4971d8c9cb6 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Becoming Leader. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Running, Role: LEADER
I20240502 01:53:20.761377 18689 consensus_queue.cc:237] T 9642b9b37a784cf881bfc4971d8c9cb6 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:20.763481 17606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9642b9b37a784cf881bfc4971d8c9cb6" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:53:20.763583 17607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9642b9b37a784cf881bfc4971d8c9cb6" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:53:20.763684 17606 raft_consensus.cc:2454] T 9642b9b37a784cf881bfc4971d8c9cb6 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 0.
I20240502 01:53:20.763806 17607 raft_consensus.cc:3037] T 9642b9b37a784cf881bfc4971d8c9cb6 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:20.764241 17331 catalog_manager.cc:5484] T 9642b9b37a784cf881bfc4971d8c9cb6 P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: term changed from 0 to 1, leader changed from <none> to 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65). New cstate: current_term: 1 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:20.766152 17607 raft_consensus.cc:2454] T 9642b9b37a784cf881bfc4971d8c9cb6 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
W20240502 01:53:21.090588 18752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:21.090896 18752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:21.098398 18752 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240502 01:53:21.194172 18737 consensus_queue.cc:1035] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:21.220713 18570 consensus_queue.cc:1035] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:21.229965 18737 consensus_queue.cc:1035] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:21.272714 17867 raft_consensus.cc:3037] T f0b3862e504d4b43a60477e5efef29b1 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:21.282945 18791 consensus_queue.cc:1035] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:21.309376 18689 consensus_queue.cc:1035] T 9642b9b37a784cf881bfc4971d8c9cb6 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:21.319886 17607 raft_consensus.cc:1261] T 9642b9b37a784cf881bfc4971d8c9cb6 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:21.320451 18689 consensus_queue.cc:1035] T 9642b9b37a784cf881bfc4971d8c9cb6 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:21.384801 18790 mvcc.cc:205] Tried to move back new op lower bound from 7023062226199932928 to 7023062223920754688. Current Snapshot: MvccSnapshot[applied={T|T < 7023062226199932928}]
INSERT report
rows total: 2000
time total: 254.586 ms
time per row: 0.127293 ms
-----------------------------------------------------
Suppressions used:
count bytes template
6 1788 libcrypto.so.1.1
-----------------------------------------------------
I20240502 01:53:22.350589 17335 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:22.359118 17455 tablet_service.cc:1467] Processing CreateTablet for tablet 2f3528ac3c114a19892bba2304474302 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0a0284719f394d1dab3221ab9abcf498]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:22.359606 17455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f3528ac3c114a19892bba2304474302. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:22.359709 17847 tablet_service.cc:1467] Processing CreateTablet for tablet 2f3528ac3c114a19892bba2304474302 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0a0284719f394d1dab3221ab9abcf498]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:22.360047 17847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f3528ac3c114a19892bba2304474302. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:22.360589 17587 tablet_service.cc:1467] Processing CreateTablet for tablet 2f3528ac3c114a19892bba2304474302 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0a0284719f394d1dab3221ab9abcf498]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:22.360981 17587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f3528ac3c114a19892bba2304474302. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:22.364099 17845 tablet_service.cc:1467] Processing CreateTablet for tablet 2f29bba45e1749eb96b08da5e15fb834 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0a0284719f394d1dab3221ab9abcf498]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:22.364329 17586 tablet_service.cc:1467] Processing CreateTablet for tablet ae24680ad75545d892c9b79833319214 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0a0284719f394d1dab3221ab9abcf498]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:22.364441 17845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f29bba45e1749eb96b08da5e15fb834. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:22.364715 17586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae24680ad75545d892c9b79833319214. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:22.367812 17715 tablet_service.cc:1467] Processing CreateTablet for tablet ae24680ad75545d892c9b79833319214 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0a0284719f394d1dab3221ab9abcf498]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:22.368046 17456 tablet_service.cc:1467] Processing CreateTablet for tablet ae24680ad75545d892c9b79833319214 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0a0284719f394d1dab3221ab9abcf498]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:22.368233 17715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae24680ad75545d892c9b79833319214. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:22.368453 17456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae24680ad75545d892c9b79833319214. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:22.369414 17454 tablet_service.cc:1467] Processing CreateTablet for tablet 2f29bba45e1749eb96b08da5e15fb834 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0a0284719f394d1dab3221ab9abcf498]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:22.369875 17454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f29bba45e1749eb96b08da5e15fb834. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:22.370393 18820 tablet_bootstrap.cc:492] T 2f3528ac3c114a19892bba2304474302 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:22.373440 17716 tablet_service.cc:1467] Processing CreateTablet for tablet 2f29bba45e1749eb96b08da5e15fb834 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0a0284719f394d1dab3221ab9abcf498]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:22.373783 18820 tablet_bootstrap.cc:654] T 2f3528ac3c114a19892bba2304474302 P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:22.373893 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f29bba45e1749eb96b08da5e15fb834. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:22.381057 18820 tablet_bootstrap.cc:492] T 2f3528ac3c114a19892bba2304474302 P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:22.381691 18820 ts_tablet_manager.cc:1367] T 2f3528ac3c114a19892bba2304474302 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.004s
I20240502 01:53:22.382584 18821 tablet_bootstrap.cc:492] T 2f3528ac3c114a19892bba2304474302 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:22.382694 18820 raft_consensus.cc:348] T 2f3528ac3c114a19892bba2304474302 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:22.383868 18820 raft_consensus.cc:374] T 2f3528ac3c114a19892bba2304474302 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:22.384088 18820 raft_consensus.cc:729] T 2f3528ac3c114a19892bba2304474302 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:22.384459 18820 consensus_queue.cc:260] T 2f3528ac3c114a19892bba2304474302 P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:22.385387 18820 ts_tablet_manager.cc:1398] T 2f3528ac3c114a19892bba2304474302 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:53:22.385828 18821 tablet_bootstrap.cc:654] T 2f3528ac3c114a19892bba2304474302 P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:22.388649 18820 tablet_bootstrap.cc:492] T ae24680ad75545d892c9b79833319214 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:22.390167 18820 tablet_bootstrap.cc:654] T ae24680ad75545d892c9b79833319214 P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:22.391357 18823 tablet_bootstrap.cc:492] T 2f29bba45e1749eb96b08da5e15fb834 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:22.392984 18823 tablet_bootstrap.cc:654] T 2f29bba45e1749eb96b08da5e15fb834 P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:22.396842 18823 tablet_bootstrap.cc:492] T 2f29bba45e1749eb96b08da5e15fb834 P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:22.397068 18823 ts_tablet_manager.cc:1367] T 2f29bba45e1749eb96b08da5e15fb834 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 01:53:22.398030 18823 raft_consensus.cc:348] T 2f29bba45e1749eb96b08da5e15fb834 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:22.398345 18823 raft_consensus.cc:374] T 2f29bba45e1749eb96b08da5e15fb834 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:22.398970 18821 tablet_bootstrap.cc:492] T 2f3528ac3c114a19892bba2304474302 P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:22.398986 18820 tablet_bootstrap.cc:492] T ae24680ad75545d892c9b79833319214 P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:22.399114 18821 ts_tablet_manager.cc:1367] T 2f3528ac3c114a19892bba2304474302 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.007s
I20240502 01:53:22.399142 18820 ts_tablet_manager.cc:1367] T ae24680ad75545d892c9b79833319214 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20240502 01:53:22.399938 18823 raft_consensus.cc:729] T 2f29bba45e1749eb96b08da5e15fb834 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:22.399881 18821 raft_consensus.cc:348] T 2f3528ac3c114a19892bba2304474302 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:22.399904 18820 raft_consensus.cc:348] T ae24680ad75545d892c9b79833319214 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:22.400071 18821 raft_consensus.cc:374] T 2f3528ac3c114a19892bba2304474302 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:22.400130 18820 raft_consensus.cc:374] T ae24680ad75545d892c9b79833319214 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:22.400161 18821 raft_consensus.cc:729] T 2f3528ac3c114a19892bba2304474302 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:22.400209 18820 raft_consensus.cc:729] T ae24680ad75545d892c9b79833319214 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:22.400367 18821 consensus_queue.cc:260] T 2f3528ac3c114a19892bba2304474302 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:22.400512 18820 consensus_queue.cc:260] T ae24680ad75545d892c9b79833319214 P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:22.400463 18823 consensus_queue.cc:260] T 2f29bba45e1749eb96b08da5e15fb834 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:22.401068 18821 ts_tablet_manager.cc:1398] T 2f3528ac3c114a19892bba2304474302 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:53:22.401234 18823 ts_tablet_manager.cc:1398] T 2f29bba45e1749eb96b08da5e15fb834 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:53:22.401392 18821 tablet_bootstrap.cc:492] T 2f29bba45e1749eb96b08da5e15fb834 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:22.402940 18821 tablet_bootstrap.cc:654] T 2f29bba45e1749eb96b08da5e15fb834 P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:22.404158 18820 ts_tablet_manager.cc:1398] T ae24680ad75545d892c9b79833319214 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 01:53:22.405331 18819 tablet_bootstrap.cc:492] T 2f3528ac3c114a19892bba2304474302 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:22.405930 18821 tablet_bootstrap.cc:492] T 2f29bba45e1749eb96b08da5e15fb834 P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:22.406138 18821 ts_tablet_manager.cc:1367] T 2f29bba45e1749eb96b08da5e15fb834 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240502 01:53:22.406672 18823 tablet_bootstrap.cc:492] T ae24680ad75545d892c9b79833319214 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:22.406853 18821 raft_consensus.cc:348] T 2f29bba45e1749eb96b08da5e15fb834 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:22.407060 18821 raft_consensus.cc:374] T 2f29bba45e1749eb96b08da5e15fb834 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:22.407145 18821 raft_consensus.cc:729] T 2f29bba45e1749eb96b08da5e15fb834 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:22.407444 18821 consensus_queue.cc:260] T 2f29bba45e1749eb96b08da5e15fb834 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:22.407572 18819 tablet_bootstrap.cc:654] T 2f3528ac3c114a19892bba2304474302 P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:22.408031 18821 ts_tablet_manager.cc:1398] T 2f29bba45e1749eb96b08da5e15fb834 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:53:22.408385 18823 tablet_bootstrap.cc:654] T ae24680ad75545d892c9b79833319214 P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:22.409649 18819 tablet_bootstrap.cc:492] T 2f3528ac3c114a19892bba2304474302 P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:22.409804 18819 ts_tablet_manager.cc:1367] T 2f3528ac3c114a19892bba2304474302 P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 01:53:22.410888 18819 raft_consensus.cc:348] T 2f3528ac3c114a19892bba2304474302 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:22.411046 18819 raft_consensus.cc:374] T 2f3528ac3c114a19892bba2304474302 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:22.411161 18819 raft_consensus.cc:729] T 2f3528ac3c114a19892bba2304474302 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:22.411419 18819 consensus_queue.cc:260] T 2f3528ac3c114a19892bba2304474302 P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:22.411899 18819 ts_tablet_manager.cc:1398] T 2f3528ac3c114a19892bba2304474302 P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:22.412134 18819 tablet_bootstrap.cc:492] T ae24680ad75545d892c9b79833319214 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:22.413553 18819 tablet_bootstrap.cc:654] T ae24680ad75545d892c9b79833319214 P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:22.414000 18823 tablet_bootstrap.cc:492] T ae24680ad75545d892c9b79833319214 P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:22.414166 18823 ts_tablet_manager.cc:1367] T ae24680ad75545d892c9b79833319214 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240502 01:53:22.414882 18823 raft_consensus.cc:348] T ae24680ad75545d892c9b79833319214 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:22.415038 18819 tablet_bootstrap.cc:492] T ae24680ad75545d892c9b79833319214 P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:22.415098 18823 raft_consensus.cc:374] T ae24680ad75545d892c9b79833319214 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:22.415175 18819 ts_tablet_manager.cc:1367] T ae24680ad75545d892c9b79833319214 P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:53:22.415184 18823 raft_consensus.cc:729] T ae24680ad75545d892c9b79833319214 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:22.415417 18823 consensus_queue.cc:260] T ae24680ad75545d892c9b79833319214 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:22.415771 18819 raft_consensus.cc:348] T ae24680ad75545d892c9b79833319214 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:22.415932 18819 raft_consensus.cc:374] T ae24680ad75545d892c9b79833319214 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:22.415967 18823 ts_tablet_manager.cc:1398] T ae24680ad75545d892c9b79833319214 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:53:22.416002 18819 raft_consensus.cc:729] T ae24680ad75545d892c9b79833319214 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:22.416291 18819 consensus_queue.cc:260] T ae24680ad75545d892c9b79833319214 P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:22.416740 18819 ts_tablet_manager.cc:1398] T ae24680ad75545d892c9b79833319214 P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:53:22.417057 18819 tablet_bootstrap.cc:492] T 2f29bba45e1749eb96b08da5e15fb834 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:22.418417 18819 tablet_bootstrap.cc:654] T 2f29bba45e1749eb96b08da5e15fb834 P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:22.420706 18819 tablet_bootstrap.cc:492] T 2f29bba45e1749eb96b08da5e15fb834 P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:22.420851 18819 ts_tablet_manager.cc:1367] T 2f29bba45e1749eb96b08da5e15fb834 P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240502 01:53:22.421568 18819 raft_consensus.cc:348] T 2f29bba45e1749eb96b08da5e15fb834 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:22.421705 18819 raft_consensus.cc:374] T 2f29bba45e1749eb96b08da5e15fb834 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:22.421783 18819 raft_consensus.cc:729] T 2f29bba45e1749eb96b08da5e15fb834 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:22.421958 18819 consensus_queue.cc:260] T 2f29bba45e1749eb96b08da5e15fb834 P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:22.422365 18819 ts_tablet_manager.cc:1398] T 2f29bba45e1749eb96b08da5e15fb834 P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:53:22.449297 18805 raft_consensus.cc:482] T 2f29bba45e1749eb96b08da5e15fb834 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:22.449435 18805 raft_consensus.cc:504] T 2f29bba45e1749eb96b08da5e15fb834 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:22.450047 18805 leader_election.cc:288] T 2f29bba45e1749eb96b08da5e15fb834 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:53:22.450595 17866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f29bba45e1749eb96b08da5e15fb834" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:53:22.450627 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f29bba45e1749eb96b08da5e15fb834" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:53:22.450852 17866 raft_consensus.cc:2454] T 2f29bba45e1749eb96b08da5e15fb834 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 0.
I20240502 01:53:22.450879 17737 raft_consensus.cc:2454] T 2f29bba45e1749eb96b08da5e15fb834 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 0.
I20240502 01:53:22.451321 17412 leader_election.cc:302] T 2f29bba45e1749eb96b08da5e15fb834 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, bb31bceba76d466db24b9341c04900c3, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:22.451565 18805 raft_consensus.cc:2781] T 2f29bba45e1749eb96b08da5e15fb834 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:22.451668 18805 raft_consensus.cc:482] T 2f29bba45e1749eb96b08da5e15fb834 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:22.451754 18805 raft_consensus.cc:3037] T 2f29bba45e1749eb96b08da5e15fb834 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:22.453624 18805 raft_consensus.cc:504] T 2f29bba45e1749eb96b08da5e15fb834 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:22.454182 18805 leader_election.cc:288] T 2f29bba45e1749eb96b08da5e15fb834 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Requested vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:53:22.454548 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f29bba45e1749eb96b08da5e15fb834" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:53:22.454648 17866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f29bba45e1749eb96b08da5e15fb834" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:53:22.454721 17737 raft_consensus.cc:3037] T 2f29bba45e1749eb96b08da5e15fb834 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:22.454860 17866 raft_consensus.cc:3037] T 2f29bba45e1749eb96b08da5e15fb834 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:22.456498 17737 raft_consensus.cc:2454] T 2f29bba45e1749eb96b08da5e15fb834 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
I20240502 01:53:22.456616 17866 raft_consensus.cc:2454] T 2f29bba45e1749eb96b08da5e15fb834 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
I20240502 01:53:22.456946 17411 leader_election.cc:302] T 2f29bba45e1749eb96b08da5e15fb834 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:22.457262 18805 raft_consensus.cc:2781] T 2f29bba45e1749eb96b08da5e15fb834 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:22.457598 18805 raft_consensus.cc:686] T 2f29bba45e1749eb96b08da5e15fb834 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Becoming Leader. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Running, Role: LEADER
I20240502 01:53:22.457895 18805 consensus_queue.cc:237] T 2f29bba45e1749eb96b08da5e15fb834 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:22.460310 17335 catalog_manager.cc:5484] T 2f29bba45e1749eb96b08da5e15fb834 P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: term changed from 0 to 1, leader changed from <none> to 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65). New cstate: current_term: 1 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:22.565215 18805 raft_consensus.cc:482] T 2f3528ac3c114a19892bba2304474302 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:22.565409 18805 raft_consensus.cc:504] T 2f3528ac3c114a19892bba2304474302 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:22.565990 18805 leader_election.cc:288] T 2f3528ac3c114a19892bba2304474302 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:53:22.566507 17866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f3528ac3c114a19892bba2304474302" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:53:22.566555 17607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f3528ac3c114a19892bba2304474302" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:53:22.566767 17607 raft_consensus.cc:2454] T 2f3528ac3c114a19892bba2304474302 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 0.
I20240502 01:53:22.566768 17866 raft_consensus.cc:2454] T 2f3528ac3c114a19892bba2304474302 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 0.
I20240502 01:53:22.567178 17409 leader_election.cc:302] T 2f3528ac3c114a19892bba2304474302 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, 4da690c0fa13407a85c0689ce8de9ade; no voters:
I20240502 01:53:22.567492 18805 raft_consensus.cc:2781] T 2f3528ac3c114a19892bba2304474302 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:22.567581 18805 raft_consensus.cc:482] T 2f3528ac3c114a19892bba2304474302 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:22.567680 18805 raft_consensus.cc:3037] T 2f3528ac3c114a19892bba2304474302 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:22.569602 18805 raft_consensus.cc:504] T 2f3528ac3c114a19892bba2304474302 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:22.570158 18805 leader_election.cc:288] T 2f3528ac3c114a19892bba2304474302 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Requested vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:53:22.570436 17607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f3528ac3c114a19892bba2304474302" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:53:22.570611 17607 raft_consensus.cc:3037] T 2f3528ac3c114a19892bba2304474302 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:22.570665 17866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f3528ac3c114a19892bba2304474302" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:53:22.570847 17866 raft_consensus.cc:3037] T 2f3528ac3c114a19892bba2304474302 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:22.572576 17607 raft_consensus.cc:2454] T 2f3528ac3c114a19892bba2304474302 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
I20240502 01:53:22.572780 17866 raft_consensus.cc:2454] T 2f3528ac3c114a19892bba2304474302 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
I20240502 01:53:22.572949 17409 leader_election.cc:302] T 2f3528ac3c114a19892bba2304474302 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, 4da690c0fa13407a85c0689ce8de9ade; no voters:
I20240502 01:53:22.573228 18805 raft_consensus.cc:2781] T 2f3528ac3c114a19892bba2304474302 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:22.573397 18805 raft_consensus.cc:686] T 2f3528ac3c114a19892bba2304474302 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Becoming Leader. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Running, Role: LEADER
I20240502 01:53:22.573647 18805 consensus_queue.cc:237] T 2f3528ac3c114a19892bba2304474302 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:22.576342 17335 catalog_manager.cc:5484] T 2f3528ac3c114a19892bba2304474302 P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: term changed from 0 to 1, leader changed from <none> to 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65). New cstate: current_term: 1 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:22.605576 18496 raft_consensus.cc:482] T ae24680ad75545d892c9b79833319214 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:22.605739 18496 raft_consensus.cc:504] T ae24680ad75545d892c9b79833319214 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:22.606384 18496 leader_election.cc:288] T ae24680ad75545d892c9b79833319214 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:22.606773 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae24680ad75545d892c9b79833319214" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:22.606791 17607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae24680ad75545d892c9b79833319214" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:53:22.607023 17607 raft_consensus.cc:2454] T ae24680ad75545d892c9b79833319214 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:53:22.607020 17475 raft_consensus.cc:2454] T ae24680ad75545d892c9b79833319214 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:53:22.607437 17672 leader_election.cc:302] T ae24680ad75545d892c9b79833319214 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:22.607695 18496 raft_consensus.cc:2781] T ae24680ad75545d892c9b79833319214 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:22.607849 18496 raft_consensus.cc:482] T ae24680ad75545d892c9b79833319214 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:22.607933 18496 raft_consensus.cc:3037] T ae24680ad75545d892c9b79833319214 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:22.609788 18496 raft_consensus.cc:504] T ae24680ad75545d892c9b79833319214 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:22.610292 18496 leader_election.cc:288] T ae24680ad75545d892c9b79833319214 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Requested vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:22.610615 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae24680ad75545d892c9b79833319214" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:53:22.610702 17607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae24680ad75545d892c9b79833319214" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:53:22.610800 17475 raft_consensus.cc:3037] T ae24680ad75545d892c9b79833319214 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:22.610900 17607 raft_consensus.cc:3037] T ae24680ad75545d892c9b79833319214 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:22.612617 17475 raft_consensus.cc:2454] T ae24680ad75545d892c9b79833319214 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:53:22.612838 17607 raft_consensus.cc:2454] T ae24680ad75545d892c9b79833319214 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:53:22.612978 17672 leader_election.cc:302] T ae24680ad75545d892c9b79833319214 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:22.613269 18496 raft_consensus.cc:2781] T ae24680ad75545d892c9b79833319214 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:22.613618 18496 raft_consensus.cc:686] T ae24680ad75545d892c9b79833319214 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Becoming Leader. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Running, Role: LEADER
I20240502 01:53:22.613876 18496 consensus_queue.cc:237] T ae24680ad75545d892c9b79833319214 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:22.616236 17335 catalog_manager.cc:5484] T ae24680ad75545d892c9b79833319214 P dfa1152987644bd781b46c7e42a7a685 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfa1152987644bd781b46c7e42a7a685 (127.11.42.67). New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:22.935691 18805 consensus_queue.cc:1035] T 2f29bba45e1749eb96b08da5e15fb834 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:22.941547 18832 consensus_queue.cc:1035] T 2f29bba45e1749eb96b08da5e15fb834 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:22.980430 18837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:22.980708 18837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:22.988507 18837 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240502 01:53:23.081632 18832 consensus_queue.cc:1035] T 2f3528ac3c114a19892bba2304474302 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:23.099539 18805 consensus_queue.cc:1035] T 2f3528ac3c114a19892bba2304474302 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:23.158087 17606 raft_consensus.cc:1261] T ae24680ad75545d892c9b79833319214 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:23.158090 17475 raft_consensus.cc:1261] T ae24680ad75545d892c9b79833319214 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:23.158699 18496 consensus_queue.cc:1035] T ae24680ad75545d892c9b79833319214 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:23.159220 18496 consensus_queue.cc:1035] T ae24680ad75545d892c9b79833319214 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:23.306818 18880 mvcc.cc:205] Tried to move back new op lower bound from 7023062233729130496 to 7023062231508357120. Current Snapshot: MvccSnapshot[applied={T|T < 7023062233729130496}]
I20240502 01:53:23.327801 18890 mvcc.cc:205] Tried to move back new op lower bound from 7023062234379956224 to 7023062231508357120. Current Snapshot: MvccSnapshot[applied={T|T < 7023062234379956224}]
INSERT report
rows total: 2000
time total: 306.806 ms
time per row: 0.153403 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 01:53:24.310392 17335 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:24.319367 17586 tablet_service.cc:1467] Processing CreateTablet for tablet 7c697210a14d4982a939ea7965cb769d (DEFAULT_TABLE table=rebalance_test_table_5 [id=1567095c8de34d6a98c56d3bb926b668]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:24.319871 17586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c697210a14d4982a939ea7965cb769d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:24.320227 17587 tablet_service.cc:1467] Processing CreateTablet for tablet 377f8b554ad1448d981d2283b865f8fe (DEFAULT_TABLE table=rebalance_test_table_5 [id=1567095c8de34d6a98c56d3bb926b668]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:24.320233 17847 tablet_service.cc:1467] Processing CreateTablet for tablet 7c697210a14d4982a939ea7965cb769d (DEFAULT_TABLE table=rebalance_test_table_5 [id=1567095c8de34d6a98c56d3bb926b668]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:24.320626 17847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c697210a14d4982a939ea7965cb769d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:24.320621 17587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 377f8b554ad1448d981d2283b865f8fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:24.321945 17715 tablet_service.cc:1467] Processing CreateTablet for tablet 377f8b554ad1448d981d2283b865f8fe (DEFAULT_TABLE table=rebalance_test_table_5 [id=1567095c8de34d6a98c56d3bb926b668]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:24.322357 17715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 377f8b554ad1448d981d2283b865f8fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:24.325433 17455 tablet_service.cc:1467] Processing CreateTablet for tablet 5152c002eadd4d54ab10a093bce45f17 (DEFAULT_TABLE table=rebalance_test_table_5 [id=1567095c8de34d6a98c56d3bb926b668]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:24.325857 17455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5152c002eadd4d54ab10a093bce45f17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:24.326051 17716 tablet_service.cc:1467] Processing CreateTablet for tablet 7c697210a14d4982a939ea7965cb769d (DEFAULT_TABLE table=rebalance_test_table_5 [id=1567095c8de34d6a98c56d3bb926b668]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:24.326431 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c697210a14d4982a939ea7965cb769d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:24.329448 17585 tablet_service.cc:1467] Processing CreateTablet for tablet 5152c002eadd4d54ab10a093bce45f17 (DEFAULT_TABLE table=rebalance_test_table_5 [id=1567095c8de34d6a98c56d3bb926b668]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:24.329834 17585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5152c002eadd4d54ab10a093bce45f17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:24.336599 17845 tablet_service.cc:1467] Processing CreateTablet for tablet 377f8b554ad1448d981d2283b865f8fe (DEFAULT_TABLE table=rebalance_test_table_5 [id=1567095c8de34d6a98c56d3bb926b668]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:24.337033 17845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 377f8b554ad1448d981d2283b865f8fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:24.337056 18903 tablet_bootstrap.cc:492] T 377f8b554ad1448d981d2283b865f8fe P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:24.338775 18903 tablet_bootstrap.cc:654] T 377f8b554ad1448d981d2283b865f8fe P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:24.341003 18903 tablet_bootstrap.cc:492] T 377f8b554ad1448d981d2283b865f8fe P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:24.341190 18903 ts_tablet_manager.cc:1367] T 377f8b554ad1448d981d2283b865f8fe P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240502 01:53:24.341341 17846 tablet_service.cc:1467] Processing CreateTablet for tablet 5152c002eadd4d54ab10a093bce45f17 (DEFAULT_TABLE table=rebalance_test_table_5 [id=1567095c8de34d6a98c56d3bb926b668]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:24.341670 17846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5152c002eadd4d54ab10a093bce45f17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:24.341948 18903 raft_consensus.cc:348] T 377f8b554ad1448d981d2283b865f8fe P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:24.342165 18903 raft_consensus.cc:374] T 377f8b554ad1448d981d2283b865f8fe P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:24.342252 18903 raft_consensus.cc:729] T 377f8b554ad1448d981d2283b865f8fe P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:24.342509 18903 consensus_queue.cc:260] T 377f8b554ad1448d981d2283b865f8fe P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:24.343173 18903 ts_tablet_manager.cc:1398] T 377f8b554ad1448d981d2283b865f8fe P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:24.343442 18903 tablet_bootstrap.cc:492] T 7c697210a14d4982a939ea7965cb769d P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:24.344945 18903 tablet_bootstrap.cc:654] T 7c697210a14d4982a939ea7965cb769d P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:24.348001 18906 tablet_bootstrap.cc:492] T 5152c002eadd4d54ab10a093bce45f17 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:24.349660 18906 tablet_bootstrap.cc:654] T 5152c002eadd4d54ab10a093bce45f17 P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:24.354023 18906 tablet_bootstrap.cc:492] T 5152c002eadd4d54ab10a093bce45f17 P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:24.354856 18903 tablet_bootstrap.cc:492] T 7c697210a14d4982a939ea7965cb769d P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:24.354954 18906 ts_tablet_manager.cc:1367] T 5152c002eadd4d54ab10a093bce45f17 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20240502 01:53:24.355562 18903 ts_tablet_manager.cc:1367] T 7c697210a14d4982a939ea7965cb769d P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20240502 01:53:24.356345 18903 raft_consensus.cc:348] T 7c697210a14d4982a939ea7965cb769d P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:24.356581 18903 raft_consensus.cc:374] T 7c697210a14d4982a939ea7965cb769d P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:24.356657 18903 raft_consensus.cc:729] T 7c697210a14d4982a939ea7965cb769d P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:24.355655 18906 raft_consensus.cc:348] T 5152c002eadd4d54ab10a093bce45f17 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:24.356799 18906 raft_consensus.cc:374] T 5152c002eadd4d54ab10a093bce45f17 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:24.356896 18906 raft_consensus.cc:729] T 5152c002eadd4d54ab10a093bce45f17 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:24.356902 18903 consensus_queue.cc:260] T 7c697210a14d4982a939ea7965cb769d P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:24.357479 18903 ts_tablet_manager.cc:1398] T 7c697210a14d4982a939ea7965cb769d P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:24.357532 18906 consensus_queue.cc:260] T 5152c002eadd4d54ab10a093bce45f17 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:24.357668 18908 tablet_bootstrap.cc:492] T 5152c002eadd4d54ab10a093bce45f17 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:24.358071 18906 ts_tablet_manager.cc:1398] T 5152c002eadd4d54ab10a093bce45f17 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:53:24.359036 18906 tablet_bootstrap.cc:492] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:24.359645 18908 tablet_bootstrap.cc:654] T 5152c002eadd4d54ab10a093bce45f17 P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:24.361061 18902 tablet_bootstrap.cc:492] T 7c697210a14d4982a939ea7965cb769d P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:24.361164 18906 tablet_bootstrap.cc:654] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:24.361460 18908 tablet_bootstrap.cc:492] T 5152c002eadd4d54ab10a093bce45f17 P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:24.361649 18908 ts_tablet_manager.cc:1367] T 5152c002eadd4d54ab10a093bce45f17 P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:53:24.362413 18908 raft_consensus.cc:348] T 5152c002eadd4d54ab10a093bce45f17 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:24.362682 18902 tablet_bootstrap.cc:654] T 7c697210a14d4982a939ea7965cb769d P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:24.364331 18908 raft_consensus.cc:374] T 5152c002eadd4d54ab10a093bce45f17 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:24.364449 18908 raft_consensus.cc:729] T 5152c002eadd4d54ab10a093bce45f17 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:24.364686 18908 consensus_queue.cc:260] T 5152c002eadd4d54ab10a093bce45f17 P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:24.365092 18906 tablet_bootstrap.cc:492] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:24.365231 18906 ts_tablet_manager.cc:1367] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20240502 01:53:24.365760 18906 raft_consensus.cc:348] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:24.365985 18906 raft_consensus.cc:374] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:24.366082 18906 raft_consensus.cc:729] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:24.366089 18908 ts_tablet_manager.cc:1398] T 5152c002eadd4d54ab10a093bce45f17 P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:53:24.366340 18906 consensus_queue.cc:260] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:24.366881 18906 ts_tablet_manager.cc:1398] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:24.367187 18906 tablet_bootstrap.cc:492] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:24.368523 18906 tablet_bootstrap.cc:654] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:24.370362 18906 tablet_bootstrap.cc:492] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:24.370544 18906 ts_tablet_manager.cc:1367] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 01:53:24.370977 18902 tablet_bootstrap.cc:492] T 7c697210a14d4982a939ea7965cb769d P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:24.371194 18902 ts_tablet_manager.cc:1367] T 7c697210a14d4982a939ea7965cb769d P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20240502 01:53:24.371363 18906 raft_consensus.cc:348] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:24.371542 18906 raft_consensus.cc:374] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:24.371634 18906 raft_consensus.cc:729] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:24.371834 18906 consensus_queue.cc:260] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:24.371969 18902 raft_consensus.cc:348] T 7c697210a14d4982a939ea7965cb769d P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:24.372180 18902 raft_consensus.cc:374] T 7c697210a14d4982a939ea7965cb769d P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:24.372282 18902 raft_consensus.cc:729] T 7c697210a14d4982a939ea7965cb769d P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:24.372352 18906 ts_tablet_manager.cc:1398] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:53:24.372506 18902 consensus_queue.cc:260] T 7c697210a14d4982a939ea7965cb769d P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:24.373044 18902 ts_tablet_manager.cc:1398] T 7c697210a14d4982a939ea7965cb769d P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:53:24.373371 18902 tablet_bootstrap.cc:492] T 377f8b554ad1448d981d2283b865f8fe P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:24.374809 18902 tablet_bootstrap.cc:654] T 377f8b554ad1448d981d2283b865f8fe P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:24.376698 18902 tablet_bootstrap.cc:492] T 377f8b554ad1448d981d2283b865f8fe P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:24.376875 18902 ts_tablet_manager.cc:1367] T 377f8b554ad1448d981d2283b865f8fe P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 01:53:24.377583 18902 raft_consensus.cc:348] T 377f8b554ad1448d981d2283b865f8fe P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:24.377732 18902 raft_consensus.cc:374] T 377f8b554ad1448d981d2283b865f8fe P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:24.377790 18902 raft_consensus.cc:729] T 377f8b554ad1448d981d2283b865f8fe P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:24.377972 18902 consensus_queue.cc:260] T 377f8b554ad1448d981d2283b865f8fe P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:24.378319 18902 ts_tablet_manager.cc:1398] T 377f8b554ad1448d981d2283b865f8fe P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:53:24.378587 18902 tablet_bootstrap.cc:492] T 5152c002eadd4d54ab10a093bce45f17 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:24.379878 18902 tablet_bootstrap.cc:654] T 5152c002eadd4d54ab10a093bce45f17 P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:24.381562 18902 tablet_bootstrap.cc:492] T 5152c002eadd4d54ab10a093bce45f17 P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:24.381731 18902 ts_tablet_manager.cc:1367] T 5152c002eadd4d54ab10a093bce45f17 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20240502 01:53:24.382412 18902 raft_consensus.cc:348] T 5152c002eadd4d54ab10a093bce45f17 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:24.382544 18902 raft_consensus.cc:374] T 5152c002eadd4d54ab10a093bce45f17 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:24.382606 18902 raft_consensus.cc:729] T 5152c002eadd4d54ab10a093bce45f17 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:24.382747 18902 consensus_queue.cc:260] T 5152c002eadd4d54ab10a093bce45f17 P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:24.383122 18902 ts_tablet_manager.cc:1398] T 5152c002eadd4d54ab10a093bce45f17 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:53:24.432256 18737 raft_consensus.cc:482] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:24.432446 18737 raft_consensus.cc:504] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:24.433090 18737 leader_election.cc:288] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:53:24.433583 17607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c697210a14d4982a939ea7965cb769d" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:53:24.433609 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c697210a14d4982a939ea7965cb769d" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:53:24.433787 17607 raft_consensus.cc:2454] T 7c697210a14d4982a939ea7965cb769d P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 0.
I20240502 01:53:24.433859 17737 raft_consensus.cc:2454] T 7c697210a14d4982a939ea7965cb769d P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 0.
I20240502 01:53:24.434216 17800 leader_election.cc:302] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:53:24.434533 18737 raft_consensus.cc:2781] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:24.434620 18737 raft_consensus.cc:482] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:24.434718 18737 raft_consensus.cc:3037] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:24.436571 18737 raft_consensus.cc:504] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:24.437139 18737 leader_election.cc:288] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 election: Requested vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:53:24.437485 17607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c697210a14d4982a939ea7965cb769d" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:53:24.437575 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c697210a14d4982a939ea7965cb769d" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:53:24.437700 17607 raft_consensus.cc:3037] T 7c697210a14d4982a939ea7965cb769d P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:24.437772 17737 raft_consensus.cc:3037] T 7c697210a14d4982a939ea7965cb769d P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:24.440397 17737 raft_consensus.cc:2454] T 7c697210a14d4982a939ea7965cb769d P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 1.
I20240502 01:53:24.440425 17607 raft_consensus.cc:2454] T 7c697210a14d4982a939ea7965cb769d P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 1.
I20240502 01:53:24.440771 17802 leader_election.cc:302] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb31bceba76d466db24b9341c04900c3, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:24.441247 18737 raft_consensus.cc:2781] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:24.441722 18737 raft_consensus.cc:686] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Becoming Leader. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Running, Role: LEADER
I20240502 01:53:24.442090 18737 consensus_queue.cc:237] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:24.445657 17331 catalog_manager.cc:5484] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb31bceba76d466db24b9341c04900c3 (127.11.42.68). New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } }
I20240502 01:53:24.546451 18737 raft_consensus.cc:482] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:24.546639 18737 raft_consensus.cc:504] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:24.547318 18737 leader_election.cc:288] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:24.547756 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377f8b554ad1448d981d2283b865f8fe" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:53:24.547879 17607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377f8b554ad1448d981d2283b865f8fe" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:53:24.548005 17737 raft_consensus.cc:2454] T 377f8b554ad1448d981d2283b865f8fe P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 0.
I20240502 01:53:24.548121 17607 raft_consensus.cc:2454] T 377f8b554ad1448d981d2283b865f8fe P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 0.
I20240502 01:53:24.548485 17802 leader_election.cc:302] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb31bceba76d466db24b9341c04900c3, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:24.548758 18737 raft_consensus.cc:2781] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:24.548887 18737 raft_consensus.cc:482] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:24.548966 18737 raft_consensus.cc:3037] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:24.551746 18737 raft_consensus.cc:504] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:24.552532 18737 leader_election.cc:288] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 election: Requested vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:24.552717 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377f8b554ad1448d981d2283b865f8fe" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:53:24.552806 17607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377f8b554ad1448d981d2283b865f8fe" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:53:24.552923 17737 raft_consensus.cc:3037] T 377f8b554ad1448d981d2283b865f8fe P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:24.552991 17607 raft_consensus.cc:3037] T 377f8b554ad1448d981d2283b865f8fe P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:24.555728 17737 raft_consensus.cc:2454] T 377f8b554ad1448d981d2283b865f8fe P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 1.
I20240502 01:53:24.555722 17607 raft_consensus.cc:2454] T 377f8b554ad1448d981d2283b865f8fe P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 1.
I20240502 01:53:24.556151 17800 leader_election.cc:302] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, bb31bceba76d466db24b9341c04900c3, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:24.556466 18737 raft_consensus.cc:2781] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:24.556596 18737 raft_consensus.cc:686] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Becoming Leader. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Running, Role: LEADER
I20240502 01:53:24.556859 18737 consensus_queue.cc:237] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:24.559634 17331 catalog_manager.cc:5484] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb31bceba76d466db24b9341c04900c3 (127.11.42.68). New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } }
I20240502 01:53:24.595777 18801 raft_consensus.cc:482] T 5152c002eadd4d54ab10a093bce45f17 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:24.595955 18801 raft_consensus.cc:504] T 5152c002eadd4d54ab10a093bce45f17 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:24.596578 18801 leader_election.cc:288] T 5152c002eadd4d54ab10a093bce45f17 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:53:24.597076 17866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5152c002eadd4d54ab10a093bce45f17" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:53:24.597103 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5152c002eadd4d54ab10a093bce45f17" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:24.597309 17866 raft_consensus.cc:2454] T 5152c002eadd4d54ab10a093bce45f17 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:53:24.597380 17475 raft_consensus.cc:2454] T 5152c002eadd4d54ab10a093bce45f17 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:53:24.597764 17543 leader_election.cc:302] T 5152c002eadd4d54ab10a093bce45f17 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:53:24.598039 18801 raft_consensus.cc:2781] T 5152c002eadd4d54ab10a093bce45f17 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:24.598155 18801 raft_consensus.cc:482] T 5152c002eadd4d54ab10a093bce45f17 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:24.598229 18801 raft_consensus.cc:3037] T 5152c002eadd4d54ab10a093bce45f17 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:24.600039 18801 raft_consensus.cc:504] T 5152c002eadd4d54ab10a093bce45f17 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:24.600540 18801 leader_election.cc:288] T 5152c002eadd4d54ab10a093bce45f17 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Requested vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:53:24.600965 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5152c002eadd4d54ab10a093bce45f17" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:53:24.600984 17866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5152c002eadd4d54ab10a093bce45f17" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:53:24.601176 17475 raft_consensus.cc:3037] T 5152c002eadd4d54ab10a093bce45f17 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:24.601186 17866 raft_consensus.cc:3037] T 5152c002eadd4d54ab10a093bce45f17 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:24.602947 17866 raft_consensus.cc:2454] T 5152c002eadd4d54ab10a093bce45f17 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:53:24.602986 17475 raft_consensus.cc:2454] T 5152c002eadd4d54ab10a093bce45f17 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:53:24.603284 17543 leader_election.cc:302] T 5152c002eadd4d54ab10a093bce45f17 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:53:24.603591 18801 raft_consensus.cc:2781] T 5152c002eadd4d54ab10a093bce45f17 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:24.603946 18801 raft_consensus.cc:686] T 5152c002eadd4d54ab10a093bce45f17 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Becoming Leader. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Running, Role: LEADER
I20240502 01:53:24.604214 18801 consensus_queue.cc:237] T 5152c002eadd4d54ab10a093bce45f17 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:24.606918 17331 catalog_manager.cc:5484] T 5152c002eadd4d54ab10a093bce45f17 P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66). New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } }
W20240502 01:53:24.958920 18921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:24.959213 18921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 01:53:24.964097 18737 consensus_queue.cc:1035] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:24.968040 18921 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:53:24.970220 18916 consensus_queue.cc:1035] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_5'
I20240502 01:53:25.023933 18801 consensus_queue.cc:1035] T 5152c002eadd4d54ab10a093bce45f17 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:25.031311 18919 consensus_queue.cc:1035] T 5152c002eadd4d54ab10a093bce45f17 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:25.098695 18916 consensus_queue.cc:1035] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:25.106870 17735 raft_consensus.cc:1261] T 377f8b554ad1448d981d2283b865f8fe P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:25.107847 18737 consensus_queue.cc:1035] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:25.169165 17542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.42.65:38321, user_credentials={real_user=slave}} blocked reactor thread for 61243.5us
INSERT report
rows total: 2000
time total: 252.351 ms
time per row: 0.126176 ms
I20240502 01:53:25.360102 17326 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:25.369431 17455 tablet_service.cc:1467] Processing CreateTablet for tablet 4280160c9b3a4b34b3c7d89f0fbec666 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9631b34b6ef449edb5476c988aed9d7a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:25.369442 17586 tablet_service.cc:1467] Processing CreateTablet for tablet 4280160c9b3a4b34b3c7d89f0fbec666 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9631b34b6ef449edb5476c988aed9d7a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:25.369558 17716 tablet_service.cc:1467] Processing CreateTablet for tablet 34038775b5fa4b3193b8e57f09102ea3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9631b34b6ef449edb5476c988aed9d7a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:25.369896 17586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4280160c9b3a4b34b3c7d89f0fbec666. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:25.369901 17455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4280160c9b3a4b34b3c7d89f0fbec666. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:25.369951 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34038775b5fa4b3193b8e57f09102ea3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:25.370962 17454 tablet_service.cc:1467] Processing CreateTablet for tablet 34038775b5fa4b3193b8e57f09102ea3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9631b34b6ef449edb5476c988aed9d7a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:25.371327 17454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34038775b5fa4b3193b8e57f09102ea3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:25.368896 17715 tablet_service.cc:1467] Processing CreateTablet for tablet 4280160c9b3a4b34b3c7d89f0fbec666 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9631b34b6ef449edb5476c988aed9d7a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:25.373442 17715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4280160c9b3a4b34b3c7d89f0fbec666. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:25.376585 17585 tablet_service.cc:1467] Processing CreateTablet for tablet e00020f7f47f4246907a8c90fca4a26b (DEFAULT_TABLE table=rebalance_test_table_6 [id=9631b34b6ef449edb5476c988aed9d7a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:25.376937 17585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e00020f7f47f4246907a8c90fca4a26b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:25.378127 17847 tablet_service.cc:1467] Processing CreateTablet for tablet 34038775b5fa4b3193b8e57f09102ea3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9631b34b6ef449edb5476c988aed9d7a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:25.378531 18988 tablet_bootstrap.cc:492] T 34038775b5fa4b3193b8e57f09102ea3 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:25.378551 17847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34038775b5fa4b3193b8e57f09102ea3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:25.380141 18987 tablet_bootstrap.cc:492] T 4280160c9b3a4b34b3c7d89f0fbec666 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:25.380438 17717 tablet_service.cc:1467] Processing CreateTablet for tablet e00020f7f47f4246907a8c90fca4a26b (DEFAULT_TABLE table=rebalance_test_table_6 [id=9631b34b6ef449edb5476c988aed9d7a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:25.380765 17717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e00020f7f47f4246907a8c90fca4a26b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:25.381683 18987 tablet_bootstrap.cc:654] T 4280160c9b3a4b34b3c7d89f0fbec666 P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:25.381624 17456 tablet_service.cc:1467] Processing CreateTablet for tablet e00020f7f47f4246907a8c90fca4a26b (DEFAULT_TABLE table=rebalance_test_table_6 [id=9631b34b6ef449edb5476c988aed9d7a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:25.381984 17456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e00020f7f47f4246907a8c90fca4a26b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:25.388186 18987 tablet_bootstrap.cc:492] T 4280160c9b3a4b34b3c7d89f0fbec666 P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:25.388355 18990 tablet_bootstrap.cc:492] T 34038775b5fa4b3193b8e57f09102ea3 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:25.388430 18987 ts_tablet_manager.cc:1367] T 4280160c9b3a4b34b3c7d89f0fbec666 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20240502 01:53:25.389287 18987 raft_consensus.cc:348] T 4280160c9b3a4b34b3c7d89f0fbec666 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:25.389518 18987 raft_consensus.cc:374] T 4280160c9b3a4b34b3c7d89f0fbec666 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:25.389636 18987 raft_consensus.cc:729] T 4280160c9b3a4b34b3c7d89f0fbec666 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:25.389895 18987 consensus_queue.cc:260] T 4280160c9b3a4b34b3c7d89f0fbec666 P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:25.390563 18987 ts_tablet_manager.cc:1398] T 4280160c9b3a4b34b3c7d89f0fbec666 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:25.390841 18991 tablet_bootstrap.cc:492] T 34038775b5fa4b3193b8e57f09102ea3 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:25.390885 18987 tablet_bootstrap.cc:492] T e00020f7f47f4246907a8c90fca4a26b P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:25.392148 18988 tablet_bootstrap.cc:654] T 34038775b5fa4b3193b8e57f09102ea3 P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:25.392433 18987 tablet_bootstrap.cc:654] T e00020f7f47f4246907a8c90fca4a26b P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:25.395226 18990 tablet_bootstrap.cc:654] T 34038775b5fa4b3193b8e57f09102ea3 P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:25.397143 18991 tablet_bootstrap.cc:654] T 34038775b5fa4b3193b8e57f09102ea3 P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:25.399195 18990 tablet_bootstrap.cc:492] T 34038775b5fa4b3193b8e57f09102ea3 P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:25.399396 18990 ts_tablet_manager.cc:1367] T 34038775b5fa4b3193b8e57f09102ea3 P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.004s
I20240502 01:53:25.400384 18990 raft_consensus.cc:348] T 34038775b5fa4b3193b8e57f09102ea3 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:25.400600 18990 raft_consensus.cc:374] T 34038775b5fa4b3193b8e57f09102ea3 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:25.400732 18990 raft_consensus.cc:729] T 34038775b5fa4b3193b8e57f09102ea3 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:25.400976 18990 consensus_queue.cc:260] T 34038775b5fa4b3193b8e57f09102ea3 P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:25.401757 18990 ts_tablet_manager.cc:1398] T 34038775b5fa4b3193b8e57f09102ea3 P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:25.402052 18990 tablet_bootstrap.cc:492] T e00020f7f47f4246907a8c90fca4a26b P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:25.402113 18988 tablet_bootstrap.cc:492] T 34038775b5fa4b3193b8e57f09102ea3 P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:25.402292 18988 ts_tablet_manager.cc:1367] T 34038775b5fa4b3193b8e57f09102ea3 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.004s
I20240502 01:53:25.402487 18991 tablet_bootstrap.cc:492] T 34038775b5fa4b3193b8e57f09102ea3 P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:25.402655 18991 ts_tablet_manager.cc:1367] T 34038775b5fa4b3193b8e57f09102ea3 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240502 01:53:25.403082 18988 raft_consensus.cc:348] T 34038775b5fa4b3193b8e57f09102ea3 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:25.403316 18988 raft_consensus.cc:374] T 34038775b5fa4b3193b8e57f09102ea3 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:25.403435 18988 raft_consensus.cc:729] T 34038775b5fa4b3193b8e57f09102ea3 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:25.403491 18991 raft_consensus.cc:348] T 34038775b5fa4b3193b8e57f09102ea3 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:25.403739 18991 raft_consensus.cc:374] T 34038775b5fa4b3193b8e57f09102ea3 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:25.403878 18991 raft_consensus.cc:729] T 34038775b5fa4b3193b8e57f09102ea3 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:25.403899 18988 consensus_queue.cc:260] T 34038775b5fa4b3193b8e57f09102ea3 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:25.404302 18991 consensus_queue.cc:260] T 34038775b5fa4b3193b8e57f09102ea3 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:25.404999 18991 ts_tablet_manager.cc:1398] T 34038775b5fa4b3193b8e57f09102ea3 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:25.405334 18988 ts_tablet_manager.cc:1398] T 34038775b5fa4b3193b8e57f09102ea3 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:53:25.405619 18988 tablet_bootstrap.cc:492] T e00020f7f47f4246907a8c90fca4a26b P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:25.406538 18987 tablet_bootstrap.cc:492] T e00020f7f47f4246907a8c90fca4a26b P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:25.406747 18987 ts_tablet_manager.cc:1367] T e00020f7f47f4246907a8c90fca4a26b P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.003s
I20240502 01:53:25.407238 18988 tablet_bootstrap.cc:654] T e00020f7f47f4246907a8c90fca4a26b P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:25.407620 18987 raft_consensus.cc:348] T e00020f7f47f4246907a8c90fca4a26b P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:25.407882 18987 raft_consensus.cc:374] T e00020f7f47f4246907a8c90fca4a26b P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:25.408005 18987 raft_consensus.cc:729] T e00020f7f47f4246907a8c90fca4a26b P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:25.408368 18987 consensus_queue.cc:260] T e00020f7f47f4246907a8c90fca4a26b P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:25.408586 18990 tablet_bootstrap.cc:654] T e00020f7f47f4246907a8c90fca4a26b P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:25.409677 18988 tablet_bootstrap.cc:492] T e00020f7f47f4246907a8c90fca4a26b P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:25.409744 18987 ts_tablet_manager.cc:1398] T e00020f7f47f4246907a8c90fca4a26b P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:53:25.409873 18988 ts_tablet_manager.cc:1367] T e00020f7f47f4246907a8c90fca4a26b P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.002s
I20240502 01:53:25.410637 18988 raft_consensus.cc:348] T e00020f7f47f4246907a8c90fca4a26b P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:25.410866 18988 raft_consensus.cc:374] T e00020f7f47f4246907a8c90fca4a26b P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:25.410955 18988 raft_consensus.cc:729] T e00020f7f47f4246907a8c90fca4a26b P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:25.411183 18988 consensus_queue.cc:260] T e00020f7f47f4246907a8c90fca4a26b P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:25.411805 18988 ts_tablet_manager.cc:1398] T e00020f7f47f4246907a8c90fca4a26b P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:53:25.412096 18988 tablet_bootstrap.cc:492] T 4280160c9b3a4b34b3c7d89f0fbec666 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:25.413645 18988 tablet_bootstrap.cc:654] T 4280160c9b3a4b34b3c7d89f0fbec666 P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:25.417665 18990 tablet_bootstrap.cc:492] T e00020f7f47f4246907a8c90fca4a26b P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:25.417819 18990 ts_tablet_manager.cc:1367] T e00020f7f47f4246907a8c90fca4a26b P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.006s
I20240502 01:53:25.417870 18988 tablet_bootstrap.cc:492] T 4280160c9b3a4b34b3c7d89f0fbec666 P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:25.418001 18988 ts_tablet_manager.cc:1367] T 4280160c9b3a4b34b3c7d89f0fbec666 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240502 01:53:25.418434 18990 raft_consensus.cc:348] T e00020f7f47f4246907a8c90fca4a26b P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:25.418658 18990 raft_consensus.cc:374] T e00020f7f47f4246907a8c90fca4a26b P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:25.418742 18990 raft_consensus.cc:729] T e00020f7f47f4246907a8c90fca4a26b P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:25.418778 18988 raft_consensus.cc:348] T 4280160c9b3a4b34b3c7d89f0fbec666 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:25.419016 18988 raft_consensus.cc:374] T 4280160c9b3a4b34b3c7d89f0fbec666 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:25.418972 18990 consensus_queue.cc:260] T e00020f7f47f4246907a8c90fca4a26b P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:25.419104 18988 raft_consensus.cc:729] T 4280160c9b3a4b34b3c7d89f0fbec666 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:25.419310 18988 consensus_queue.cc:260] T 4280160c9b3a4b34b3c7d89f0fbec666 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:25.419533 18990 ts_tablet_manager.cc:1398] T e00020f7f47f4246907a8c90fca4a26b P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:53:25.419754 18988 ts_tablet_manager.cc:1398] T 4280160c9b3a4b34b3c7d89f0fbec666 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:25.419881 18990 tablet_bootstrap.cc:492] T 4280160c9b3a4b34b3c7d89f0fbec666 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:25.421317 18990 tablet_bootstrap.cc:654] T 4280160c9b3a4b34b3c7d89f0fbec666 P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:25.423041 18990 tablet_bootstrap.cc:492] T 4280160c9b3a4b34b3c7d89f0fbec666 P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:25.423177 18990 ts_tablet_manager.cc:1367] T 4280160c9b3a4b34b3c7d89f0fbec666 P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 01:53:25.423908 18990 raft_consensus.cc:348] T 4280160c9b3a4b34b3c7d89f0fbec666 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:25.424154 18990 raft_consensus.cc:374] T 4280160c9b3a4b34b3c7d89f0fbec666 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:25.424264 18990 raft_consensus.cc:729] T 4280160c9b3a4b34b3c7d89f0fbec666 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:25.424458 18990 consensus_queue.cc:260] T 4280160c9b3a4b34b3c7d89f0fbec666 P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:25.424927 18990 ts_tablet_manager.cc:1398] T 4280160c9b3a4b34b3c7d89f0fbec666 P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:25.432837 18915 raft_consensus.cc:482] T e00020f7f47f4246907a8c90fca4a26b P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:25.432969 18915 raft_consensus.cc:504] T e00020f7f47f4246907a8c90fca4a26b P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:25.433642 18915 leader_election.cc:288] T e00020f7f47f4246907a8c90fca4a26b P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:25.434020 17607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e00020f7f47f4246907a8c90fca4a26b" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:53:25.434108 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e00020f7f47f4246907a8c90fca4a26b" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:25.434229 17607 raft_consensus.cc:2454] T e00020f7f47f4246907a8c90fca4a26b P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:53:25.434298 17475 raft_consensus.cc:2454] T e00020f7f47f4246907a8c90fca4a26b P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:53:25.434623 17670 leader_election.cc:302] T e00020f7f47f4246907a8c90fca4a26b P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:25.434868 18915 raft_consensus.cc:2781] T e00020f7f47f4246907a8c90fca4a26b P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:25.434980 18915 raft_consensus.cc:482] T e00020f7f47f4246907a8c90fca4a26b P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:25.435071 18915 raft_consensus.cc:3037] T e00020f7f47f4246907a8c90fca4a26b P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:25.436992 18915 raft_consensus.cc:504] T e00020f7f47f4246907a8c90fca4a26b P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:25.437578 18915 leader_election.cc:288] T e00020f7f47f4246907a8c90fca4a26b P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Requested vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:25.437991 17607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e00020f7f47f4246907a8c90fca4a26b" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:53:25.437976 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e00020f7f47f4246907a8c90fca4a26b" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:53:25.438156 17475 raft_consensus.cc:3037] T e00020f7f47f4246907a8c90fca4a26b P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:25.438161 17607 raft_consensus.cc:3037] T e00020f7f47f4246907a8c90fca4a26b P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:25.440762 17607 raft_consensus.cc:2454] T e00020f7f47f4246907a8c90fca4a26b P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:53:25.440771 17475 raft_consensus.cc:2454] T e00020f7f47f4246907a8c90fca4a26b P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:53:25.441128 17672 leader_election.cc:302] T e00020f7f47f4246907a8c90fca4a26b P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:25.441381 18915 raft_consensus.cc:2781] T e00020f7f47f4246907a8c90fca4a26b P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:25.441763 18915 raft_consensus.cc:686] T e00020f7f47f4246907a8c90fca4a26b P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Becoming Leader. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Running, Role: LEADER
I20240502 01:53:25.442028 18915 consensus_queue.cc:237] T e00020f7f47f4246907a8c90fca4a26b P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:25.444475 17326 catalog_manager.cc:5484] T e00020f7f47f4246907a8c90fca4a26b P dfa1152987644bd781b46c7e42a7a685 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfa1152987644bd781b46c7e42a7a685 (127.11.42.67). New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:25.510515 18919 raft_consensus.cc:482] T 4280160c9b3a4b34b3c7d89f0fbec666 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:25.510686 18919 raft_consensus.cc:504] T 4280160c9b3a4b34b3c7d89f0fbec666 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:25.511312 18919 leader_election.cc:288] T 4280160c9b3a4b34b3c7d89f0fbec666 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:25.511734 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4280160c9b3a4b34b3c7d89f0fbec666" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:53:25.511876 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4280160c9b3a4b34b3c7d89f0fbec666" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:25.512019 17735 raft_consensus.cc:2454] T 4280160c9b3a4b34b3c7d89f0fbec666 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:53:25.512113 17475 raft_consensus.cc:2454] T 4280160c9b3a4b34b3c7d89f0fbec666 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:53:25.512460 17542 leader_election.cc:302] T 4280160c9b3a4b34b3c7d89f0fbec666 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:25.512712 18919 raft_consensus.cc:2781] T 4280160c9b3a4b34b3c7d89f0fbec666 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:25.512802 18919 raft_consensus.cc:482] T 4280160c9b3a4b34b3c7d89f0fbec666 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:25.512902 18919 raft_consensus.cc:3037] T 4280160c9b3a4b34b3c7d89f0fbec666 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:25.514743 18919 raft_consensus.cc:504] T 4280160c9b3a4b34b3c7d89f0fbec666 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:25.515241 18919 leader_election.cc:288] T 4280160c9b3a4b34b3c7d89f0fbec666 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Requested vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:25.515776 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4280160c9b3a4b34b3c7d89f0fbec666" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:53:25.515858 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4280160c9b3a4b34b3c7d89f0fbec666" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:53:25.516014 17735 raft_consensus.cc:3037] T 4280160c9b3a4b34b3c7d89f0fbec666 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:25.516065 17475 raft_consensus.cc:3037] T 4280160c9b3a4b34b3c7d89f0fbec666 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:25.518801 17475 raft_consensus.cc:2454] T 4280160c9b3a4b34b3c7d89f0fbec666 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:53:25.518801 17735 raft_consensus.cc:2454] T 4280160c9b3a4b34b3c7d89f0fbec666 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:53:25.519191 17542 leader_election.cc:302] T 4280160c9b3a4b34b3c7d89f0fbec666 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, 4da690c0fa13407a85c0689ce8de9ade; no voters:
I20240502 01:53:25.519412 18919 raft_consensus.cc:2781] T 4280160c9b3a4b34b3c7d89f0fbec666 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:25.519582 18919 raft_consensus.cc:686] T 4280160c9b3a4b34b3c7d89f0fbec666 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Becoming Leader. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Running, Role: LEADER
I20240502 01:53:25.519838 18919 consensus_queue.cc:237] T 4280160c9b3a4b34b3c7d89f0fbec666 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:25.522663 17326 catalog_manager.cc:5484] T 4280160c9b3a4b34b3c7d89f0fbec666 P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66). New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } }
I20240502 01:53:25.580047 18963 raft_consensus.cc:482] T 34038775b5fa4b3193b8e57f09102ea3 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:25.580216 18963 raft_consensus.cc:504] T 34038775b5fa4b3193b8e57f09102ea3 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:25.580830 18963 leader_election.cc:288] T 34038775b5fa4b3193b8e57f09102ea3 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:25.581334 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34038775b5fa4b3193b8e57f09102ea3" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:53:25.581419 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34038775b5fa4b3193b8e57f09102ea3" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:25.581542 17735 raft_consensus.cc:2454] T 34038775b5fa4b3193b8e57f09102ea3 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 0.
I20240502 01:53:25.581635 17475 raft_consensus.cc:2454] T 34038775b5fa4b3193b8e57f09102ea3 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 0.
I20240502 01:53:25.581905 17802 leader_election.cc:302] T 34038775b5fa4b3193b8e57f09102ea3 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb31bceba76d466db24b9341c04900c3, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:25.582152 18963 raft_consensus.cc:2781] T 34038775b5fa4b3193b8e57f09102ea3 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:25.582257 18963 raft_consensus.cc:482] T 34038775b5fa4b3193b8e57f09102ea3 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:25.582338 18963 raft_consensus.cc:3037] T 34038775b5fa4b3193b8e57f09102ea3 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:25.584188 18963 raft_consensus.cc:504] T 34038775b5fa4b3193b8e57f09102ea3 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:25.584669 18963 leader_election.cc:288] T 34038775b5fa4b3193b8e57f09102ea3 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 election: Requested vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:25.585103 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34038775b5fa4b3193b8e57f09102ea3" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:53:25.585198 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34038775b5fa4b3193b8e57f09102ea3" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:53:25.585350 17735 raft_consensus.cc:3037] T 34038775b5fa4b3193b8e57f09102ea3 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:25.585379 17475 raft_consensus.cc:3037] T 34038775b5fa4b3193b8e57f09102ea3 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:25.588017 17735 raft_consensus.cc:2454] T 34038775b5fa4b3193b8e57f09102ea3 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 1.
I20240502 01:53:25.588209 17475 raft_consensus.cc:2454] T 34038775b5fa4b3193b8e57f09102ea3 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 1.
I20240502 01:53:25.588434 17802 leader_election.cc:302] T 34038775b5fa4b3193b8e57f09102ea3 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb31bceba76d466db24b9341c04900c3, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:25.588762 18963 raft_consensus.cc:2781] T 34038775b5fa4b3193b8e57f09102ea3 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:25.588923 18963 raft_consensus.cc:686] T 34038775b5fa4b3193b8e57f09102ea3 P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Becoming Leader. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Running, Role: LEADER
I20240502 01:53:25.589677 18963 consensus_queue.cc:237] T 34038775b5fa4b3193b8e57f09102ea3 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:25.592651 17326 catalog_manager.cc:5484] T 34038775b5fa4b3193b8e57f09102ea3 P bb31bceba76d466db24b9341c04900c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb31bceba76d466db24b9341c04900c3 (127.11.42.68). New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } }
I20240502 01:53:25.936120 18915 consensus_queue.cc:1035] T e00020f7f47f4246907a8c90fca4a26b P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:25.943511 18915 consensus_queue.cc:1035] T e00020f7f47f4246907a8c90fca4a26b P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:25.948135 19004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:25.948552 19004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:25.956506 19004 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240502 01:53:25.993818 18963 consensus_queue.cc:1035] T 34038775b5fa4b3193b8e57f09102ea3 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:26.007117 18963 consensus_queue.cc:1035] T 34038775b5fa4b3193b8e57f09102ea3 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:26.013600 18919 consensus_queue.cc:1035] T 4280160c9b3a4b34b3c7d89f0fbec666 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:26.019845 18959 consensus_queue.cc:1035] T 4280160c9b3a4b34b3c7d89f0fbec666 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 208.735 ms
time per row: 0.104368 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 01:53:27.186314 17329 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:27.195459 17847 tablet_service.cc:1467] Processing CreateTablet for tablet 52ac2775e02844d195677035796c3847 (DEFAULT_TABLE table=rebalance_test_table_7 [id=c6c3a53b24874f31aa38118254253dcb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:27.195953 17847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52ac2775e02844d195677035796c3847. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:27.195901 17455 tablet_service.cc:1467] Processing CreateTablet for tablet 52ac2775e02844d195677035796c3847 (DEFAULT_TABLE table=rebalance_test_table_7 [id=c6c3a53b24874f31aa38118254253dcb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:27.196286 17455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52ac2775e02844d195677035796c3847. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:27.196766 17585 tablet_service.cc:1467] Processing CreateTablet for tablet 52ac2775e02844d195677035796c3847 (DEFAULT_TABLE table=rebalance_test_table_7 [id=c6c3a53b24874f31aa38118254253dcb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:27.197074 17586 tablet_service.cc:1467] Processing CreateTablet for tablet 7e9be84ef6a043818f166f6b47898855 (DEFAULT_TABLE table=rebalance_test_table_7 [id=c6c3a53b24874f31aa38118254253dcb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:27.197207 17585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52ac2775e02844d195677035796c3847. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:27.197467 17586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e9be84ef6a043818f166f6b47898855. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:27.198493 17715 tablet_service.cc:1467] Processing CreateTablet for tablet 5df83f83d2674155afba93c2d992af0a (DEFAULT_TABLE table=rebalance_test_table_7 [id=c6c3a53b24874f31aa38118254253dcb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:27.198890 17715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5df83f83d2674155afba93c2d992af0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:27.202163 17845 tablet_service.cc:1467] Processing CreateTablet for tablet 5df83f83d2674155afba93c2d992af0a (DEFAULT_TABLE table=rebalance_test_table_7 [id=c6c3a53b24874f31aa38118254253dcb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:27.202546 17845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5df83f83d2674155afba93c2d992af0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:27.205032 17717 tablet_service.cc:1467] Processing CreateTablet for tablet 7e9be84ef6a043818f166f6b47898855 (DEFAULT_TABLE table=rebalance_test_table_7 [id=c6c3a53b24874f31aa38118254253dcb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:27.205402 17717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e9be84ef6a043818f166f6b47898855. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:27.205420 17846 tablet_service.cc:1467] Processing CreateTablet for tablet 7e9be84ef6a043818f166f6b47898855 (DEFAULT_TABLE table=rebalance_test_table_7 [id=c6c3a53b24874f31aa38118254253dcb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:27.205854 17846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e9be84ef6a043818f166f6b47898855. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:27.209681 17456 tablet_service.cc:1467] Processing CreateTablet for tablet 5df83f83d2674155afba93c2d992af0a (DEFAULT_TABLE table=rebalance_test_table_7 [id=c6c3a53b24874f31aa38118254253dcb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:27.210106 17456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5df83f83d2674155afba93c2d992af0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:27.214264 19078 tablet_bootstrap.cc:492] T 7e9be84ef6a043818f166f6b47898855 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:27.216877 19078 tablet_bootstrap.cc:654] T 7e9be84ef6a043818f166f6b47898855 P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:27.219821 19079 tablet_bootstrap.cc:492] T 52ac2775e02844d195677035796c3847 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:27.220804 19077 tablet_bootstrap.cc:492] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:27.221777 19079 tablet_bootstrap.cc:654] T 52ac2775e02844d195677035796c3847 P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:27.223392 19077 tablet_bootstrap.cc:654] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:27.230141 19079 tablet_bootstrap.cc:492] T 52ac2775e02844d195677035796c3847 P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:27.230376 19079 ts_tablet_manager.cc:1367] T 52ac2775e02844d195677035796c3847 P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20240502 01:53:27.231236 19079 raft_consensus.cc:348] T 52ac2775e02844d195677035796c3847 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:27.231581 19079 raft_consensus.cc:374] T 52ac2775e02844d195677035796c3847 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:27.231711 19079 raft_consensus.cc:729] T 52ac2775e02844d195677035796c3847 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:27.232041 19079 consensus_queue.cc:260] T 52ac2775e02844d195677035796c3847 P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:27.232842 19079 ts_tablet_manager.cc:1398] T 52ac2775e02844d195677035796c3847 P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:53:27.233180 19079 tablet_bootstrap.cc:492] T 5df83f83d2674155afba93c2d992af0a P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:27.234905 19079 tablet_bootstrap.cc:654] T 5df83f83d2674155afba93c2d992af0a P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:27.235757 19078 tablet_bootstrap.cc:492] T 7e9be84ef6a043818f166f6b47898855 P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:27.235960 19078 ts_tablet_manager.cc:1367] T 7e9be84ef6a043818f166f6b47898855 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.000s
I20240502 01:53:27.236721 19078 raft_consensus.cc:348] T 7e9be84ef6a043818f166f6b47898855 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:27.237001 19078 raft_consensus.cc:374] T 7e9be84ef6a043818f166f6b47898855 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:27.237165 19078 raft_consensus.cc:729] T 7e9be84ef6a043818f166f6b47898855 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:27.237443 19078 consensus_queue.cc:260] T 7e9be84ef6a043818f166f6b47898855 P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:27.238031 19078 ts_tablet_manager.cc:1398] T 7e9be84ef6a043818f166f6b47898855 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:27.238345 19078 tablet_bootstrap.cc:492] T 52ac2775e02844d195677035796c3847 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:27.240449 19078 tablet_bootstrap.cc:654] T 52ac2775e02844d195677035796c3847 P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:27.243038 19076 tablet_bootstrap.cc:492] T 52ac2775e02844d195677035796c3847 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:27.243737 19078 tablet_bootstrap.cc:492] T 52ac2775e02844d195677035796c3847 P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:27.243809 19077 tablet_bootstrap.cc:492] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:27.243885 19078 ts_tablet_manager.cc:1367] T 52ac2775e02844d195677035796c3847 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20240502 01:53:27.243959 19077 ts_tablet_manager.cc:1367] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.004s
I20240502 01:53:27.244557 19078 raft_consensus.cc:348] T 52ac2775e02844d195677035796c3847 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:27.244771 19078 raft_consensus.cc:374] T 52ac2775e02844d195677035796c3847 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:27.244853 19078 raft_consensus.cc:729] T 52ac2775e02844d195677035796c3847 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:27.244871 19077 raft_consensus.cc:348] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:27.245142 19078 consensus_queue.cc:260] T 52ac2775e02844d195677035796c3847 P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:27.245234 19077 raft_consensus.cc:374] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:27.245435 19077 raft_consensus.cc:729] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:27.245725 19078 ts_tablet_manager.cc:1398] T 52ac2775e02844d195677035796c3847 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:27.245906 19076 tablet_bootstrap.cc:654] T 52ac2775e02844d195677035796c3847 P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:27.245734 19077 consensus_queue.cc:260] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:27.247817 19077 ts_tablet_manager.cc:1398] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:53:27.248083 19077 tablet_bootstrap.cc:492] T 7e9be84ef6a043818f166f6b47898855 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:27.248472 19079 tablet_bootstrap.cc:492] T 5df83f83d2674155afba93c2d992af0a P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:27.248644 19079 ts_tablet_manager.cc:1367] T 5df83f83d2674155afba93c2d992af0a P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.006s
I20240502 01:53:27.249394 19079 raft_consensus.cc:348] T 5df83f83d2674155afba93c2d992af0a P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:27.249632 19079 raft_consensus.cc:374] T 5df83f83d2674155afba93c2d992af0a P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:27.249740 19077 tablet_bootstrap.cc:654] T 7e9be84ef6a043818f166f6b47898855 P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:27.249732 19079 raft_consensus.cc:729] T 5df83f83d2674155afba93c2d992af0a P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:27.250036 19079 consensus_queue.cc:260] T 5df83f83d2674155afba93c2d992af0a P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:27.250483 18959 raft_consensus.cc:482] T 52ac2775e02844d195677035796c3847 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:27.250628 19079 ts_tablet_manager.cc:1398] T 5df83f83d2674155afba93c2d992af0a P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:53:27.250640 18959 raft_consensus.cc:504] T 52ac2775e02844d195677035796c3847 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:27.251286 18959 leader_election.cc:288] T 52ac2775e02844d195677035796c3847 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:27.251860 19071 raft_consensus.cc:482] T 52ac2775e02844d195677035796c3847 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:27.252010 19071 raft_consensus.cc:504] T 52ac2775e02844d195677035796c3847 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:27.252070 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52ac2775e02844d195677035796c3847" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:27.252316 17475 raft_consensus.cc:2454] T 52ac2775e02844d195677035796c3847 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:53:27.252661 19071 leader_election.cc:288] T 52ac2775e02844d195677035796c3847 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:27.253360 17542 leader_election.cc:302] T 52ac2775e02844d195677035796c3847 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, 4da690c0fa13407a85c0689ce8de9ade; no voters:
I20240502 01:53:27.253482 17866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52ac2775e02844d195677035796c3847" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
W20240502 01:53:27.254319 17412 leader_election.cc:341] T 52ac2775e02844d195677035796c3847 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:53:27.254815 19076 tablet_bootstrap.cc:492] T 52ac2775e02844d195677035796c3847 P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:27.254971 19076 ts_tablet_manager.cc:1367] T 52ac2775e02844d195677035796c3847 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240502 01:53:27.255964 19076 raft_consensus.cc:348] T 52ac2775e02844d195677035796c3847 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:27.256587 19076 raft_consensus.cc:374] T 52ac2775e02844d195677035796c3847 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:27.256800 19076 raft_consensus.cc:729] T 52ac2775e02844d195677035796c3847 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:27.257067 19076 consensus_queue.cc:260] T 52ac2775e02844d195677035796c3847 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:27.257696 19076 ts_tablet_manager.cc:1398] T 52ac2775e02844d195677035796c3847 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:27.258054 19076 tablet_bootstrap.cc:492] T 7e9be84ef6a043818f166f6b47898855 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:27.259616 19076 tablet_bootstrap.cc:654] T 7e9be84ef6a043818f166f6b47898855 P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:27.261199 19086 raft_consensus.cc:2781] T 52ac2775e02844d195677035796c3847 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:27.261304 19086 raft_consensus.cc:482] T 52ac2775e02844d195677035796c3847 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:27.261409 19086 raft_consensus.cc:3037] T 52ac2775e02844d195677035796c3847 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:27.262439 17866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52ac2775e02844d195677035796c3847" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:53:27.262643 17866 raft_consensus.cc:2454] T 52ac2775e02844d195677035796c3847 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:53:27.263490 17606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52ac2775e02844d195677035796c3847" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:53:27.264467 19086 raft_consensus.cc:504] T 52ac2775e02844d195677035796c3847 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:27.264889 17606 raft_consensus.cc:2379] T 52ac2775e02844d195677035796c3847 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4da690c0fa13407a85c0689ce8de9ade in current term 1: Already voted for candidate 3fceccefba274b1dbbd8216af8b0bc29 in this term.
I20240502 01:53:27.265185 19086 leader_election.cc:288] T 52ac2775e02844d195677035796c3847 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Requested vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:27.265791 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52ac2775e02844d195677035796c3847" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:53:27.266012 17475 raft_consensus.cc:3037] T 52ac2775e02844d195677035796c3847 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:27.266125 17409 leader_election.cc:302] T 52ac2775e02844d195677035796c3847 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade; no voters: 3fceccefba274b1dbbd8216af8b0bc29, bb31bceba76d466db24b9341c04900c3
I20240502 01:53:27.266808 17866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52ac2775e02844d195677035796c3847" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:53:27.267006 17866 raft_consensus.cc:3037] T 52ac2775e02844d195677035796c3847 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:27.268620 17475 raft_consensus.cc:2454] T 52ac2775e02844d195677035796c3847 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:53:27.269055 17542 leader_election.cc:302] T 52ac2775e02844d195677035796c3847 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, 4da690c0fa13407a85c0689ce8de9ade; no voters:
I20240502 01:53:27.269124 19071 raft_consensus.cc:2726] T 52ac2775e02844d195677035796c3847 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:53:27.269315 19086 raft_consensus.cc:2781] T 52ac2775e02844d195677035796c3847 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:27.269445 19086 raft_consensus.cc:686] T 52ac2775e02844d195677035796c3847 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Becoming Leader. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Running, Role: LEADER
I20240502 01:53:27.269608 17866 raft_consensus.cc:2454] T 52ac2775e02844d195677035796c3847 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:53:27.269687 19086 consensus_queue.cc:237] T 52ac2775e02844d195677035796c3847 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:27.270742 19076 tablet_bootstrap.cc:492] T 7e9be84ef6a043818f166f6b47898855 P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:27.270898 19076 ts_tablet_manager.cc:1367] T 7e9be84ef6a043818f166f6b47898855 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240502 01:53:27.271167 19077 tablet_bootstrap.cc:492] T 7e9be84ef6a043818f166f6b47898855 P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:27.271330 19077 ts_tablet_manager.cc:1367] T 7e9be84ef6a043818f166f6b47898855 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.023s user 0.020s sys 0.003s
I20240502 01:53:27.271497 19076 raft_consensus.cc:348] T 7e9be84ef6a043818f166f6b47898855 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:27.271660 19076 raft_consensus.cc:374] T 7e9be84ef6a043818f166f6b47898855 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:27.271739 19076 raft_consensus.cc:729] T 7e9be84ef6a043818f166f6b47898855 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:27.271962 19076 consensus_queue.cc:260] T 7e9be84ef6a043818f166f6b47898855 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:27.271982 19077 raft_consensus.cc:348] T 7e9be84ef6a043818f166f6b47898855 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:27.272164 19077 raft_consensus.cc:374] T 7e9be84ef6a043818f166f6b47898855 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:27.272241 19077 raft_consensus.cc:729] T 7e9be84ef6a043818f166f6b47898855 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:27.272508 19076 ts_tablet_manager.cc:1398] T 7e9be84ef6a043818f166f6b47898855 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:27.272471 19077 consensus_queue.cc:260] T 7e9be84ef6a043818f166f6b47898855 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:27.273046 19077 ts_tablet_manager.cc:1398] T 7e9be84ef6a043818f166f6b47898855 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:53:27.273412 19076 tablet_bootstrap.cc:492] T 5df83f83d2674155afba93c2d992af0a P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:27.274962 19076 tablet_bootstrap.cc:654] T 5df83f83d2674155afba93c2d992af0a P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:27.279780 17329 catalog_manager.cc:5484] T 52ac2775e02844d195677035796c3847 P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66). New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } }
I20240502 01:53:27.280328 19076 tablet_bootstrap.cc:492] T 5df83f83d2674155afba93c2d992af0a P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:27.280450 19076 ts_tablet_manager.cc:1367] T 5df83f83d2674155afba93c2d992af0a P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240502 01:53:27.281179 19076 raft_consensus.cc:348] T 5df83f83d2674155afba93c2d992af0a P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:27.281381 19076 raft_consensus.cc:374] T 5df83f83d2674155afba93c2d992af0a P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:27.281965 19076 raft_consensus.cc:729] T 5df83f83d2674155afba93c2d992af0a P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:27.282186 19076 consensus_queue.cc:260] T 5df83f83d2674155afba93c2d992af0a P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:27.283332 19076 ts_tablet_manager.cc:1398] T 5df83f83d2674155afba93c2d992af0a P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 01:53:27.473484 19000 raft_consensus.cc:482] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:27.473670 19000 raft_consensus.cc:504] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:27.474272 19000 leader_election.cc:288] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:27.474679 17866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df83f83d2674155afba93c2d992af0a" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:53:27.474717 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df83f83d2674155afba93c2d992af0a" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:27.474862 17866 raft_consensus.cc:2454] T 5df83f83d2674155afba93c2d992af0a P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:53:27.474920 17475 raft_consensus.cc:2454] T 5df83f83d2674155afba93c2d992af0a P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:53:27.475243 17673 leader_election.cc:302] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb31bceba76d466db24b9341c04900c3, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:27.475549 19000 raft_consensus.cc:2781] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:27.475653 19000 raft_consensus.cc:482] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:27.475715 19000 raft_consensus.cc:3037] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:27.477047 19093 raft_consensus.cc:482] T 7e9be84ef6a043818f166f6b47898855 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:27.477195 19093 raft_consensus.cc:504] T 7e9be84ef6a043818f166f6b47898855 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:27.478111 17866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9be84ef6a043818f166f6b47898855" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:53:27.478196 17606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9be84ef6a043818f166f6b47898855" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:53:27.478201 19000 raft_consensus.cc:504] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:27.478312 19093 leader_election.cc:288] T 7e9be84ef6a043818f166f6b47898855 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:27.478318 17866 raft_consensus.cc:2454] T 7e9be84ef6a043818f166f6b47898855 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:53:27.478422 17606 raft_consensus.cc:2454] T 7e9be84ef6a043818f166f6b47898855 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:53:27.478840 19000 leader_election.cc:288] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Requested vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:27.478839 17670 leader_election.cc:302] T 7e9be84ef6a043818f166f6b47898855 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:27.479120 19000 raft_consensus.cc:2781] T 7e9be84ef6a043818f166f6b47898855 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:27.479215 19000 raft_consensus.cc:482] T 7e9be84ef6a043818f166f6b47898855 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:27.479321 19000 raft_consensus.cc:3037] T 7e9be84ef6a043818f166f6b47898855 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:27.479344 17866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df83f83d2674155afba93c2d992af0a" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:53:27.479398 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df83f83d2674155afba93c2d992af0a" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:53:27.479511 17866 raft_consensus.cc:3037] T 5df83f83d2674155afba93c2d992af0a P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:27.479533 17475 raft_consensus.cc:3037] T 5df83f83d2674155afba93c2d992af0a P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:27.481768 19000 raft_consensus.cc:504] T 7e9be84ef6a043818f166f6b47898855 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:27.481858 17475 raft_consensus.cc:2454] T 5df83f83d2674155afba93c2d992af0a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:53:27.482129 17866 raft_consensus.cc:2454] T 5df83f83d2674155afba93c2d992af0a P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:53:27.482549 17673 leader_election.cc:302] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb31bceba76d466db24b9341c04900c3, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:27.482739 17867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9be84ef6a043818f166f6b47898855" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:53:27.482813 19093 raft_consensus.cc:2781] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:27.482950 17867 raft_consensus.cc:3037] T 7e9be84ef6a043818f166f6b47898855 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:27.483318 19093 raft_consensus.cc:686] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Becoming Leader. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Running, Role: LEADER
I20240502 01:53:27.483654 19093 consensus_queue.cc:237] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:27.484869 17606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9be84ef6a043818f166f6b47898855" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:53:27.485203 17606 raft_consensus.cc:3037] T 7e9be84ef6a043818f166f6b47898855 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:27.485553 17867 raft_consensus.cc:2454] T 7e9be84ef6a043818f166f6b47898855 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:53:27.482544 19000 leader_election.cc:288] T 7e9be84ef6a043818f166f6b47898855 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Requested vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:27.487181 17673 leader_election.cc:302] T 7e9be84ef6a043818f166f6b47898855 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb31bceba76d466db24b9341c04900c3, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:27.487896 17606 raft_consensus.cc:2454] T 7e9be84ef6a043818f166f6b47898855 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:53:27.488113 19000 raft_consensus.cc:2781] T 7e9be84ef6a043818f166f6b47898855 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:27.490633 19000 raft_consensus.cc:686] T 7e9be84ef6a043818f166f6b47898855 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Becoming Leader. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Running, Role: LEADER
I20240502 01:53:27.490881 19000 consensus_queue.cc:237] T 7e9be84ef6a043818f166f6b47898855 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:27.491590 17329 catalog_manager.cc:5484] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfa1152987644bd781b46c7e42a7a685 (127.11.42.67). New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:27.496260 17329 catalog_manager.cc:5484] T 7e9be84ef6a043818f166f6b47898855 P dfa1152987644bd781b46c7e42a7a685 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfa1152987644bd781b46c7e42a7a685 (127.11.42.67). New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:27.738943 19086 consensus_queue.cc:1035] T 52ac2775e02844d195677035796c3847 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:27.744061 19100 consensus_queue.cc:1035] T 52ac2775e02844d195677035796c3847 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:27.848690 19097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:27.848976 19097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:27.855947 19097 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240502 01:53:27.907531 19093 consensus_queue.cc:1035] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:27.915647 19094 consensus_queue.cc:1035] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:28.022307 17607 raft_consensus.cc:1261] T 7e9be84ef6a043818f166f6b47898855 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:28.023443 17866 raft_consensus.cc:1261] T 7e9be84ef6a043818f166f6b47898855 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:28.023500 19137 consensus_queue.cc:1035] T 7e9be84ef6a043818f166f6b47898855 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:28.024972 19094 consensus_queue.cc:1035] T 7e9be84ef6a043818f166f6b47898855 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 203.011 ms
time per row: 0.101506 ms
-----------------------------------------------------
Suppressions used:
count bytes template
4 1192 libcrypto.so.1.1
-----------------------------------------------------
I20240502 01:53:29.077601 17335 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:29.086530 17585 tablet_service.cc:1467] Processing CreateTablet for tablet a3ccdd1dffe846508262911fe0c22261 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3f381b46ce4c4598900dada557836f91]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:29.086803 17456 tablet_service.cc:1467] Processing CreateTablet for tablet a3ccdd1dffe846508262911fe0c22261 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3f381b46ce4c4598900dada557836f91]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:29.087057 17585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3ccdd1dffe846508262911fe0c22261. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:29.087217 17456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3ccdd1dffe846508262911fe0c22261. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:29.088215 17717 tablet_service.cc:1467] Processing CreateTablet for tablet a3ccdd1dffe846508262911fe0c22261 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3f381b46ce4c4598900dada557836f91]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:29.088537 17847 tablet_service.cc:1467] Processing CreateTablet for tablet 4cd692f7f2ca4dce901b7b2d2dc3a726 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3f381b46ce4c4598900dada557836f91]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:29.088675 17717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3ccdd1dffe846508262911fe0c22261. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:29.088920 17847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cd692f7f2ca4dce901b7b2d2dc3a726. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:29.091919 17845 tablet_service.cc:1467] Processing CreateTablet for tablet 1d2be05ef3b045a6bbd2de1798ef0460 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3f381b46ce4c4598900dada557836f91]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:29.092281 17845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d2be05ef3b045a6bbd2de1798ef0460. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:29.092648 17586 tablet_service.cc:1467] Processing CreateTablet for tablet 4cd692f7f2ca4dce901b7b2d2dc3a726 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3f381b46ce4c4598900dada557836f91]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:29.093034 17586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cd692f7f2ca4dce901b7b2d2dc3a726. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:29.093254 17455 tablet_service.cc:1467] Processing CreateTablet for tablet 4cd692f7f2ca4dce901b7b2d2dc3a726 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3f381b46ce4c4598900dada557836f91]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:29.093654 17455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cd692f7f2ca4dce901b7b2d2dc3a726. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:29.097402 17454 tablet_service.cc:1467] Processing CreateTablet for tablet 1d2be05ef3b045a6bbd2de1798ef0460 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3f381b46ce4c4598900dada557836f91]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:29.097784 17454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d2be05ef3b045a6bbd2de1798ef0460. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:29.101681 17715 tablet_service.cc:1467] Processing CreateTablet for tablet 1d2be05ef3b045a6bbd2de1798ef0460 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3f381b46ce4c4598900dada557836f91]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:29.102058 17715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d2be05ef3b045a6bbd2de1798ef0460. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:29.109755 19160 tablet_bootstrap.cc:492] T a3ccdd1dffe846508262911fe0c22261 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:29.110734 19158 tablet_bootstrap.cc:492] T 1d2be05ef3b045a6bbd2de1798ef0460 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:29.111133 19159 tablet_bootstrap.cc:492] T a3ccdd1dffe846508262911fe0c22261 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:29.111438 19160 tablet_bootstrap.cc:654] T a3ccdd1dffe846508262911fe0c22261 P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:29.112085 19158 tablet_bootstrap.cc:654] T 1d2be05ef3b045a6bbd2de1798ef0460 P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:29.112668 19159 tablet_bootstrap.cc:654] T a3ccdd1dffe846508262911fe0c22261 P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:29.113447 19160 tablet_bootstrap.cc:492] T a3ccdd1dffe846508262911fe0c22261 P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:29.110555 19157 tablet_bootstrap.cc:492] T a3ccdd1dffe846508262911fe0c22261 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:29.113646 19160 ts_tablet_manager.cc:1367] T a3ccdd1dffe846508262911fe0c22261 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:53:29.113850 19158 tablet_bootstrap.cc:492] T 1d2be05ef3b045a6bbd2de1798ef0460 P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:29.114235 19158 ts_tablet_manager.cc:1367] T 1d2be05ef3b045a6bbd2de1798ef0460 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 01:53:29.114750 19160 raft_consensus.cc:348] T a3ccdd1dffe846508262911fe0c22261 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:29.114974 19160 raft_consensus.cc:374] T a3ccdd1dffe846508262911fe0c22261 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:29.115005 19157 tablet_bootstrap.cc:654] T a3ccdd1dffe846508262911fe0c22261 P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:29.115072 19160 raft_consensus.cc:729] T a3ccdd1dffe846508262911fe0c22261 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:29.115307 19158 raft_consensus.cc:348] T 1d2be05ef3b045a6bbd2de1798ef0460 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:29.115303 19160 consensus_queue.cc:260] T a3ccdd1dffe846508262911fe0c22261 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:29.115473 19158 raft_consensus.cc:374] T 1d2be05ef3b045a6bbd2de1798ef0460 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:29.115572 19158 raft_consensus.cc:729] T 1d2be05ef3b045a6bbd2de1798ef0460 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:29.115842 19158 consensus_queue.cc:260] T 1d2be05ef3b045a6bbd2de1798ef0460 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:29.115965 19160 ts_tablet_manager.cc:1398] T a3ccdd1dffe846508262911fe0c22261 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:29.116650 19159 tablet_bootstrap.cc:492] T a3ccdd1dffe846508262911fe0c22261 P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:29.116789 19159 ts_tablet_manager.cc:1367] T a3ccdd1dffe846508262911fe0c22261 P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240502 01:53:29.116890 19160 tablet_bootstrap.cc:492] T 1d2be05ef3b045a6bbd2de1798ef0460 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:29.117555 19159 raft_consensus.cc:348] T a3ccdd1dffe846508262911fe0c22261 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:29.117828 19159 raft_consensus.cc:374] T a3ccdd1dffe846508262911fe0c22261 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:29.117959 19159 raft_consensus.cc:729] T a3ccdd1dffe846508262911fe0c22261 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:29.118490 19160 tablet_bootstrap.cc:654] T 1d2be05ef3b045a6bbd2de1798ef0460 P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:29.119516 19158 ts_tablet_manager.cc:1398] T 1d2be05ef3b045a6bbd2de1798ef0460 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 01:53:29.122651 19158 tablet_bootstrap.cc:492] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:29.123752 19158 tablet_bootstrap.cc:654] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:29.125226 19158 tablet_bootstrap.cc:492] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:29.125403 19158 ts_tablet_manager.cc:1367] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.002s
I20240502 01:53:29.126048 19158 raft_consensus.cc:348] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:29.126235 19158 raft_consensus.cc:374] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:29.126307 19158 raft_consensus.cc:729] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:29.126422 19160 tablet_bootstrap.cc:492] T 1d2be05ef3b045a6bbd2de1798ef0460 P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:29.126633 19160 ts_tablet_manager.cc:1367] T 1d2be05ef3b045a6bbd2de1798ef0460 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20240502 01:53:29.126596 19158 consensus_queue.cc:260] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:29.127343 19160 raft_consensus.cc:348] T 1d2be05ef3b045a6bbd2de1798ef0460 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:29.127588 19160 raft_consensus.cc:374] T 1d2be05ef3b045a6bbd2de1798ef0460 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:29.127686 19160 raft_consensus.cc:729] T 1d2be05ef3b045a6bbd2de1798ef0460 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:29.127982 19160 consensus_queue.cc:260] T 1d2be05ef3b045a6bbd2de1798ef0460 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:29.128563 19160 ts_tablet_manager.cc:1398] T 1d2be05ef3b045a6bbd2de1798ef0460 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:53:29.127701 19158 ts_tablet_manager.cc:1398] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:29.118273 19159 consensus_queue.cc:260] T a3ccdd1dffe846508262911fe0c22261 P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:29.133778 19159 ts_tablet_manager.cc:1398] T a3ccdd1dffe846508262911fe0c22261 P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.017s user 0.003s sys 0.000s
I20240502 01:53:29.134107 19159 tablet_bootstrap.cc:492] T 1d2be05ef3b045a6bbd2de1798ef0460 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:29.135648 19159 tablet_bootstrap.cc:654] T 1d2be05ef3b045a6bbd2de1798ef0460 P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:29.138068 19157 tablet_bootstrap.cc:492] T a3ccdd1dffe846508262911fe0c22261 P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:29.138267 19157 ts_tablet_manager.cc:1367] T a3ccdd1dffe846508262911fe0c22261 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.000s
I20240502 01:53:29.139040 19157 raft_consensus.cc:348] T a3ccdd1dffe846508262911fe0c22261 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:29.139315 19157 raft_consensus.cc:374] T a3ccdd1dffe846508262911fe0c22261 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:29.139458 19157 raft_consensus.cc:729] T a3ccdd1dffe846508262911fe0c22261 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:29.139765 19157 consensus_queue.cc:260] T a3ccdd1dffe846508262911fe0c22261 P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:29.141069 19157 ts_tablet_manager.cc:1398] T a3ccdd1dffe846508262911fe0c22261 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:53:29.141394 19157 tablet_bootstrap.cc:492] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:29.143610 19157 tablet_bootstrap.cc:654] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:29.144452 19159 tablet_bootstrap.cc:492] T 1d2be05ef3b045a6bbd2de1798ef0460 P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:29.144620 19159 ts_tablet_manager.cc:1367] T 1d2be05ef3b045a6bbd2de1798ef0460 P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20240502 01:53:29.145125 19157 tablet_bootstrap.cc:492] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:29.145263 19157 ts_tablet_manager.cc:1367] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:53:29.145448 19159 raft_consensus.cc:348] T 1d2be05ef3b045a6bbd2de1798ef0460 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:29.145632 19159 raft_consensus.cc:374] T 1d2be05ef3b045a6bbd2de1798ef0460 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:29.145699 19159 raft_consensus.cc:729] T 1d2be05ef3b045a6bbd2de1798ef0460 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:29.145915 19157 raft_consensus.cc:348] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:29.145920 19159 consensus_queue.cc:260] T 1d2be05ef3b045a6bbd2de1798ef0460 P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:29.146076 19157 raft_consensus.cc:374] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:29.146170 19157 raft_consensus.cc:729] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:29.146378 19159 ts_tablet_manager.cc:1398] T 1d2be05ef3b045a6bbd2de1798ef0460 P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:53:29.146448 19157 consensus_queue.cc:260] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:29.146757 19159 tablet_bootstrap.cc:492] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:29.147048 19157 ts_tablet_manager.cc:1398] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:53:29.148207 19159 tablet_bootstrap.cc:654] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:29.152784 19159 tablet_bootstrap.cc:492] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:29.152912 19159 ts_tablet_manager.cc:1367] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240502 01:53:29.153462 19159 raft_consensus.cc:348] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:29.153650 19159 raft_consensus.cc:374] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:29.153748 19159 raft_consensus.cc:729] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:29.153952 19159 consensus_queue.cc:260] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:29.154349 19159 ts_tablet_manager.cc:1398] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:53:29.187939 19071 raft_consensus.cc:482] T a3ccdd1dffe846508262911fe0c22261 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:29.188097 19071 raft_consensus.cc:504] T a3ccdd1dffe846508262911fe0c22261 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:29.188807 19071 leader_election.cc:288] T a3ccdd1dffe846508262911fe0c22261 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:53:29.189237 17606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccdd1dffe846508262911fe0c22261" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:53:29.189438 17606 raft_consensus.cc:2454] T a3ccdd1dffe846508262911fe0c22261 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 0.
I20240502 01:53:29.189425 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccdd1dffe846508262911fe0c22261" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:53:29.189625 17735 raft_consensus.cc:2454] T a3ccdd1dffe846508262911fe0c22261 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 0.
I20240502 01:53:29.189805 17409 leader_election.cc:302] T a3ccdd1dffe846508262911fe0c22261 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, 4da690c0fa13407a85c0689ce8de9ade; no voters:
I20240502 01:53:29.190044 19071 raft_consensus.cc:2781] T a3ccdd1dffe846508262911fe0c22261 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:29.190145 19071 raft_consensus.cc:482] T a3ccdd1dffe846508262911fe0c22261 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:29.190230 19071 raft_consensus.cc:3037] T a3ccdd1dffe846508262911fe0c22261 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:29.192013 19071 raft_consensus.cc:504] T a3ccdd1dffe846508262911fe0c22261 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:29.192519 19071 leader_election.cc:288] T a3ccdd1dffe846508262911fe0c22261 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Requested vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:53:29.192965 17606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccdd1dffe846508262911fe0c22261" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:53:29.192941 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ccdd1dffe846508262911fe0c22261" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:53:29.193172 17606 raft_consensus.cc:3037] T a3ccdd1dffe846508262911fe0c22261 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:29.193172 17735 raft_consensus.cc:3037] T a3ccdd1dffe846508262911fe0c22261 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:29.194896 17606 raft_consensus.cc:2454] T a3ccdd1dffe846508262911fe0c22261 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
I20240502 01:53:29.194909 17735 raft_consensus.cc:2454] T a3ccdd1dffe846508262911fe0c22261 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
I20240502 01:53:29.195256 17411 leader_election.cc:302] T a3ccdd1dffe846508262911fe0c22261 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, 4da690c0fa13407a85c0689ce8de9ade, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:29.195539 19071 raft_consensus.cc:2781] T a3ccdd1dffe846508262911fe0c22261 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:29.195649 19071 raft_consensus.cc:686] T a3ccdd1dffe846508262911fe0c22261 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Becoming Leader. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Running, Role: LEADER
I20240502 01:53:29.195995 19071 consensus_queue.cc:237] T a3ccdd1dffe846508262911fe0c22261 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:29.198712 17335 catalog_manager.cc:5484] T a3ccdd1dffe846508262911fe0c22261 P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: term changed from 0 to 1, leader changed from <none> to 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65). New cstate: current_term: 1 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:29.233075 19172 raft_consensus.cc:482] T 1d2be05ef3b045a6bbd2de1798ef0460 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:29.233253 19172 raft_consensus.cc:504] T 1d2be05ef3b045a6bbd2de1798ef0460 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:29.233914 19172 leader_election.cc:288] T 1d2be05ef3b045a6bbd2de1798ef0460 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:53:29.234329 17866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d2be05ef3b045a6bbd2de1798ef0460" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:53:29.234336 17474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d2be05ef3b045a6bbd2de1798ef0460" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:29.234513 17866 raft_consensus.cc:2454] T 1d2be05ef3b045a6bbd2de1798ef0460 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:53:29.234534 17474 raft_consensus.cc:2454] T 1d2be05ef3b045a6bbd2de1798ef0460 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:53:29.234910 17673 leader_election.cc:302] T 1d2be05ef3b045a6bbd2de1798ef0460 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb31bceba76d466db24b9341c04900c3, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:29.235177 19172 raft_consensus.cc:2781] T 1d2be05ef3b045a6bbd2de1798ef0460 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:29.235317 19172 raft_consensus.cc:482] T 1d2be05ef3b045a6bbd2de1798ef0460 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:29.235405 19172 raft_consensus.cc:3037] T 1d2be05ef3b045a6bbd2de1798ef0460 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:29.237318 19172 raft_consensus.cc:504] T 1d2be05ef3b045a6bbd2de1798ef0460 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:29.237798 19172 leader_election.cc:288] T 1d2be05ef3b045a6bbd2de1798ef0460 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Requested vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:53:29.238178 17474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d2be05ef3b045a6bbd2de1798ef0460" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:53:29.238266 17866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d2be05ef3b045a6bbd2de1798ef0460" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:53:29.238339 17474 raft_consensus.cc:3037] T 1d2be05ef3b045a6bbd2de1798ef0460 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:29.238430 17866 raft_consensus.cc:3037] T 1d2be05ef3b045a6bbd2de1798ef0460 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:29.240538 17866 raft_consensus.cc:2454] T 1d2be05ef3b045a6bbd2de1798ef0460 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:53:29.240731 17474 raft_consensus.cc:2454] T 1d2be05ef3b045a6bbd2de1798ef0460 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:53:29.240911 17673 leader_election.cc:302] T 1d2be05ef3b045a6bbd2de1798ef0460 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb31bceba76d466db24b9341c04900c3, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:29.241237 19172 raft_consensus.cc:2781] T 1d2be05ef3b045a6bbd2de1798ef0460 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:29.241365 19172 raft_consensus.cc:686] T 1d2be05ef3b045a6bbd2de1798ef0460 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Becoming Leader. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Running, Role: LEADER
I20240502 01:53:29.241587 19172 consensus_queue.cc:237] T 1d2be05ef3b045a6bbd2de1798ef0460 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:29.244529 17335 catalog_manager.cc:5484] T 1d2be05ef3b045a6bbd2de1798ef0460 P dfa1152987644bd781b46c7e42a7a685 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfa1152987644bd781b46c7e42a7a685 (127.11.42.67). New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:29.274088 19100 raft_consensus.cc:482] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:29.274276 19100 raft_consensus.cc:504] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:29.274930 19100 leader_election.cc:288] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:29.275278 17866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd692f7f2ca4dce901b7b2d2dc3a726" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:53:29.275441 17474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd692f7f2ca4dce901b7b2d2dc3a726" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:29.275537 17866 raft_consensus.cc:2454] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:53:29.275691 17474 raft_consensus.cc:2454] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:53:29.275952 17543 leader_election.cc:302] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:53:29.276239 19100 raft_consensus.cc:2781] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:29.276340 19100 raft_consensus.cc:482] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:29.276440 19100 raft_consensus.cc:3037] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:29.278393 19100 raft_consensus.cc:504] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:29.278934 19100 leader_election.cc:288] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Requested vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:29.279260 17866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd692f7f2ca4dce901b7b2d2dc3a726" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:53:29.279383 17474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd692f7f2ca4dce901b7b2d2dc3a726" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:53:29.279475 17866 raft_consensus.cc:3037] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:29.279582 17474 raft_consensus.cc:3037] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:29.282215 17474 raft_consensus.cc:2454] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:53:29.282231 17866 raft_consensus.cc:2454] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:53:29.282717 17542 leader_election.cc:302] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, 4da690c0fa13407a85c0689ce8de9ade; no voters:
I20240502 01:53:29.282981 19100 raft_consensus.cc:2781] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:29.284113 19100 raft_consensus.cc:686] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Becoming Leader. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Running, Role: LEADER
I20240502 01:53:29.284351 19100 consensus_queue.cc:237] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:29.286924 17335 catalog_manager.cc:5484] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66). New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:29.595490 19071 consensus_queue.cc:1035] T a3ccdd1dffe846508262911fe0c22261 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:29.597299 19071 consensus_queue.cc:1035] T a3ccdd1dffe846508262911fe0c22261 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:29.668391 19177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:29.668678 19177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:29.676245 19177 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240502 01:53:29.724038 19094 consensus_queue.cc:1035] T 1d2be05ef3b045a6bbd2de1798ef0460 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:29.729815 19204 consensus_queue.cc:1035] T 1d2be05ef3b045a6bbd2de1798ef0460 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:29.750519 19100 consensus_queue.cc:1035] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:29.768249 19175 consensus_queue.cc:1035] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 249.382 ms
time per row: 0.124691 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 01:53:30.928226 17332 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:30.938416 17585 tablet_service.cc:1467] Processing CreateTablet for tablet de174ba4f7d848718ab8dbeb74369055 (DEFAULT_TABLE table=rebalance_test_table_9 [id=ffe6d29170f24ad48433354f548b9b2a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:30.938683 17845 tablet_service.cc:1467] Processing CreateTablet for tablet de174ba4f7d848718ab8dbeb74369055 (DEFAULT_TABLE table=rebalance_test_table_9 [id=ffe6d29170f24ad48433354f548b9b2a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:30.938894 17585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de174ba4f7d848718ab8dbeb74369055. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:30.939124 17845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de174ba4f7d848718ab8dbeb74369055. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:30.939388 17587 tablet_service.cc:1467] Processing CreateTablet for tablet b39b88755cd74b0294d264215aea99cf (DEFAULT_TABLE table=rebalance_test_table_9 [id=ffe6d29170f24ad48433354f548b9b2a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:30.939780 17587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b39b88755cd74b0294d264215aea99cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:30.940696 17716 tablet_service.cc:1467] Processing CreateTablet for tablet b39b88755cd74b0294d264215aea99cf (DEFAULT_TABLE table=rebalance_test_table_9 [id=ffe6d29170f24ad48433354f548b9b2a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:30.941129 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b39b88755cd74b0294d264215aea99cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:30.945210 17586 tablet_service.cc:1467] Processing CreateTablet for tablet 0d9f168de0d1410da2518abf936035ff (DEFAULT_TABLE table=rebalance_test_table_9 [id=ffe6d29170f24ad48433354f548b9b2a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:30.945428 17456 tablet_service.cc:1467] Processing CreateTablet for tablet b39b88755cd74b0294d264215aea99cf (DEFAULT_TABLE table=rebalance_test_table_9 [id=ffe6d29170f24ad48433354f548b9b2a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:30.945559 17586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d9f168de0d1410da2518abf936035ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:30.945740 17715 tablet_service.cc:1467] Processing CreateTablet for tablet 0d9f168de0d1410da2518abf936035ff (DEFAULT_TABLE table=rebalance_test_table_9 [id=ffe6d29170f24ad48433354f548b9b2a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:30.945856 17456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b39b88755cd74b0294d264215aea99cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:30.946092 17715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d9f168de0d1410da2518abf936035ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:30.949373 17717 tablet_service.cc:1467] Processing CreateTablet for tablet de174ba4f7d848718ab8dbeb74369055 (DEFAULT_TABLE table=rebalance_test_table_9 [id=ffe6d29170f24ad48433354f548b9b2a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:30.949712 17717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de174ba4f7d848718ab8dbeb74369055. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:30.949934 17847 tablet_service.cc:1467] Processing CreateTablet for tablet 0d9f168de0d1410da2518abf936035ff (DEFAULT_TABLE table=rebalance_test_table_9 [id=ffe6d29170f24ad48433354f548b9b2a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:30.950285 17847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d9f168de0d1410da2518abf936035ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:30.958156 19243 tablet_bootstrap.cc:492] T b39b88755cd74b0294d264215aea99cf P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:30.959388 19242 tablet_bootstrap.cc:492] T de174ba4f7d848718ab8dbeb74369055 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:30.959789 19241 tablet_bootstrap.cc:492] T de174ba4f7d848718ab8dbeb74369055 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:30.960199 19243 tablet_bootstrap.cc:654] T b39b88755cd74b0294d264215aea99cf P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:30.961889 19244 tablet_bootstrap.cc:492] T b39b88755cd74b0294d264215aea99cf P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:30.964149 19244 tablet_bootstrap.cc:654] T b39b88755cd74b0294d264215aea99cf P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:30.964718 19242 tablet_bootstrap.cc:654] T de174ba4f7d848718ab8dbeb74369055 P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:30.965842 19241 tablet_bootstrap.cc:654] T de174ba4f7d848718ab8dbeb74369055 P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:30.973364 19243 tablet_bootstrap.cc:492] T b39b88755cd74b0294d264215aea99cf P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:30.973551 19243 ts_tablet_manager.cc:1367] T b39b88755cd74b0294d264215aea99cf P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.015s user 0.013s sys 0.000s
I20240502 01:53:30.973758 19244 tablet_bootstrap.cc:492] T b39b88755cd74b0294d264215aea99cf P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:30.973989 19244 ts_tablet_manager.cc:1367] T b39b88755cd74b0294d264215aea99cf P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20240502 01:53:30.974933 19243 raft_consensus.cc:348] T b39b88755cd74b0294d264215aea99cf P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:30.975004 19244 raft_consensus.cc:348] T b39b88755cd74b0294d264215aea99cf P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:30.975176 19244 raft_consensus.cc:374] T b39b88755cd74b0294d264215aea99cf P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:30.975271 19244 raft_consensus.cc:729] T b39b88755cd74b0294d264215aea99cf P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:30.975301 19243 raft_consensus.cc:374] T b39b88755cd74b0294d264215aea99cf P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:30.975492 19243 raft_consensus.cc:729] T b39b88755cd74b0294d264215aea99cf P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:30.975520 19244 consensus_queue.cc:260] T b39b88755cd74b0294d264215aea99cf P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:30.975831 19243 consensus_queue.cc:260] T b39b88755cd74b0294d264215aea99cf P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:30.976784 19243 ts_tablet_manager.cc:1398] T b39b88755cd74b0294d264215aea99cf P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:30.977118 19243 tablet_bootstrap.cc:492] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:30.978684 19243 tablet_bootstrap.cc:654] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:30.979288 19244 ts_tablet_manager.cc:1398] T b39b88755cd74b0294d264215aea99cf P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 01:53:30.980274 19242 tablet_bootstrap.cc:492] T de174ba4f7d848718ab8dbeb74369055 P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:30.980403 19242 ts_tablet_manager.cc:1367] T de174ba4f7d848718ab8dbeb74369055 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.000s
I20240502 01:53:30.981184 19242 raft_consensus.cc:348] T de174ba4f7d848718ab8dbeb74369055 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:30.981384 19242 raft_consensus.cc:374] T de174ba4f7d848718ab8dbeb74369055 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:30.981473 19242 raft_consensus.cc:729] T de174ba4f7d848718ab8dbeb74369055 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:30.981701 19242 consensus_queue.cc:260] T de174ba4f7d848718ab8dbeb74369055 P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:30.982273 19242 ts_tablet_manager.cc:1398] T de174ba4f7d848718ab8dbeb74369055 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:30.982563 19242 tablet_bootstrap.cc:492] T b39b88755cd74b0294d264215aea99cf P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:30.984071 19242 tablet_bootstrap.cc:654] T b39b88755cd74b0294d264215aea99cf P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:30.992897 19241 tablet_bootstrap.cc:492] T de174ba4f7d848718ab8dbeb74369055 P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:30.993142 19241 ts_tablet_manager.cc:1367] T de174ba4f7d848718ab8dbeb74369055 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.022s
I20240502 01:53:30.993247 19242 tablet_bootstrap.cc:492] T b39b88755cd74b0294d264215aea99cf P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:30.993475 19242 ts_tablet_manager.cc:1367] T b39b88755cd74b0294d264215aea99cf P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.001s
I20240502 01:53:30.993950 19241 raft_consensus.cc:348] T de174ba4f7d848718ab8dbeb74369055 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:30.994210 19241 raft_consensus.cc:374] T de174ba4f7d848718ab8dbeb74369055 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:30.994182 19242 raft_consensus.cc:348] T b39b88755cd74b0294d264215aea99cf P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:30.994392 19242 raft_consensus.cc:374] T b39b88755cd74b0294d264215aea99cf P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:30.994416 19241 raft_consensus.cc:729] T de174ba4f7d848718ab8dbeb74369055 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:30.994472 19242 raft_consensus.cc:729] T b39b88755cd74b0294d264215aea99cf P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:30.994751 19241 consensus_queue.cc:260] T de174ba4f7d848718ab8dbeb74369055 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:30.994760 19242 consensus_queue.cc:260] T b39b88755cd74b0294d264215aea99cf P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:30.995285 19242 ts_tablet_manager.cc:1398] T b39b88755cd74b0294d264215aea99cf P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:53:30.995626 19242 tablet_bootstrap.cc:492] T 0d9f168de0d1410da2518abf936035ff P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:30.996037 19241 ts_tablet_manager.cc:1398] T de174ba4f7d848718ab8dbeb74369055 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:30.996353 19241 tablet_bootstrap.cc:492] T 0d9f168de0d1410da2518abf936035ff P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:30.997201 19242 tablet_bootstrap.cc:654] T 0d9f168de0d1410da2518abf936035ff P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:30.998983 19241 tablet_bootstrap.cc:654] T 0d9f168de0d1410da2518abf936035ff P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:31.005504 19243 tablet_bootstrap.cc:492] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:31.005751 19243 ts_tablet_manager.cc:1367] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.004s
I20240502 01:53:31.007422 19243 raft_consensus.cc:348] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:31.008626 19241 tablet_bootstrap.cc:492] T 0d9f168de0d1410da2518abf936035ff P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:31.008831 19241 ts_tablet_manager.cc:1367] T 0d9f168de0d1410da2518abf936035ff P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.002s
I20240502 01:53:31.009164 19243 raft_consensus.cc:374] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:31.009346 19243 raft_consensus.cc:729] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:31.009692 19243 consensus_queue.cc:260] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:31.009725 19241 raft_consensus.cc:348] T 0d9f168de0d1410da2518abf936035ff P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:31.009932 19241 raft_consensus.cc:374] T 0d9f168de0d1410da2518abf936035ff P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:31.010063 19241 raft_consensus.cc:729] T 0d9f168de0d1410da2518abf936035ff P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:31.010422 19243 ts_tablet_manager.cc:1398] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:53:31.010372 19241 consensus_queue.cc:260] T 0d9f168de0d1410da2518abf936035ff P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:31.011731 19241 ts_tablet_manager.cc:1398] T 0d9f168de0d1410da2518abf936035ff P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:53:31.012673 19243 tablet_bootstrap.cc:492] T de174ba4f7d848718ab8dbeb74369055 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:31.016311 19243 tablet_bootstrap.cc:654] T de174ba4f7d848718ab8dbeb74369055 P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:31.018083 19243 tablet_bootstrap.cc:492] T de174ba4f7d848718ab8dbeb74369055 P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:31.018085 19242 tablet_bootstrap.cc:492] T 0d9f168de0d1410da2518abf936035ff P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:31.018227 19243 ts_tablet_manager.cc:1367] T de174ba4f7d848718ab8dbeb74369055 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240502 01:53:31.018227 19242 ts_tablet_manager.cc:1367] T 0d9f168de0d1410da2518abf936035ff P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.009s
I20240502 01:53:31.018927 19243 raft_consensus.cc:348] T de174ba4f7d848718ab8dbeb74369055 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:31.018929 19242 raft_consensus.cc:348] T 0d9f168de0d1410da2518abf936035ff P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:31.019130 19243 raft_consensus.cc:374] T de174ba4f7d848718ab8dbeb74369055 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:31.019126 19242 raft_consensus.cc:374] T 0d9f168de0d1410da2518abf936035ff P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:31.019222 19243 raft_consensus.cc:729] T de174ba4f7d848718ab8dbeb74369055 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:31.019222 19242 raft_consensus.cc:729] T 0d9f168de0d1410da2518abf936035ff P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:31.019435 19243 consensus_queue.cc:260] T de174ba4f7d848718ab8dbeb74369055 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:31.019434 19242 consensus_queue.cc:260] T 0d9f168de0d1410da2518abf936035ff P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:31.019989 19243 ts_tablet_manager.cc:1398] T de174ba4f7d848718ab8dbeb74369055 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:53:31.020005 19242 ts_tablet_manager.cc:1398] T 0d9f168de0d1410da2518abf936035ff P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:53:31.033283 19204 raft_consensus.cc:482] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:31.033417 19204 raft_consensus.cc:504] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:31.034039 19204 leader_election.cc:288] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:53:31.034425 17606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d9f168de0d1410da2518abf936035ff" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:53:31.034515 17865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d9f168de0d1410da2518abf936035ff" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:53:31.034633 17606 raft_consensus.cc:2454] T 0d9f168de0d1410da2518abf936035ff P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:53:31.034696 17865 raft_consensus.cc:2454] T 0d9f168de0d1410da2518abf936035ff P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:53:31.035053 17670 leader_election.cc:302] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:31.035323 19204 raft_consensus.cc:2781] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:31.035410 19204 raft_consensus.cc:482] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:31.035519 19204 raft_consensus.cc:3037] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:31.037613 19204 raft_consensus.cc:504] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:31.038125 19204 leader_election.cc:288] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Requested vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:53:31.038458 17606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d9f168de0d1410da2518abf936035ff" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:53:31.038616 17606 raft_consensus.cc:3037] T 0d9f168de0d1410da2518abf936035ff P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:31.038573 17865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d9f168de0d1410da2518abf936035ff" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:53:31.038722 17865 raft_consensus.cc:3037] T 0d9f168de0d1410da2518abf936035ff P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:31.040441 17865 raft_consensus.cc:2454] T 0d9f168de0d1410da2518abf936035ff P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:53:31.040438 17606 raft_consensus.cc:2454] T 0d9f168de0d1410da2518abf936035ff P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:53:31.040787 17673 leader_election.cc:302] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb31bceba76d466db24b9341c04900c3, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:31.041069 19204 raft_consensus.cc:2781] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:31.041249 19204 raft_consensus.cc:686] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Becoming Leader. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Running, Role: LEADER
I20240502 01:53:31.041497 19204 consensus_queue.cc:237] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:31.044286 17332 catalog_manager.cc:5484] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfa1152987644bd781b46c7e42a7a685 (127.11.42.67). New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:31.059985 19170 raft_consensus.cc:482] T b39b88755cd74b0294d264215aea99cf P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:31.060148 19170 raft_consensus.cc:504] T b39b88755cd74b0294d264215aea99cf P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:31.060806 19170 leader_election.cc:288] T b39b88755cd74b0294d264215aea99cf P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:53:31.061101 17606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b39b88755cd74b0294d264215aea99cf" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:53:31.061214 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b39b88755cd74b0294d264215aea99cf" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:53:31.061347 17606 raft_consensus.cc:2454] T b39b88755cd74b0294d264215aea99cf P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 0.
I20240502 01:53:31.061419 17737 raft_consensus.cc:2454] T b39b88755cd74b0294d264215aea99cf P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 0.
I20240502 01:53:31.061712 17409 leader_election.cc:302] T b39b88755cd74b0294d264215aea99cf P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, 4da690c0fa13407a85c0689ce8de9ade; no voters:
I20240502 01:53:31.061987 19170 raft_consensus.cc:2781] T b39b88755cd74b0294d264215aea99cf P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:31.062083 19170 raft_consensus.cc:482] T b39b88755cd74b0294d264215aea99cf P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:31.062148 19170 raft_consensus.cc:3037] T b39b88755cd74b0294d264215aea99cf P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:31.063839 19170 raft_consensus.cc:504] T b39b88755cd74b0294d264215aea99cf P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:31.064328 19170 leader_election.cc:288] T b39b88755cd74b0294d264215aea99cf P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Requested vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:53:31.064742 17606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b39b88755cd74b0294d264215aea99cf" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:53:31.064785 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b39b88755cd74b0294d264215aea99cf" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:53:31.064916 17606 raft_consensus.cc:3037] T b39b88755cd74b0294d264215aea99cf P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:31.064944 17737 raft_consensus.cc:3037] T b39b88755cd74b0294d264215aea99cf P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:31.066656 17737 raft_consensus.cc:2454] T b39b88755cd74b0294d264215aea99cf P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
I20240502 01:53:31.066663 17606 raft_consensus.cc:2454] T b39b88755cd74b0294d264215aea99cf P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
I20240502 01:53:31.067003 17409 leader_election.cc:302] T b39b88755cd74b0294d264215aea99cf P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, 4da690c0fa13407a85c0689ce8de9ade; no voters:
I20240502 01:53:31.067307 19170 raft_consensus.cc:2781] T b39b88755cd74b0294d264215aea99cf P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:31.067612 19170 raft_consensus.cc:686] T b39b88755cd74b0294d264215aea99cf P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Becoming Leader. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Running, Role: LEADER
I20240502 01:53:31.067898 19170 consensus_queue.cc:237] T b39b88755cd74b0294d264215aea99cf P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:31.070439 17332 catalog_manager.cc:5484] T b39b88755cd74b0294d264215aea99cf P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: term changed from 0 to 1, leader changed from <none> to 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65). New cstate: current_term: 1 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } }
I20240502 01:53:31.173846 19258 raft_consensus.cc:482] T de174ba4f7d848718ab8dbeb74369055 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:31.174012 19258 raft_consensus.cc:504] T de174ba4f7d848718ab8dbeb74369055 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:31.174666 19258 leader_election.cc:288] T de174ba4f7d848718ab8dbeb74369055 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:31.175067 17606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de174ba4f7d848718ab8dbeb74369055" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:53:31.175060 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de174ba4f7d848718ab8dbeb74369055" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:53:31.175261 17606 raft_consensus.cc:2454] T de174ba4f7d848718ab8dbeb74369055 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 0.
I20240502 01:53:31.175292 17737 raft_consensus.cc:2454] T de174ba4f7d848718ab8dbeb74369055 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 0.
I20240502 01:53:31.175627 17800 leader_election.cc:302] T de174ba4f7d848718ab8dbeb74369055 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:53:31.175900 19258 raft_consensus.cc:2781] T de174ba4f7d848718ab8dbeb74369055 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:31.176008 19258 raft_consensus.cc:482] T de174ba4f7d848718ab8dbeb74369055 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:31.176095 19258 raft_consensus.cc:3037] T de174ba4f7d848718ab8dbeb74369055 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:31.178064 19258 raft_consensus.cc:504] T de174ba4f7d848718ab8dbeb74369055 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:31.178539 19258 leader_election.cc:288] T de174ba4f7d848718ab8dbeb74369055 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 election: Requested vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:31.178966 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de174ba4f7d848718ab8dbeb74369055" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:53:31.178990 17606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de174ba4f7d848718ab8dbeb74369055" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:53:31.179152 17606 raft_consensus.cc:3037] T de174ba4f7d848718ab8dbeb74369055 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:31.179153 17737 raft_consensus.cc:3037] T de174ba4f7d848718ab8dbeb74369055 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:31.180981 17606 raft_consensus.cc:2454] T de174ba4f7d848718ab8dbeb74369055 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 1.
I20240502 01:53:31.181388 17800 leader_election.cc:302] T de174ba4f7d848718ab8dbeb74369055 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:53:31.181638 19258 raft_consensus.cc:2781] T de174ba4f7d848718ab8dbeb74369055 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:31.181777 19258 raft_consensus.cc:686] T de174ba4f7d848718ab8dbeb74369055 P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Becoming Leader. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Running, Role: LEADER
I20240502 01:53:31.181895 17737 raft_consensus.cc:2454] T de174ba4f7d848718ab8dbeb74369055 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 1.
I20240502 01:53:31.182301 19258 consensus_queue.cc:237] T de174ba4f7d848718ab8dbeb74369055 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:31.185104 17332 catalog_manager.cc:5484] T de174ba4f7d848718ab8dbeb74369055 P bb31bceba76d466db24b9341c04900c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb31bceba76d466db24b9341c04900c3 (127.11.42.68). New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } }
I20240502 01:53:31.492193 19204 consensus_queue.cc:1035] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:31.501483 19204 consensus_queue.cc:1035] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:31.502478 19170 consensus_queue.cc:1035] T b39b88755cd74b0294d264215aea99cf P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:31.515269 19255 consensus_queue.cc:1035] T b39b88755cd74b0294d264215aea99cf P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:31.516940 19260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:31.517355 19260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:31.526001 19260 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240502 01:53:31.672187 19258 consensus_queue.cc:1035] T de174ba4f7d848718ab8dbeb74369055 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:31.690483 17735 raft_consensus.cc:1261] T de174ba4f7d848718ab8dbeb74369055 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:31.694110 19092 consensus_queue.cc:1035] T de174ba4f7d848718ab8dbeb74369055 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:31.832592 17672 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.42.65:38321, user_credentials={real_user=slave}} blocked reactor thread for 66093.5us
INSERT report
rows total: 2000
time total: 262.596 ms
time per row: 0.131298 ms
I20240502 01:53:31.927865 17332 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:31.936939 17585 tablet_service.cc:1467] Processing CreateTablet for tablet 8652d27049f64a6eb65cdba6f3875765 (DEFAULT_TABLE table=rebalance_test_table_10 [id=b69ea3782c3c43dfa2760155beffc87b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:31.937211 17456 tablet_service.cc:1467] Processing CreateTablet for tablet 8652d27049f64a6eb65cdba6f3875765 (DEFAULT_TABLE table=rebalance_test_table_10 [id=b69ea3782c3c43dfa2760155beffc87b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:31.937436 17585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8652d27049f64a6eb65cdba6f3875765. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:31.937546 17456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8652d27049f64a6eb65cdba6f3875765. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:31.937577 17845 tablet_service.cc:1467] Processing CreateTablet for tablet 8652d27049f64a6eb65cdba6f3875765 (DEFAULT_TABLE table=rebalance_test_table_10 [id=b69ea3782c3c43dfa2760155beffc87b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:31.937963 17845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8652d27049f64a6eb65cdba6f3875765. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:31.940151 17847 tablet_service.cc:1467] Processing CreateTablet for tablet b572585ef00942df88881c4afbe7b7cd (DEFAULT_TABLE table=rebalance_test_table_10 [id=b69ea3782c3c43dfa2760155beffc87b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:31.940495 17847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b572585ef00942df88881c4afbe7b7cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:31.940670 17716 tablet_service.cc:1467] Processing CreateTablet for tablet b572585ef00942df88881c4afbe7b7cd (DEFAULT_TABLE table=rebalance_test_table_10 [id=b69ea3782c3c43dfa2760155beffc87b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:31.941156 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b572585ef00942df88881c4afbe7b7cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:31.945372 17846 tablet_service.cc:1467] Processing CreateTablet for tablet 9e4bc5ed97814e79b602697a9204aa5f (DEFAULT_TABLE table=rebalance_test_table_10 [id=b69ea3782c3c43dfa2760155beffc87b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:31.945469 17455 tablet_service.cc:1467] Processing CreateTablet for tablet b572585ef00942df88881c4afbe7b7cd (DEFAULT_TABLE table=rebalance_test_table_10 [id=b69ea3782c3c43dfa2760155beffc87b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:31.945761 17846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e4bc5ed97814e79b602697a9204aa5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:31.945890 17455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b572585ef00942df88881c4afbe7b7cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:31.950009 17586 tablet_service.cc:1467] Processing CreateTablet for tablet 9e4bc5ed97814e79b602697a9204aa5f (DEFAULT_TABLE table=rebalance_test_table_10 [id=b69ea3782c3c43dfa2760155beffc87b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:31.950408 17586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e4bc5ed97814e79b602697a9204aa5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:31.951575 17454 tablet_service.cc:1467] Processing CreateTablet for tablet 9e4bc5ed97814e79b602697a9204aa5f (DEFAULT_TABLE table=rebalance_test_table_10 [id=b69ea3782c3c43dfa2760155beffc87b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:31.951921 17454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e4bc5ed97814e79b602697a9204aa5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:31.963186 19332 tablet_bootstrap.cc:492] T 9e4bc5ed97814e79b602697a9204aa5f P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:31.963873 19330 tablet_bootstrap.cc:492] T 8652d27049f64a6eb65cdba6f3875765 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:31.964421 19331 tablet_bootstrap.cc:492] T b572585ef00942df88881c4afbe7b7cd P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:31.965636 19330 tablet_bootstrap.cc:654] T 8652d27049f64a6eb65cdba6f3875765 P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:31.966446 19332 tablet_bootstrap.cc:654] T 9e4bc5ed97814e79b602697a9204aa5f P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:31.966782 19331 tablet_bootstrap.cc:654] T b572585ef00942df88881c4afbe7b7cd P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:31.972599 19332 tablet_bootstrap.cc:492] T 9e4bc5ed97814e79b602697a9204aa5f P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:31.973126 19332 ts_tablet_manager.cc:1367] T 9e4bc5ed97814e79b602697a9204aa5f P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20240502 01:53:31.974169 19332 raft_consensus.cc:348] T 9e4bc5ed97814e79b602697a9204aa5f P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:31.975359 19332 raft_consensus.cc:374] T 9e4bc5ed97814e79b602697a9204aa5f P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:31.975632 19332 raft_consensus.cc:729] T 9e4bc5ed97814e79b602697a9204aa5f P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:31.976151 19331 tablet_bootstrap.cc:492] T b572585ef00942df88881c4afbe7b7cd P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:31.976316 19331 ts_tablet_manager.cc:1367] T b572585ef00942df88881c4afbe7b7cd P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20240502 01:53:31.976063 19332 consensus_queue.cc:260] T 9e4bc5ed97814e79b602697a9204aa5f P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:31.977157 19332 ts_tablet_manager.cc:1398] T 9e4bc5ed97814e79b602697a9204aa5f P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:53:31.977119 19331 raft_consensus.cc:348] T b572585ef00942df88881c4afbe7b7cd P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:31.977346 19331 raft_consensus.cc:374] T b572585ef00942df88881c4afbe7b7cd P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:31.977442 19331 raft_consensus.cc:729] T b572585ef00942df88881c4afbe7b7cd P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:31.977746 19331 consensus_queue.cc:260] T b572585ef00942df88881c4afbe7b7cd P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:31.978204 19332 tablet_bootstrap.cc:492] T 8652d27049f64a6eb65cdba6f3875765 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:31.979310 19331 ts_tablet_manager.cc:1398] T b572585ef00942df88881c4afbe7b7cd P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:31.980894 19332 tablet_bootstrap.cc:654] T 8652d27049f64a6eb65cdba6f3875765 P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:31.990746 19332 tablet_bootstrap.cc:492] T 8652d27049f64a6eb65cdba6f3875765 P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:31.990895 19332 ts_tablet_manager.cc:1367] T 8652d27049f64a6eb65cdba6f3875765 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.006s
I20240502 01:53:31.991458 19332 raft_consensus.cc:348] T 8652d27049f64a6eb65cdba6f3875765 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:31.991621 19332 raft_consensus.cc:374] T 8652d27049f64a6eb65cdba6f3875765 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:31.991698 19332 raft_consensus.cc:729] T 8652d27049f64a6eb65cdba6f3875765 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:31.993131 19332 consensus_queue.cc:260] T 8652d27049f64a6eb65cdba6f3875765 P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:31.993956 19332 ts_tablet_manager.cc:1398] T 8652d27049f64a6eb65cdba6f3875765 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:53:31.994663 19329 tablet_bootstrap.cc:492] T 8652d27049f64a6eb65cdba6f3875765 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:31.998045 19330 tablet_bootstrap.cc:492] T 8652d27049f64a6eb65cdba6f3875765 P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:31.998265 19330 ts_tablet_manager.cc:1367] T 8652d27049f64a6eb65cdba6f3875765 P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.029s
I20240502 01:53:31.998414 19329 tablet_bootstrap.cc:654] T 8652d27049f64a6eb65cdba6f3875765 P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:31.999100 19330 raft_consensus.cc:348] T 8652d27049f64a6eb65cdba6f3875765 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:31.999279 19330 raft_consensus.cc:374] T 8652d27049f64a6eb65cdba6f3875765 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:31.999526 19330 raft_consensus.cc:729] T 8652d27049f64a6eb65cdba6f3875765 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:31.999850 19330 consensus_queue.cc:260] T 8652d27049f64a6eb65cdba6f3875765 P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:32.000566 19330 ts_tablet_manager.cc:1398] T 8652d27049f64a6eb65cdba6f3875765 P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:53:32.001821 19330 tablet_bootstrap.cc:492] T 9e4bc5ed97814e79b602697a9204aa5f P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:32.003369 19330 tablet_bootstrap.cc:654] T 9e4bc5ed97814e79b602697a9204aa5f P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:32.005194 19330 tablet_bootstrap.cc:492] T 9e4bc5ed97814e79b602697a9204aa5f P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:32.005267 19329 tablet_bootstrap.cc:492] T 8652d27049f64a6eb65cdba6f3875765 P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:32.005379 19330 ts_tablet_manager.cc:1367] T 9e4bc5ed97814e79b602697a9204aa5f P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240502 01:53:32.005455 19329 ts_tablet_manager.cc:1367] T 8652d27049f64a6eb65cdba6f3875765 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.003s
I20240502 01:53:32.006235 19329 raft_consensus.cc:348] T 8652d27049f64a6eb65cdba6f3875765 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:32.006247 19330 raft_consensus.cc:348] T 9e4bc5ed97814e79b602697a9204aa5f P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:32.006448 19330 raft_consensus.cc:374] T 9e4bc5ed97814e79b602697a9204aa5f P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:32.006500 19329 raft_consensus.cc:374] T 8652d27049f64a6eb65cdba6f3875765 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:32.006537 19330 raft_consensus.cc:729] T 9e4bc5ed97814e79b602697a9204aa5f P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:32.006670 19329 raft_consensus.cc:729] T 8652d27049f64a6eb65cdba6f3875765 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:32.006778 19236 raft_consensus.cc:482] T 9e4bc5ed97814e79b602697a9204aa5f P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:32.006918 19236 raft_consensus.cc:504] T 9e4bc5ed97814e79b602697a9204aa5f P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:32.006752 19330 consensus_queue.cc:260] T 9e4bc5ed97814e79b602697a9204aa5f P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:32.006978 19329 consensus_queue.cc:260] T 8652d27049f64a6eb65cdba6f3875765 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:32.007380 19258 raft_consensus.cc:482] T 8652d27049f64a6eb65cdba6f3875765 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:32.007552 19258 raft_consensus.cc:504] T 8652d27049f64a6eb65cdba6f3875765 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:32.007656 19236 leader_election.cc:288] T 9e4bc5ed97814e79b602697a9204aa5f P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:53:32.007800 19329 ts_tablet_manager.cc:1398] T 8652d27049f64a6eb65cdba6f3875765 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:53:32.008096 19329 tablet_bootstrap.cc:492] T b572585ef00942df88881c4afbe7b7cd P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:32.008284 19258 leader_election.cc:288] T 8652d27049f64a6eb65cdba6f3875765 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:32.008754 19330 ts_tablet_manager.cc:1398] T 9e4bc5ed97814e79b602697a9204aa5f P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:32.008734 17474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8652d27049f64a6eb65cdba6f3875765" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:32.009048 17474 raft_consensus.cc:2454] T 8652d27049f64a6eb65cdba6f3875765 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 0.
I20240502 01:53:32.009156 19330 tablet_bootstrap.cc:492] T b572585ef00942df88881c4afbe7b7cd P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:32.009624 17802 leader_election.cc:302] T 8652d27049f64a6eb65cdba6f3875765 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:53:32.009949 19258 raft_consensus.cc:2781] T 8652d27049f64a6eb65cdba6f3875765 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:32.010051 19258 raft_consensus.cc:482] T 8652d27049f64a6eb65cdba6f3875765 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:32.010130 19258 raft_consensus.cc:3037] T 8652d27049f64a6eb65cdba6f3875765 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:32.010883 19330 tablet_bootstrap.cc:654] T b572585ef00942df88881c4afbe7b7cd P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:32.010917 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e4bc5ed97814e79b602697a9204aa5f" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:32.011116 17475 raft_consensus.cc:2454] T 9e4bc5ed97814e79b602697a9204aa5f P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:53:32.011466 17605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8652d27049f64a6eb65cdba6f3875765" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:53:32.011672 17605 raft_consensus.cc:2454] T 8652d27049f64a6eb65cdba6f3875765 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 0.
I20240502 01:53:32.012979 19258 raft_consensus.cc:504] T 8652d27049f64a6eb65cdba6f3875765 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:32.014539 19258 leader_election.cc:288] T 8652d27049f64a6eb65cdba6f3875765 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 election: Requested vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:32.016009 17542 leader_election.cc:302] T 9e4bc5ed97814e79b602697a9204aa5f P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, 4da690c0fa13407a85c0689ce8de9ade; no voters:
I20240502 01:53:32.016511 19329 tablet_bootstrap.cc:654] T b572585ef00942df88881c4afbe7b7cd P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:32.016788 17605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8652d27049f64a6eb65cdba6f3875765" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:53:32.017273 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8652d27049f64a6eb65cdba6f3875765" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:53:32.017505 17475 raft_consensus.cc:3037] T 8652d27049f64a6eb65cdba6f3875765 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:32.017835 19339 raft_consensus.cc:2781] T 9e4bc5ed97814e79b602697a9204aa5f P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:32.017971 19339 raft_consensus.cc:482] T 9e4bc5ed97814e79b602697a9204aa5f P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:32.018117 19339 raft_consensus.cc:3037] T 9e4bc5ed97814e79b602697a9204aa5f P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:32.020460 17475 raft_consensus.cc:2454] T 8652d27049f64a6eb65cdba6f3875765 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 1.
I20240502 01:53:32.020947 17802 leader_election.cc:302] T 8652d27049f64a6eb65cdba6f3875765 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:53:32.021317 19258 raft_consensus.cc:2781] T 8652d27049f64a6eb65cdba6f3875765 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:32.021445 19258 raft_consensus.cc:686] T 8652d27049f64a6eb65cdba6f3875765 P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Becoming Leader. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Running, Role: LEADER
I20240502 01:53:32.021710 19258 consensus_queue.cc:237] T 8652d27049f64a6eb65cdba6f3875765 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:32.017006 17605 raft_consensus.cc:3037] T 8652d27049f64a6eb65cdba6f3875765 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:32.024039 19339 raft_consensus.cc:504] T 9e4bc5ed97814e79b602697a9204aa5f P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:32.025286 19339 leader_election.cc:288] T 9e4bc5ed97814e79b602697a9204aa5f P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Requested vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:53:32.025223 17865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e4bc5ed97814e79b602697a9204aa5f" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:53:32.026072 17865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e4bc5ed97814e79b602697a9204aa5f" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
W20240502 01:53:32.026100 17543 leader_election.cc:341] T 9e4bc5ed97814e79b602697a9204aa5f P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): Illegal state: must be running to vote when last-logged opid is not known
W20240502 01:53:32.026592 17543 leader_election.cc:341] T 9e4bc5ed97814e79b602697a9204aa5f P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:53:32.027149 19329 tablet_bootstrap.cc:492] T b572585ef00942df88881c4afbe7b7cd P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:32.027173 17605 raft_consensus.cc:2454] T 8652d27049f64a6eb65cdba6f3875765 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 1.
I20240502 01:53:32.027412 19329 ts_tablet_manager.cc:1367] T b572585ef00942df88881c4afbe7b7cd P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.003s
I20240502 01:53:32.028265 19329 raft_consensus.cc:348] T b572585ef00942df88881c4afbe7b7cd P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:32.028522 19329 raft_consensus.cc:374] T b572585ef00942df88881c4afbe7b7cd P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:32.028631 19329 raft_consensus.cc:729] T b572585ef00942df88881c4afbe7b7cd P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:32.028914 19329 consensus_queue.cc:260] T b572585ef00942df88881c4afbe7b7cd P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:32.029590 19329 ts_tablet_manager.cc:1398] T b572585ef00942df88881c4afbe7b7cd P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:32.029953 19329 tablet_bootstrap.cc:492] T 9e4bc5ed97814e79b602697a9204aa5f P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:32.030961 17474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e4bc5ed97814e79b602697a9204aa5f" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:53:32.031189 17474 raft_consensus.cc:3037] T 9e4bc5ed97814e79b602697a9204aa5f P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:32.031698 19329 tablet_bootstrap.cc:654] T 9e4bc5ed97814e79b602697a9204aa5f P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:32.033852 19330 tablet_bootstrap.cc:492] T b572585ef00942df88881c4afbe7b7cd P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:32.033918 17474 raft_consensus.cc:2454] T 9e4bc5ed97814e79b602697a9204aa5f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:53:32.034014 19330 ts_tablet_manager.cc:1367] T b572585ef00942df88881c4afbe7b7cd P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.019s
I20240502 01:53:32.034302 17542 leader_election.cc:302] T 9e4bc5ed97814e79b602697a9204aa5f P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, 4da690c0fa13407a85c0689ce8de9ade; no voters: bb31bceba76d466db24b9341c04900c3
I20240502 01:53:32.034878 19330 raft_consensus.cc:348] T b572585ef00942df88881c4afbe7b7cd P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:32.035072 19330 raft_consensus.cc:374] T b572585ef00942df88881c4afbe7b7cd P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:32.035092 19236 raft_consensus.cc:2781] T 9e4bc5ed97814e79b602697a9204aa5f P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:32.035177 19330 raft_consensus.cc:729] T b572585ef00942df88881c4afbe7b7cd P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:32.035218 19236 raft_consensus.cc:686] T 9e4bc5ed97814e79b602697a9204aa5f P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Becoming Leader. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Running, Role: LEADER
I20240502 01:53:32.035418 19330 consensus_queue.cc:260] T b572585ef00942df88881c4afbe7b7cd P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:32.035476 19236 consensus_queue.cc:237] T 9e4bc5ed97814e79b602697a9204aa5f P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:32.039111 19330 ts_tablet_manager.cc:1398] T b572585ef00942df88881c4afbe7b7cd P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 01:53:32.045979 17332 catalog_manager.cc:5484] T 8652d27049f64a6eb65cdba6f3875765 P bb31bceba76d466db24b9341c04900c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb31bceba76d466db24b9341c04900c3 (127.11.42.68). New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } }
I20240502 01:53:32.046187 17331 catalog_manager.cc:5484] T 9e4bc5ed97814e79b602697a9204aa5f P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66). New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:32.048868 19329 tablet_bootstrap.cc:492] T 9e4bc5ed97814e79b602697a9204aa5f P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:32.048991 19329 ts_tablet_manager.cc:1367] T 9e4bc5ed97814e79b602697a9204aa5f P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.005s
I20240502 01:53:32.049767 19329 raft_consensus.cc:348] T 9e4bc5ed97814e79b602697a9204aa5f P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:32.049971 19329 raft_consensus.cc:374] T 9e4bc5ed97814e79b602697a9204aa5f P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:32.050072 19329 raft_consensus.cc:729] T 9e4bc5ed97814e79b602697a9204aa5f P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:32.050298 19329 consensus_queue.cc:260] T 9e4bc5ed97814e79b602697a9204aa5f P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:32.050833 19329 ts_tablet_manager.cc:1398] T 9e4bc5ed97814e79b602697a9204aa5f P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:53:32.182307 19170 raft_consensus.cc:482] T b572585ef00942df88881c4afbe7b7cd P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:32.182464 19170 raft_consensus.cc:504] T b572585ef00942df88881c4afbe7b7cd P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:32.183313 19170 leader_election.cc:288] T b572585ef00942df88881c4afbe7b7cd P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:53:32.183691 17865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b572585ef00942df88881c4afbe7b7cd" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:53:32.183691 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b572585ef00942df88881c4afbe7b7cd" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:53:32.183908 17735 raft_consensus.cc:2454] T b572585ef00942df88881c4afbe7b7cd P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 0.
I20240502 01:53:32.183907 17865 raft_consensus.cc:2454] T b572585ef00942df88881c4afbe7b7cd P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 0.
I20240502 01:53:32.184309 17412 leader_election.cc:302] T b572585ef00942df88881c4afbe7b7cd P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:53:32.184569 19170 raft_consensus.cc:2781] T b572585ef00942df88881c4afbe7b7cd P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:32.184679 19170 raft_consensus.cc:482] T b572585ef00942df88881c4afbe7b7cd P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:32.184747 19170 raft_consensus.cc:3037] T b572585ef00942df88881c4afbe7b7cd P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:32.186540 19170 raft_consensus.cc:504] T b572585ef00942df88881c4afbe7b7cd P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:32.187067 19170 leader_election.cc:288] T b572585ef00942df88881c4afbe7b7cd P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Requested vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:53:32.187397 17865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b572585ef00942df88881c4afbe7b7cd" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:53:32.187577 17865 raft_consensus.cc:3037] T b572585ef00942df88881c4afbe7b7cd P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:32.187538 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b572585ef00942df88881c4afbe7b7cd" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:53:32.187739 17735 raft_consensus.cc:3037] T b572585ef00942df88881c4afbe7b7cd P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:32.189417 17865 raft_consensus.cc:2454] T b572585ef00942df88881c4afbe7b7cd P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
I20240502 01:53:32.189625 17735 raft_consensus.cc:2454] T b572585ef00942df88881c4afbe7b7cd P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
I20240502 01:53:32.189760 17412 leader_election.cc:302] T b572585ef00942df88881c4afbe7b7cd P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:53:32.190083 19170 raft_consensus.cc:2781] T b572585ef00942df88881c4afbe7b7cd P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:32.190245 19170 raft_consensus.cc:686] T b572585ef00942df88881c4afbe7b7cd P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Becoming Leader. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Running, Role: LEADER
I20240502 01:53:32.190500 19170 consensus_queue.cc:237] T b572585ef00942df88881c4afbe7b7cd P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:32.193228 17331 catalog_manager.cc:5484] T b572585ef00942df88881c4afbe7b7cd P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: term changed from 0 to 1, leader changed from <none> to 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65). New cstate: current_term: 1 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } }
I20240502 01:53:32.219317 19258 raft_consensus.cc:482] T 9e4bc5ed97814e79b602697a9204aa5f P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:32.219509 19258 raft_consensus.cc:504] T 9e4bc5ed97814e79b602697a9204aa5f P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:32.220310 19258 leader_election.cc:288] T 9e4bc5ed97814e79b602697a9204aa5f P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:32.220705 17605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e4bc5ed97814e79b602697a9204aa5f" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:53:32.220767 17474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e4bc5ed97814e79b602697a9204aa5f" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:32.221176 17474 raft_consensus.cc:2379] T 9e4bc5ed97814e79b602697a9204aa5f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bb31bceba76d466db24b9341c04900c3 in current term 1: Already voted for candidate 3fceccefba274b1dbbd8216af8b0bc29 in this term.
I20240502 01:53:32.221598 17802 leader_election.cc:302] T 9e4bc5ed97814e79b602697a9204aa5f P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb31bceba76d466db24b9341c04900c3; no voters: 3fceccefba274b1dbbd8216af8b0bc29, 4da690c0fa13407a85c0689ce8de9ade
I20240502 01:53:32.221877 19258 raft_consensus.cc:3037] T 9e4bc5ed97814e79b602697a9204aa5f P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:32.223753 19258 raft_consensus.cc:2726] T 9e4bc5ed97814e79b602697a9204aa5f P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:53:32.484218 19236 consensus_queue.cc:1035] T 9e4bc5ed97814e79b602697a9204aa5f P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:32.492189 19339 consensus_queue.cc:1035] T 9e4bc5ed97814e79b602697a9204aa5f P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:32.502812 19258 consensus_queue.cc:1035] T 8652d27049f64a6eb65cdba6f3875765 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:32.508700 19092 consensus_queue.cc:1035] T 8652d27049f64a6eb65cdba6f3875765 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:32.561578 19348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:32.561992 19348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:32.573765 19348 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_10'
I20240502 01:53:32.638793 19170 consensus_queue.cc:1035] T b572585ef00942df88881c4afbe7b7cd P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:32.649843 19170 consensus_queue.cc:1035] T b572585ef00942df88881c4afbe7b7cd P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 245.024 ms
time per row: 0.122512 ms
I20240502 01:53:32.971882 17331 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:32.981297 17456 tablet_service.cc:1467] Processing CreateTablet for tablet b29218a879e74895bfe00021ad358aa4 (DEFAULT_TABLE table=rebalance_test_table_11 [id=5e2315325f6c4e7f8834471d07555cf5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:32.981771 17456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b29218a879e74895bfe00021ad358aa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:32.982194 17845 tablet_service.cc:1467] Processing CreateTablet for tablet ed1522ec150a46f28fd553f9b563651f (DEFAULT_TABLE table=rebalance_test_table_11 [id=5e2315325f6c4e7f8834471d07555cf5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:32.982595 17845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed1522ec150a46f28fd553f9b563651f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:32.985682 17716 tablet_service.cc:1467] Processing CreateTablet for tablet b29218a879e74895bfe00021ad358aa4 (DEFAULT_TABLE table=rebalance_test_table_11 [id=5e2315325f6c4e7f8834471d07555cf5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:32.985806 17717 tablet_service.cc:1467] Processing CreateTablet for tablet ed1522ec150a46f28fd553f9b563651f (DEFAULT_TABLE table=rebalance_test_table_11 [id=5e2315325f6c4e7f8834471d07555cf5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:32.985805 17715 tablet_service.cc:1467] Processing CreateTablet for tablet 1f01949cf70440b38e59a5020fd48c44 (DEFAULT_TABLE table=rebalance_test_table_11 [id=5e2315325f6c4e7f8834471d07555cf5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:32.986106 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b29218a879e74895bfe00021ad358aa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:32.986335 17715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f01949cf70440b38e59a5020fd48c44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:32.986430 17586 tablet_service.cc:1467] Processing CreateTablet for tablet 1f01949cf70440b38e59a5020fd48c44 (DEFAULT_TABLE table=rebalance_test_table_11 [id=5e2315325f6c4e7f8834471d07555cf5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:32.986822 17586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f01949cf70440b38e59a5020fd48c44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:32.986770 17455 tablet_service.cc:1467] Processing CreateTablet for tablet ed1522ec150a46f28fd553f9b563651f (DEFAULT_TABLE table=rebalance_test_table_11 [id=5e2315325f6c4e7f8834471d07555cf5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:32.987114 17455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed1522ec150a46f28fd553f9b563651f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:32.989418 17846 tablet_service.cc:1467] Processing CreateTablet for tablet 1f01949cf70440b38e59a5020fd48c44 (DEFAULT_TABLE table=rebalance_test_table_11 [id=5e2315325f6c4e7f8834471d07555cf5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:32.989784 17846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f01949cf70440b38e59a5020fd48c44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:32.991657 17717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed1522ec150a46f28fd553f9b563651f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:32.993352 17585 tablet_service.cc:1467] Processing CreateTablet for tablet b29218a879e74895bfe00021ad358aa4 (DEFAULT_TABLE table=rebalance_test_table_11 [id=5e2315325f6c4e7f8834471d07555cf5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:32.993714 17585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b29218a879e74895bfe00021ad358aa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:33.001479 19413 tablet_bootstrap.cc:492] T b29218a879e74895bfe00021ad358aa4 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:33.004804 19413 tablet_bootstrap.cc:654] T b29218a879e74895bfe00021ad358aa4 P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:33.010432 19411 tablet_bootstrap.cc:492] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:33.011677 19413 tablet_bootstrap.cc:492] T b29218a879e74895bfe00021ad358aa4 P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:33.011824 19413 ts_tablet_manager.cc:1367] T b29218a879e74895bfe00021ad358aa4 P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.004s
I20240502 01:53:33.012877 19413 raft_consensus.cc:348] T b29218a879e74895bfe00021ad358aa4 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:33.013139 19413 raft_consensus.cc:374] T b29218a879e74895bfe00021ad358aa4 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:33.013218 19413 raft_consensus.cc:729] T b29218a879e74895bfe00021ad358aa4 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:33.013451 19413 consensus_queue.cc:260] T b29218a879e74895bfe00021ad358aa4 P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:33.013980 19413 ts_tablet_manager.cc:1398] T b29218a879e74895bfe00021ad358aa4 P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:33.014698 19411 tablet_bootstrap.cc:654] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:33.022367 19413 tablet_bootstrap.cc:492] T ed1522ec150a46f28fd553f9b563651f P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:33.023730 19412 tablet_bootstrap.cc:492] T 1f01949cf70440b38e59a5020fd48c44 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:33.023999 19413 tablet_bootstrap.cc:654] T ed1522ec150a46f28fd553f9b563651f P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:33.025542 19414 tablet_bootstrap.cc:492] T 1f01949cf70440b38e59a5020fd48c44 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:33.026899 19412 tablet_bootstrap.cc:654] T 1f01949cf70440b38e59a5020fd48c44 P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:33.028050 19414 tablet_bootstrap.cc:654] T 1f01949cf70440b38e59a5020fd48c44 P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:33.029829 19413 tablet_bootstrap.cc:492] T ed1522ec150a46f28fd553f9b563651f P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:33.029992 19413 ts_tablet_manager.cc:1367] T ed1522ec150a46f28fd553f9b563651f P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240502 01:53:33.030694 19413 raft_consensus.cc:348] T ed1522ec150a46f28fd553f9b563651f P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:33.030923 19413 raft_consensus.cc:374] T ed1522ec150a46f28fd553f9b563651f P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:33.031001 19413 raft_consensus.cc:729] T ed1522ec150a46f28fd553f9b563651f P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:33.031261 19413 consensus_queue.cc:260] T ed1522ec150a46f28fd553f9b563651f P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:33.031922 19413 ts_tablet_manager.cc:1398] T ed1522ec150a46f28fd553f9b563651f P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 01:53:33.032421 19412 tablet_bootstrap.cc:492] T 1f01949cf70440b38e59a5020fd48c44 P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:33.032994 19412 ts_tablet_manager.cc:1367] T 1f01949cf70440b38e59a5020fd48c44 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240502 01:53:33.033329 19411 tablet_bootstrap.cc:492] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:33.033951 19412 raft_consensus.cc:348] T 1f01949cf70440b38e59a5020fd48c44 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:33.034157 19412 raft_consensus.cc:374] T 1f01949cf70440b38e59a5020fd48c44 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:33.034256 19412 raft_consensus.cc:729] T 1f01949cf70440b38e59a5020fd48c44 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:33.034322 19414 tablet_bootstrap.cc:492] T 1f01949cf70440b38e59a5020fd48c44 P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:33.034484 19414 ts_tablet_manager.cc:1367] T 1f01949cf70440b38e59a5020fd48c44 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240502 01:53:33.034516 19412 consensus_queue.cc:260] T 1f01949cf70440b38e59a5020fd48c44 P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:33.035120 19412 ts_tablet_manager.cc:1398] T 1f01949cf70440b38e59a5020fd48c44 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:53:33.035092 19414 raft_consensus.cc:348] T 1f01949cf70440b38e59a5020fd48c44 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:33.035331 19414 raft_consensus.cc:374] T 1f01949cf70440b38e59a5020fd48c44 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:33.035450 19414 raft_consensus.cc:729] T 1f01949cf70440b38e59a5020fd48c44 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:33.035742 19411 ts_tablet_manager.cc:1367] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.025s user 0.013s sys 0.000s
I20240502 01:53:33.035671 19414 consensus_queue.cc:260] T 1f01949cf70440b38e59a5020fd48c44 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:33.036482 19414 ts_tablet_manager.cc:1398] T 1f01949cf70440b38e59a5020fd48c44 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:33.036540 19411 raft_consensus.cc:348] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:33.036758 19411 raft_consensus.cc:374] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:33.036845 19412 tablet_bootstrap.cc:492] T b29218a879e74895bfe00021ad358aa4 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:33.036877 19411 raft_consensus.cc:729] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:33.037184 19411 consensus_queue.cc:260] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:33.037820 19411 ts_tablet_manager.cc:1398] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:33.038137 19411 tablet_bootstrap.cc:492] T 1f01949cf70440b38e59a5020fd48c44 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:33.039134 19412 tablet_bootstrap.cc:654] T b29218a879e74895bfe00021ad358aa4 P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:33.039662 19411 tablet_bootstrap.cc:654] T 1f01949cf70440b38e59a5020fd48c44 P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:33.041410 19411 tablet_bootstrap.cc:492] T 1f01949cf70440b38e59a5020fd48c44 P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:33.041558 19411 ts_tablet_manager.cc:1367] T 1f01949cf70440b38e59a5020fd48c44 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 01:53:33.042167 19411 raft_consensus.cc:348] T 1f01949cf70440b38e59a5020fd48c44 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:33.042363 19411 raft_consensus.cc:374] T 1f01949cf70440b38e59a5020fd48c44 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:33.042441 19411 raft_consensus.cc:729] T 1f01949cf70440b38e59a5020fd48c44 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:33.042658 19411 consensus_queue.cc:260] T 1f01949cf70440b38e59a5020fd48c44 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:33.043112 19414 tablet_bootstrap.cc:492] T ed1522ec150a46f28fd553f9b563651f P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:33.043164 19411 ts_tablet_manager.cc:1398] T 1f01949cf70440b38e59a5020fd48c44 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:53:33.044096 19412 tablet_bootstrap.cc:492] T b29218a879e74895bfe00021ad358aa4 P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:33.044273 19412 ts_tablet_manager.cc:1367] T b29218a879e74895bfe00021ad358aa4 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240502 01:53:33.044555 19414 tablet_bootstrap.cc:654] T ed1522ec150a46f28fd553f9b563651f P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:33.045053 19412 raft_consensus.cc:348] T b29218a879e74895bfe00021ad358aa4 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:33.045269 19412 raft_consensus.cc:374] T b29218a879e74895bfe00021ad358aa4 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:33.045352 19412 raft_consensus.cc:729] T b29218a879e74895bfe00021ad358aa4 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:33.045600 19412 consensus_queue.cc:260] T b29218a879e74895bfe00021ad358aa4 P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:33.046137 19412 ts_tablet_manager.cc:1398] T b29218a879e74895bfe00021ad358aa4 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:53:33.047618 19414 tablet_bootstrap.cc:492] T ed1522ec150a46f28fd553f9b563651f P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:33.047883 19414 ts_tablet_manager.cc:1367] T ed1522ec150a46f28fd553f9b563651f P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 01:53:33.048774 19414 raft_consensus.cc:348] T ed1522ec150a46f28fd553f9b563651f P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:33.048942 19414 raft_consensus.cc:374] T ed1522ec150a46f28fd553f9b563651f P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:33.049036 19414 raft_consensus.cc:729] T ed1522ec150a46f28fd553f9b563651f P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:33.049285 19414 consensus_queue.cc:260] T ed1522ec150a46f28fd553f9b563651f P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:33.050350 19414 ts_tablet_manager.cc:1398] T ed1522ec150a46f28fd553f9b563651f P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:53:33.050580 19414 tablet_bootstrap.cc:492] T b29218a879e74895bfe00021ad358aa4 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:33.051803 19414 tablet_bootstrap.cc:654] T b29218a879e74895bfe00021ad358aa4 P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:33.053601 19414 tablet_bootstrap.cc:492] T b29218a879e74895bfe00021ad358aa4 P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:33.053735 19414 ts_tablet_manager.cc:1367] T b29218a879e74895bfe00021ad358aa4 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20240502 01:53:33.054416 19414 raft_consensus.cc:348] T b29218a879e74895bfe00021ad358aa4 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:33.054598 19414 raft_consensus.cc:374] T b29218a879e74895bfe00021ad358aa4 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:33.054697 19414 raft_consensus.cc:729] T b29218a879e74895bfe00021ad358aa4 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:33.054903 19414 consensus_queue.cc:260] T b29218a879e74895bfe00021ad358aa4 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:33.055398 19414 ts_tablet_manager.cc:1398] T b29218a879e74895bfe00021ad358aa4 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:53:33.067121 19349 raft_consensus.cc:482] T b29218a879e74895bfe00021ad358aa4 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:33.067247 19349 raft_consensus.cc:504] T b29218a879e74895bfe00021ad358aa4 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:33.068044 19349 leader_election.cc:288] T b29218a879e74895bfe00021ad358aa4 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:33.068382 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b29218a879e74895bfe00021ad358aa4" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:33.068466 17606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b29218a879e74895bfe00021ad358aa4" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:53:33.068567 17475 raft_consensus.cc:2454] T b29218a879e74895bfe00021ad358aa4 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:53:33.068675 17606 raft_consensus.cc:2454] T b29218a879e74895bfe00021ad358aa4 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:53:33.069000 17672 leader_election.cc:302] T b29218a879e74895bfe00021ad358aa4 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:33.069274 19349 raft_consensus.cc:2781] T b29218a879e74895bfe00021ad358aa4 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:33.069406 19349 raft_consensus.cc:482] T b29218a879e74895bfe00021ad358aa4 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:33.069500 19349 raft_consensus.cc:3037] T b29218a879e74895bfe00021ad358aa4 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:33.071427 19349 raft_consensus.cc:504] T b29218a879e74895bfe00021ad358aa4 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:33.071923 19349 leader_election.cc:288] T b29218a879e74895bfe00021ad358aa4 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Requested vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:33.072322 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b29218a879e74895bfe00021ad358aa4" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:53:33.072330 17606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b29218a879e74895bfe00021ad358aa4" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:53:33.072487 17475 raft_consensus.cc:3037] T b29218a879e74895bfe00021ad358aa4 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:33.072496 17606 raft_consensus.cc:3037] T b29218a879e74895bfe00021ad358aa4 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:33.074322 17606 raft_consensus.cc:2454] T b29218a879e74895bfe00021ad358aa4 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:53:33.074354 17475 raft_consensus.cc:2454] T b29218a879e74895bfe00021ad358aa4 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:53:33.074678 17670 leader_election.cc:302] T b29218a879e74895bfe00021ad358aa4 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:33.074980 19349 raft_consensus.cc:2781] T b29218a879e74895bfe00021ad358aa4 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:33.075103 19349 raft_consensus.cc:686] T b29218a879e74895bfe00021ad358aa4 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Becoming Leader. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Running, Role: LEADER
I20240502 01:53:33.075357 19349 consensus_queue.cc:237] T b29218a879e74895bfe00021ad358aa4 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:33.078279 17330 catalog_manager.cc:5484] T b29218a879e74895bfe00021ad358aa4 P dfa1152987644bd781b46c7e42a7a685 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfa1152987644bd781b46c7e42a7a685 (127.11.42.67). New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:33.167507 19236 raft_consensus.cc:482] T 1f01949cf70440b38e59a5020fd48c44 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:33.167690 19236 raft_consensus.cc:504] T 1f01949cf70440b38e59a5020fd48c44 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:33.168633 19236 leader_election.cc:288] T 1f01949cf70440b38e59a5020fd48c44 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:53:33.169200 17866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f01949cf70440b38e59a5020fd48c44" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:53:33.169450 17866 raft_consensus.cc:2454] T 1f01949cf70440b38e59a5020fd48c44 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:53:33.169776 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f01949cf70440b38e59a5020fd48c44" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:53:33.169934 17543 leader_election.cc:302] T 1f01949cf70440b38e59a5020fd48c44 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:53:33.170025 17737 raft_consensus.cc:2454] T 1f01949cf70440b38e59a5020fd48c44 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:53:33.170234 19236 raft_consensus.cc:2781] T 1f01949cf70440b38e59a5020fd48c44 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:33.170333 19236 raft_consensus.cc:482] T 1f01949cf70440b38e59a5020fd48c44 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:33.170429 19236 raft_consensus.cc:3037] T 1f01949cf70440b38e59a5020fd48c44 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:33.173205 19236 raft_consensus.cc:504] T 1f01949cf70440b38e59a5020fd48c44 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:33.174026 19236 leader_election.cc:288] T 1f01949cf70440b38e59a5020fd48c44 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Requested vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:53:33.174118 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f01949cf70440b38e59a5020fd48c44" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:53:33.174331 17737 raft_consensus.cc:3037] T 1f01949cf70440b38e59a5020fd48c44 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:33.174861 19400 raft_consensus.cc:482] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:33.175035 19400 raft_consensus.cc:504] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:33.175837 19400 leader_election.cc:288] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:53:33.176393 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed1522ec150a46f28fd553f9b563651f" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:53:33.176555 17737 raft_consensus.cc:2454] T 1f01949cf70440b38e59a5020fd48c44 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:53:33.176581 17735 raft_consensus.cc:2454] T ed1522ec150a46f28fd553f9b563651f P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 0.
I20240502 01:53:33.176896 17542 leader_election.cc:302] T 1f01949cf70440b38e59a5020fd48c44 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:33.177104 17866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f01949cf70440b38e59a5020fd48c44" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:53:33.177235 19339 raft_consensus.cc:2781] T 1f01949cf70440b38e59a5020fd48c44 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:33.177275 17866 raft_consensus.cc:3037] T 1f01949cf70440b38e59a5020fd48c44 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:33.177354 19339 raft_consensus.cc:686] T 1f01949cf70440b38e59a5020fd48c44 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Becoming Leader. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Running, Role: LEADER
I20240502 01:53:33.177619 19339 consensus_queue.cc:237] T 1f01949cf70440b38e59a5020fd48c44 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:33.179198 17802 leader_election.cc:302] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb31bceba76d466db24b9341c04900c3, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:33.179543 17866 raft_consensus.cc:2454] T 1f01949cf70440b38e59a5020fd48c44 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:53:33.180219 19400 raft_consensus.cc:2781] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:33.180325 19400 raft_consensus.cc:482] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:33.180393 19400 raft_consensus.cc:3037] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:33.181911 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed1522ec150a46f28fd553f9b563651f" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:33.182121 17475 raft_consensus.cc:2454] T ed1522ec150a46f28fd553f9b563651f P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 0.
I20240502 01:53:33.183058 19400 raft_consensus.cc:504] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:33.183686 19400 leader_election.cc:288] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 election: Requested vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:53:33.184155 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed1522ec150a46f28fd553f9b563651f" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:53:33.184355 17737 raft_consensus.cc:3037] T ed1522ec150a46f28fd553f9b563651f P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:33.185225 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed1522ec150a46f28fd553f9b563651f" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:53:33.185403 17475 raft_consensus.cc:3037] T ed1522ec150a46f28fd553f9b563651f P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:33.186200 17330 catalog_manager.cc:5484] T 1f01949cf70440b38e59a5020fd48c44 P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66). New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:33.187207 17475 raft_consensus.cc:2454] T ed1522ec150a46f28fd553f9b563651f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 1.
I20240502 01:53:33.188385 17802 leader_election.cc:302] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:53:33.188694 19400 raft_consensus.cc:2781] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:33.188860 19400 raft_consensus.cc:686] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Becoming Leader. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Running, Role: LEADER
I20240502 01:53:33.189167 19400 consensus_queue.cc:237] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:33.190259 17737 raft_consensus.cc:2454] T ed1522ec150a46f28fd553f9b563651f P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 1.
I20240502 01:53:33.192214 17331 catalog_manager.cc:5484] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb31bceba76d466db24b9341c04900c3 (127.11.42.68). New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } }
W20240502 01:53:33.519291 19428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:33.519641 19428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:33.527563 19428 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:53:33.535158 19349 consensus_queue.cc:1035] T b29218a879e74895bfe00021ad358aa4 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:33.542708 19437 consensus_queue.cc:1035] T b29218a879e74895bfe00021ad358aa4 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:33.557897 19342 consensus_queue.cc:1035] T 1f01949cf70440b38e59a5020fd48c44 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:33.571813 19342 consensus_queue.cc:1035] T 1f01949cf70440b38e59a5020fd48c44 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:33.586040 19258 consensus_queue.cc:1035] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:33.592533 19258 consensus_queue.cc:1035] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_11'
INSERT report
rows total: 2000
time total: 223.34 ms
time per row: 0.11167 ms
I20240502 01:53:33.943261 17330 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalance_test_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:33.952462 17586 tablet_service.cc:1467] Processing CreateTablet for tablet f67557c8f4a2434bb594b1a3d527b95a (DEFAULT_TABLE table=rebalance_test_table_12 [id=128b9c63621549098646906560fbc531]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:33.952904 17586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f67557c8f4a2434bb594b1a3d527b95a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:33.952913 17455 tablet_service.cc:1467] Processing CreateTablet for tablet f67557c8f4a2434bb594b1a3d527b95a (DEFAULT_TABLE table=rebalance_test_table_12 [id=128b9c63621549098646906560fbc531]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:33.953430 17455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f67557c8f4a2434bb594b1a3d527b95a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:33.953888 17845 tablet_service.cc:1467] Processing CreateTablet for tablet 4cd17e0649214dbcb26d76addc10ddda (DEFAULT_TABLE table=rebalance_test_table_12 [id=128b9c63621549098646906560fbc531]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:33.954298 17845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cd17e0649214dbcb26d76addc10ddda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:33.954188 17585 tablet_service.cc:1467] Processing CreateTablet for tablet 7d86e5e0319c4056bee6b2110784fd7a (DEFAULT_TABLE table=rebalance_test_table_12 [id=128b9c63621549098646906560fbc531]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:33.954542 17585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d86e5e0319c4056bee6b2110784fd7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:33.954440 17715 tablet_service.cc:1467] Processing CreateTablet for tablet f67557c8f4a2434bb594b1a3d527b95a (DEFAULT_TABLE table=rebalance_test_table_12 [id=128b9c63621549098646906560fbc531]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:33.954856 17715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f67557c8f4a2434bb594b1a3d527b95a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:33.957396 17846 tablet_service.cc:1467] Processing CreateTablet for tablet 7d86e5e0319c4056bee6b2110784fd7a (DEFAULT_TABLE table=rebalance_test_table_12 [id=128b9c63621549098646906560fbc531]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:33.957743 17846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d86e5e0319c4056bee6b2110784fd7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:33.957691 17456 tablet_service.cc:1467] Processing CreateTablet for tablet 7d86e5e0319c4056bee6b2110784fd7a (DEFAULT_TABLE table=rebalance_test_table_12 [id=128b9c63621549098646906560fbc531]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:33.958046 17456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d86e5e0319c4056bee6b2110784fd7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:33.961381 17716 tablet_service.cc:1467] Processing CreateTablet for tablet 4cd17e0649214dbcb26d76addc10ddda (DEFAULT_TABLE table=rebalance_test_table_12 [id=128b9c63621549098646906560fbc531]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:33.961776 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cd17e0649214dbcb26d76addc10ddda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:33.965377 17454 tablet_service.cc:1467] Processing CreateTablet for tablet 4cd17e0649214dbcb26d76addc10ddda (DEFAULT_TABLE table=rebalance_test_table_12 [id=128b9c63621549098646906560fbc531]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:33.965750 17454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cd17e0649214dbcb26d76addc10ddda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:33.978488 19491 tablet_bootstrap.cc:492] T 4cd17e0649214dbcb26d76addc10ddda P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:33.980160 19491 tablet_bootstrap.cc:654] T 4cd17e0649214dbcb26d76addc10ddda P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:33.982095 19494 tablet_bootstrap.cc:492] T 4cd17e0649214dbcb26d76addc10ddda P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:33.983006 19495 tablet_bootstrap.cc:492] T f67557c8f4a2434bb594b1a3d527b95a P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:33.983996 19494 tablet_bootstrap.cc:654] T 4cd17e0649214dbcb26d76addc10ddda P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:33.984630 19495 tablet_bootstrap.cc:654] T f67557c8f4a2434bb594b1a3d527b95a P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:33.988345 19491 tablet_bootstrap.cc:492] T 4cd17e0649214dbcb26d76addc10ddda P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:33.988484 19491 ts_tablet_manager.cc:1367] T 4cd17e0649214dbcb26d76addc10ddda P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240502 01:53:33.989130 19491 raft_consensus.cc:348] T 4cd17e0649214dbcb26d76addc10ddda P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:33.990774 19491 raft_consensus.cc:374] T 4cd17e0649214dbcb26d76addc10ddda P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:33.991096 19491 raft_consensus.cc:729] T 4cd17e0649214dbcb26d76addc10ddda P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:33.991561 19491 consensus_queue.cc:260] T 4cd17e0649214dbcb26d76addc10ddda P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:33.992236 19492 tablet_bootstrap.cc:492] T 7d86e5e0319c4056bee6b2110784fd7a P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:33.992863 19495 tablet_bootstrap.cc:492] T f67557c8f4a2434bb594b1a3d527b95a P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:33.993081 19495 ts_tablet_manager.cc:1367] T f67557c8f4a2434bb594b1a3d527b95a P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240502 01:53:33.993649 19495 raft_consensus.cc:348] T f67557c8f4a2434bb594b1a3d527b95a P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:33.993806 19495 raft_consensus.cc:374] T f67557c8f4a2434bb594b1a3d527b95a P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:33.994652 19492 tablet_bootstrap.cc:654] T 7d86e5e0319c4056bee6b2110784fd7a P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:33.994917 19491 ts_tablet_manager.cc:1398] T 4cd17e0649214dbcb26d76addc10ddda P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 01:53:33.995254 19491 tablet_bootstrap.cc:492] T 7d86e5e0319c4056bee6b2110784fd7a P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:33.997327 19495 raft_consensus.cc:729] T f67557c8f4a2434bb594b1a3d527b95a P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:33.997642 19495 consensus_queue.cc:260] T f67557c8f4a2434bb594b1a3d527b95a P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:33.999011 19495 ts_tablet_manager.cc:1398] T f67557c8f4a2434bb594b1a3d527b95a P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 01:53:33.999328 19495 tablet_bootstrap.cc:492] T 7d86e5e0319c4056bee6b2110784fd7a P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:34.000155 19494 tablet_bootstrap.cc:492] T 4cd17e0649214dbcb26d76addc10ddda P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:34.000329 19494 ts_tablet_manager.cc:1367] T 4cd17e0649214dbcb26d76addc10ddda P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.012s
I20240502 01:53:34.001047 19494 raft_consensus.cc:348] T 4cd17e0649214dbcb26d76addc10ddda P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:34.001240 19494 raft_consensus.cc:374] T 4cd17e0649214dbcb26d76addc10ddda P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:34.001379 19494 raft_consensus.cc:729] T 4cd17e0649214dbcb26d76addc10ddda P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:34.001399 19491 tablet_bootstrap.cc:654] T 7d86e5e0319c4056bee6b2110784fd7a P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:34.001633 19494 consensus_queue.cc:260] T 4cd17e0649214dbcb26d76addc10ddda P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:34.003054 19494 ts_tablet_manager.cc:1398] T 4cd17e0649214dbcb26d76addc10ddda P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 01:53:34.003343 19494 tablet_bootstrap.cc:492] T f67557c8f4a2434bb594b1a3d527b95a P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:34.004289 19495 tablet_bootstrap.cc:654] T 7d86e5e0319c4056bee6b2110784fd7a P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:34.005671 19494 tablet_bootstrap.cc:654] T f67557c8f4a2434bb594b1a3d527b95a P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:34.008950 19491 tablet_bootstrap.cc:492] T 7d86e5e0319c4056bee6b2110784fd7a P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:34.009218 19491 ts_tablet_manager.cc:1367] T 7d86e5e0319c4056bee6b2110784fd7a P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.003s
I20240502 01:53:34.010506 19495 tablet_bootstrap.cc:492] T 7d86e5e0319c4056bee6b2110784fd7a P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:34.010655 19495 ts_tablet_manager.cc:1367] T 7d86e5e0319c4056bee6b2110784fd7a P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.001s
I20240502 01:53:34.011225 19495 raft_consensus.cc:348] T 7d86e5e0319c4056bee6b2110784fd7a P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:34.011391 19495 raft_consensus.cc:374] T 7d86e5e0319c4056bee6b2110784fd7a P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:34.011466 19495 raft_consensus.cc:729] T 7d86e5e0319c4056bee6b2110784fd7a P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:34.011695 19495 consensus_queue.cc:260] T 7d86e5e0319c4056bee6b2110784fd7a P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:34.012326 19495 ts_tablet_manager.cc:1398] T 7d86e5e0319c4056bee6b2110784fd7a P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:53:34.010067 19491 raft_consensus.cc:348] T 7d86e5e0319c4056bee6b2110784fd7a P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:34.012897 19492 tablet_bootstrap.cc:492] T 7d86e5e0319c4056bee6b2110784fd7a P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:34.012897 19491 raft_consensus.cc:374] T 7d86e5e0319c4056bee6b2110784fd7a P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:34.013082 19492 ts_tablet_manager.cc:1367] T 7d86e5e0319c4056bee6b2110784fd7a P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.000s
I20240502 01:53:34.013099 19491 raft_consensus.cc:729] T 7d86e5e0319c4056bee6b2110784fd7a P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:34.013427 19491 consensus_queue.cc:260] T 7d86e5e0319c4056bee6b2110784fd7a P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:34.013648 19494 tablet_bootstrap.cc:492] T f67557c8f4a2434bb594b1a3d527b95a P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:34.013870 19494 ts_tablet_manager.cc:1367] T f67557c8f4a2434bb594b1a3d527b95a P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20240502 01:53:34.013890 19492 raft_consensus.cc:348] T 7d86e5e0319c4056bee6b2110784fd7a P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:34.014089 19492 raft_consensus.cc:374] T 7d86e5e0319c4056bee6b2110784fd7a P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:34.014184 19492 raft_consensus.cc:729] T 7d86e5e0319c4056bee6b2110784fd7a P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:34.014410 19492 consensus_queue.cc:260] T 7d86e5e0319c4056bee6b2110784fd7a P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:34.014837 19491 ts_tablet_manager.cc:1398] T 7d86e5e0319c4056bee6b2110784fd7a P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 01:53:34.014864 19494 raft_consensus.cc:348] T f67557c8f4a2434bb594b1a3d527b95a P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:34.015074 19494 raft_consensus.cc:374] T f67557c8f4a2434bb594b1a3d527b95a P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:34.015153 19494 raft_consensus.cc:729] T f67557c8f4a2434bb594b1a3d527b95a P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:34.015372 19494 consensus_queue.cc:260] T f67557c8f4a2434bb594b1a3d527b95a P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:34.015846 19492 ts_tablet_manager.cc:1398] T 7d86e5e0319c4056bee6b2110784fd7a P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:53:34.016098 19492 tablet_bootstrap.cc:492] T f67557c8f4a2434bb594b1a3d527b95a P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:34.016534 19494 ts_tablet_manager.cc:1398] T f67557c8f4a2434bb594b1a3d527b95a P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:53:34.018313 19492 tablet_bootstrap.cc:654] T f67557c8f4a2434bb594b1a3d527b95a P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:34.020838 19492 tablet_bootstrap.cc:492] T f67557c8f4a2434bb594b1a3d527b95a P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:34.021062 19492 ts_tablet_manager.cc:1367] T f67557c8f4a2434bb594b1a3d527b95a P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 01:53:34.021803 19492 raft_consensus.cc:348] T f67557c8f4a2434bb594b1a3d527b95a P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:34.022063 19492 raft_consensus.cc:374] T f67557c8f4a2434bb594b1a3d527b95a P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:34.022187 19492 raft_consensus.cc:729] T f67557c8f4a2434bb594b1a3d527b95a P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:34.022505 19492 consensus_queue.cc:260] T f67557c8f4a2434bb594b1a3d527b95a P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:34.023805 19492 ts_tablet_manager.cc:1398] T f67557c8f4a2434bb594b1a3d527b95a P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 01:53:34.024061 19492 tablet_bootstrap.cc:492] T 4cd17e0649214dbcb26d76addc10ddda P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:34.025223 19492 tablet_bootstrap.cc:654] T 4cd17e0649214dbcb26d76addc10ddda P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:34.028554 19492 tablet_bootstrap.cc:492] T 4cd17e0649214dbcb26d76addc10ddda P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:34.028674 19492 ts_tablet_manager.cc:1367] T 4cd17e0649214dbcb26d76addc10ddda P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240502 01:53:34.029248 19492 raft_consensus.cc:348] T 4cd17e0649214dbcb26d76addc10ddda P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:34.029397 19492 raft_consensus.cc:374] T 4cd17e0649214dbcb26d76addc10ddda P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:34.029490 19492 raft_consensus.cc:729] T 4cd17e0649214dbcb26d76addc10ddda P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:34.029664 19492 consensus_queue.cc:260] T 4cd17e0649214dbcb26d76addc10ddda P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:34.030148 19492 ts_tablet_manager.cc:1398] T 4cd17e0649214dbcb26d76addc10ddda P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:53:34.035884 19349 raft_consensus.cc:482] T 4cd17e0649214dbcb26d76addc10ddda P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:34.036011 19349 raft_consensus.cc:504] T 4cd17e0649214dbcb26d76addc10ddda P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:34.036649 19349 leader_election.cc:288] T 4cd17e0649214dbcb26d76addc10ddda P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:34.036994 17865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd17e0649214dbcb26d76addc10ddda" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:53:34.037092 17474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd17e0649214dbcb26d76addc10ddda" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:34.037209 17865 raft_consensus.cc:2454] T 4cd17e0649214dbcb26d76addc10ddda P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:53:34.037285 17474 raft_consensus.cc:2454] T 4cd17e0649214dbcb26d76addc10ddda P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:53:34.037637 17673 leader_election.cc:302] T 4cd17e0649214dbcb26d76addc10ddda P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb31bceba76d466db24b9341c04900c3, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:34.038264 19349 raft_consensus.cc:2781] T 4cd17e0649214dbcb26d76addc10ddda P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:34.038385 19349 raft_consensus.cc:482] T 4cd17e0649214dbcb26d76addc10ddda P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:34.038486 19349 raft_consensus.cc:3037] T 4cd17e0649214dbcb26d76addc10ddda P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:34.040321 19349 raft_consensus.cc:504] T 4cd17e0649214dbcb26d76addc10ddda P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:34.040802 19349 leader_election.cc:288] T 4cd17e0649214dbcb26d76addc10ddda P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Requested vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:34.041177 17474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd17e0649214dbcb26d76addc10ddda" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:53:34.041234 17865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd17e0649214dbcb26d76addc10ddda" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:53:34.041335 17474 raft_consensus.cc:3037] T 4cd17e0649214dbcb26d76addc10ddda P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:34.041411 17865 raft_consensus.cc:3037] T 4cd17e0649214dbcb26d76addc10ddda P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:34.043053 17474 raft_consensus.cc:2454] T 4cd17e0649214dbcb26d76addc10ddda P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:53:34.043140 17865 raft_consensus.cc:2454] T 4cd17e0649214dbcb26d76addc10ddda P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:53:34.043449 17672 leader_election.cc:302] T 4cd17e0649214dbcb26d76addc10ddda P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:34.043737 19349 raft_consensus.cc:2781] T 4cd17e0649214dbcb26d76addc10ddda P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:34.043879 19349 raft_consensus.cc:686] T 4cd17e0649214dbcb26d76addc10ddda P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Becoming Leader. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Running, Role: LEADER
I20240502 01:53:34.044100 19349 consensus_queue.cc:237] T 4cd17e0649214dbcb26d76addc10ddda P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:34.046959 17335 catalog_manager.cc:5484] T 4cd17e0649214dbcb26d76addc10ddda P dfa1152987644bd781b46c7e42a7a685 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfa1152987644bd781b46c7e42a7a685 (127.11.42.67). New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:34.114465 19426 raft_consensus.cc:482] T f67557c8f4a2434bb594b1a3d527b95a P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:34.114621 19426 raft_consensus.cc:504] T f67557c8f4a2434bb594b1a3d527b95a P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:34.115347 19426 leader_election.cc:288] T f67557c8f4a2434bb594b1a3d527b95a P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:53:34.115712 17474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f67557c8f4a2434bb594b1a3d527b95a" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:34.115837 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f67557c8f4a2434bb594b1a3d527b95a" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:53:34.115931 17474 raft_consensus.cc:2454] T f67557c8f4a2434bb594b1a3d527b95a P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:53:34.116048 17735 raft_consensus.cc:2454] T f67557c8f4a2434bb594b1a3d527b95a P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:53:34.116331 17542 leader_election.cc:302] T f67557c8f4a2434bb594b1a3d527b95a P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, 4da690c0fa13407a85c0689ce8de9ade; no voters:
I20240502 01:53:34.116597 19426 raft_consensus.cc:2781] T f67557c8f4a2434bb594b1a3d527b95a P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:34.116736 19426 raft_consensus.cc:482] T f67557c8f4a2434bb594b1a3d527b95a P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:34.116832 19426 raft_consensus.cc:3037] T f67557c8f4a2434bb594b1a3d527b95a P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:34.118788 19426 raft_consensus.cc:504] T f67557c8f4a2434bb594b1a3d527b95a P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:34.119293 19426 leader_election.cc:288] T f67557c8f4a2434bb594b1a3d527b95a P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Requested vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:53:34.119792 17474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f67557c8f4a2434bb594b1a3d527b95a" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:53:34.119988 17474 raft_consensus.cc:3037] T f67557c8f4a2434bb594b1a3d527b95a P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:34.121187 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f67557c8f4a2434bb594b1a3d527b95a" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:53:34.121353 17735 raft_consensus.cc:3037] T f67557c8f4a2434bb594b1a3d527b95a P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:34.122776 17474 raft_consensus.cc:2454] T f67557c8f4a2434bb594b1a3d527b95a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:53:34.123174 17542 leader_election.cc:302] T f67557c8f4a2434bb594b1a3d527b95a P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, 4da690c0fa13407a85c0689ce8de9ade; no voters:
I20240502 01:53:34.123258 17735 raft_consensus.cc:2454] T f67557c8f4a2434bb594b1a3d527b95a P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:53:34.123452 19426 raft_consensus.cc:2781] T f67557c8f4a2434bb594b1a3d527b95a P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:34.123574 19426 raft_consensus.cc:686] T f67557c8f4a2434bb594b1a3d527b95a P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Becoming Leader. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Running, Role: LEADER
I20240502 01:53:34.123848 19426 consensus_queue.cc:237] T f67557c8f4a2434bb594b1a3d527b95a P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:34.127177 17335 catalog_manager.cc:5484] T f67557c8f4a2434bb594b1a3d527b95a P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66). New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:34.259503 19432 raft_consensus.cc:482] T 7d86e5e0319c4056bee6b2110784fd7a P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:34.259716 19432 raft_consensus.cc:504] T 7d86e5e0319c4056bee6b2110784fd7a P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:34.260401 19432 leader_election.cc:288] T 7d86e5e0319c4056bee6b2110784fd7a P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:34.260756 17865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d86e5e0319c4056bee6b2110784fd7a" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:53:34.260771 17606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d86e5e0319c4056bee6b2110784fd7a" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:53:34.260975 17606 raft_consensus.cc:2454] T 7d86e5e0319c4056bee6b2110784fd7a P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 0.
I20240502 01:53:34.260974 17865 raft_consensus.cc:2454] T 7d86e5e0319c4056bee6b2110784fd7a P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 0.
I20240502 01:53:34.261368 17412 leader_election.cc:302] T 7d86e5e0319c4056bee6b2110784fd7a P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:53:34.261641 19432 raft_consensus.cc:2781] T 7d86e5e0319c4056bee6b2110784fd7a P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:34.261755 19432 raft_consensus.cc:482] T 7d86e5e0319c4056bee6b2110784fd7a P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:34.261844 19432 raft_consensus.cc:3037] T 7d86e5e0319c4056bee6b2110784fd7a P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:34.263876 19432 raft_consensus.cc:504] T 7d86e5e0319c4056bee6b2110784fd7a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:34.264422 19432 leader_election.cc:288] T 7d86e5e0319c4056bee6b2110784fd7a P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Requested vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:34.264816 17865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d86e5e0319c4056bee6b2110784fd7a" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:53:34.264925 17606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d86e5e0319c4056bee6b2110784fd7a" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:53:34.265065 17865 raft_consensus.cc:3037] T 7d86e5e0319c4056bee6b2110784fd7a P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:34.265120 17606 raft_consensus.cc:3037] T 7d86e5e0319c4056bee6b2110784fd7a P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:34.267724 17865 raft_consensus.cc:2454] T 7d86e5e0319c4056bee6b2110784fd7a P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
I20240502 01:53:34.267768 17606 raft_consensus.cc:2454] T 7d86e5e0319c4056bee6b2110784fd7a P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
I20240502 01:53:34.268100 17412 leader_election.cc:302] T 7d86e5e0319c4056bee6b2110784fd7a P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:53:34.268368 19432 raft_consensus.cc:2781] T 7d86e5e0319c4056bee6b2110784fd7a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:34.268498 19432 raft_consensus.cc:686] T 7d86e5e0319c4056bee6b2110784fd7a P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Becoming Leader. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Running, Role: LEADER
I20240502 01:53:34.268776 19432 consensus_queue.cc:237] T 7d86e5e0319c4056bee6b2110784fd7a P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:34.271884 17335 catalog_manager.cc:5484] T 7d86e5e0319c4056bee6b2110784fd7a P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: term changed from 0 to 1, leader changed from <none> to 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65). New cstate: current_term: 1 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } }
I20240502 01:53:34.560073 19426 consensus_queue.cc:1035] T f67557c8f4a2434bb594b1a3d527b95a P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:34.586737 19236 consensus_queue.cc:1035] T f67557c8f4a2434bb594b1a3d527b95a P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:34.592866 19437 consensus_queue.cc:1035] T 4cd17e0649214dbcb26d76addc10ddda P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:34.599049 19437 consensus_queue.cc:1035] T 4cd17e0649214dbcb26d76addc10ddda P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:34.676983 19507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:34.677372 19507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:34.684911 19507 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_12'
I20240502 01:53:34.732385 19255 consensus_queue.cc:1035] T 7d86e5e0319c4056bee6b2110784fd7a P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:34.740208 19540 consensus_queue.cc:1035] T 7d86e5e0319c4056bee6b2110784fd7a P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 228.989 ms
time per row: 0.114495 ms
I20240502 01:53:35.070029 17334 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalance_test_table_13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:35.079737 17586 tablet_service.cc:1467] Processing CreateTablet for tablet 4aefc61cb95f4e949e7a23a1c677749e (DEFAULT_TABLE table=rebalance_test_table_13 [id=dbf97af96f07462286eb1fabeddf25fa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:35.079993 17845 tablet_service.cc:1467] Processing CreateTablet for tablet 4aefc61cb95f4e949e7a23a1c677749e (DEFAULT_TABLE table=rebalance_test_table_13 [id=dbf97af96f07462286eb1fabeddf25fa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:35.080220 17586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aefc61cb95f4e949e7a23a1c677749e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:35.080452 17845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aefc61cb95f4e949e7a23a1c677749e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:35.080482 17716 tablet_service.cc:1467] Processing CreateTablet for tablet 4aefc61cb95f4e949e7a23a1c677749e (DEFAULT_TABLE table=rebalance_test_table_13 [id=dbf97af96f07462286eb1fabeddf25fa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:35.080927 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aefc61cb95f4e949e7a23a1c677749e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:35.081409 17585 tablet_service.cc:1467] Processing CreateTablet for tablet 1fdd104f1a28446cbc25d4ca16d03d7d (DEFAULT_TABLE table=rebalance_test_table_13 [id=dbf97af96f07462286eb1fabeddf25fa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:35.081807 17585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fdd104f1a28446cbc25d4ca16d03d7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:35.082445 17717 tablet_service.cc:1467] Processing CreateTablet for tablet 52430bbf347a4c5191edb23e1aa5d385 (DEFAULT_TABLE table=rebalance_test_table_13 [id=dbf97af96f07462286eb1fabeddf25fa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:35.082820 17717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52430bbf347a4c5191edb23e1aa5d385. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:35.086926 17846 tablet_service.cc:1467] Processing CreateTablet for tablet 1fdd104f1a28446cbc25d4ca16d03d7d (DEFAULT_TABLE table=rebalance_test_table_13 [id=dbf97af96f07462286eb1fabeddf25fa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:35.087342 17846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fdd104f1a28446cbc25d4ca16d03d7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:35.089437 17715 tablet_service.cc:1467] Processing CreateTablet for tablet 1fdd104f1a28446cbc25d4ca16d03d7d (DEFAULT_TABLE table=rebalance_test_table_13 [id=dbf97af96f07462286eb1fabeddf25fa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:35.089838 17715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fdd104f1a28446cbc25d4ca16d03d7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:35.089866 17456 tablet_service.cc:1467] Processing CreateTablet for tablet 52430bbf347a4c5191edb23e1aa5d385 (DEFAULT_TABLE table=rebalance_test_table_13 [id=dbf97af96f07462286eb1fabeddf25fa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:35.090294 17456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52430bbf347a4c5191edb23e1aa5d385. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:35.091091 19576 tablet_bootstrap.cc:492] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:35.093966 19576 tablet_bootstrap.cc:654] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:35.099061 17585 tablet_service.cc:1467] Processing CreateTablet for tablet 52430bbf347a4c5191edb23e1aa5d385 (DEFAULT_TABLE table=rebalance_test_table_13 [id=dbf97af96f07462286eb1fabeddf25fa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:35.099452 17585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52430bbf347a4c5191edb23e1aa5d385. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:35.105535 19579 tablet_bootstrap.cc:492] T 52430bbf347a4c5191edb23e1aa5d385 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:35.107415 19575 tablet_bootstrap.cc:492] T 4aefc61cb95f4e949e7a23a1c677749e P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:35.107564 19579 tablet_bootstrap.cc:654] T 52430bbf347a4c5191edb23e1aa5d385 P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:35.108966 19576 tablet_bootstrap.cc:492] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:35.109210 19576 ts_tablet_manager.cc:1367] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.008s
I20240502 01:53:35.109324 19575 tablet_bootstrap.cc:654] T 4aefc61cb95f4e949e7a23a1c677749e P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:35.110061 19576 raft_consensus.cc:348] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:35.110399 19576 raft_consensus.cc:374] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:35.110551 19576 raft_consensus.cc:729] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:35.110877 19576 consensus_queue.cc:260] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:35.112565 19576 ts_tablet_manager.cc:1398] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:35.112871 19576 tablet_bootstrap.cc:492] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:35.112959 19577 tablet_bootstrap.cc:492] T 4aefc61cb95f4e949e7a23a1c677749e P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:35.114670 19577 tablet_bootstrap.cc:654] T 4aefc61cb95f4e949e7a23a1c677749e P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:35.115294 19576 tablet_bootstrap.cc:654] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:35.117331 19579 tablet_bootstrap.cc:492] T 52430bbf347a4c5191edb23e1aa5d385 P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:35.117482 19579 ts_tablet_manager.cc:1367] T 52430bbf347a4c5191edb23e1aa5d385 P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20240502 01:53:35.118304 19579 raft_consensus.cc:348] T 52430bbf347a4c5191edb23e1aa5d385 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:35.118521 19579 raft_consensus.cc:374] T 52430bbf347a4c5191edb23e1aa5d385 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:35.118608 19579 raft_consensus.cc:729] T 52430bbf347a4c5191edb23e1aa5d385 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:35.118826 19579 consensus_queue.cc:260] T 52430bbf347a4c5191edb23e1aa5d385 P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:35.119457 19579 ts_tablet_manager.cc:1398] T 52430bbf347a4c5191edb23e1aa5d385 P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:53:35.120246 19575 tablet_bootstrap.cc:492] T 4aefc61cb95f4e949e7a23a1c677749e P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:35.120467 19575 ts_tablet_manager.cc:1367] T 4aefc61cb95f4e949e7a23a1c677749e P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20240502 01:53:35.122026 19575 raft_consensus.cc:348] T 4aefc61cb95f4e949e7a23a1c677749e P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:35.122392 19575 raft_consensus.cc:374] T 4aefc61cb95f4e949e7a23a1c677749e P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:35.122542 19575 raft_consensus.cc:729] T 4aefc61cb95f4e949e7a23a1c677749e P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:35.122730 19577 tablet_bootstrap.cc:492] T 4aefc61cb95f4e949e7a23a1c677749e P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:35.122915 19577 ts_tablet_manager.cc:1367] T 4aefc61cb95f4e949e7a23a1c677749e P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20240502 01:53:35.122818 19575 consensus_queue.cc:260] T 4aefc61cb95f4e949e7a23a1c677749e P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:35.123459 19575 ts_tablet_manager.cc:1398] T 4aefc61cb95f4e949e7a23a1c677749e P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:53:35.124615 19575 tablet_bootstrap.cc:492] T 1fdd104f1a28446cbc25d4ca16d03d7d P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:35.124650 19576 tablet_bootstrap.cc:492] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:35.124558 19577 raft_consensus.cc:348] T 4aefc61cb95f4e949e7a23a1c677749e P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:35.124821 19576 ts_tablet_manager.cc:1367] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20240502 01:53:35.124845 19577 raft_consensus.cc:374] T 4aefc61cb95f4e949e7a23a1c677749e P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:35.125072 19577 raft_consensus.cc:729] T 4aefc61cb95f4e949e7a23a1c677749e P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:35.125402 19577 consensus_queue.cc:260] T 4aefc61cb95f4e949e7a23a1c677749e P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:35.125617 19576 raft_consensus.cc:348] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:35.125833 19576 raft_consensus.cc:374] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:35.125917 19576 raft_consensus.cc:729] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:35.126134 19576 consensus_queue.cc:260] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:35.126704 19576 ts_tablet_manager.cc:1398] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:53:35.127017 19576 tablet_bootstrap.cc:492] T 52430bbf347a4c5191edb23e1aa5d385 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:35.127334 19575 tablet_bootstrap.cc:654] T 1fdd104f1a28446cbc25d4ca16d03d7d P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:35.127614 19577 ts_tablet_manager.cc:1398] T 4aefc61cb95f4e949e7a23a1c677749e P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 01:53:35.127909 19577 tablet_bootstrap.cc:492] T 52430bbf347a4c5191edb23e1aa5d385 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:35.128521 19576 tablet_bootstrap.cc:654] T 52430bbf347a4c5191edb23e1aa5d385 P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:35.129348 19577 tablet_bootstrap.cc:654] T 52430bbf347a4c5191edb23e1aa5d385 P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:35.133991 19576 tablet_bootstrap.cc:492] T 52430bbf347a4c5191edb23e1aa5d385 P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:35.134124 19576 ts_tablet_manager.cc:1367] T 52430bbf347a4c5191edb23e1aa5d385 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20240502 01:53:35.134766 19577 tablet_bootstrap.cc:492] T 52430bbf347a4c5191edb23e1aa5d385 P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:35.134816 19576 raft_consensus.cc:348] T 52430bbf347a4c5191edb23e1aa5d385 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:35.134922 19577 ts_tablet_manager.cc:1367] T 52430bbf347a4c5191edb23e1aa5d385 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20240502 01:53:35.134991 19576 raft_consensus.cc:374] T 52430bbf347a4c5191edb23e1aa5d385 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:35.135068 19576 raft_consensus.cc:729] T 52430bbf347a4c5191edb23e1aa5d385 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:35.135269 19576 consensus_queue.cc:260] T 52430bbf347a4c5191edb23e1aa5d385 P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:35.136756 19576 ts_tablet_manager.cc:1398] T 52430bbf347a4c5191edb23e1aa5d385 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:53:35.137831 19575 tablet_bootstrap.cc:492] T 1fdd104f1a28446cbc25d4ca16d03d7d P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:35.138011 19575 ts_tablet_manager.cc:1367] T 1fdd104f1a28446cbc25d4ca16d03d7d P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.003s
I20240502 01:53:35.138700 19575 raft_consensus.cc:348] T 1fdd104f1a28446cbc25d4ca16d03d7d P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:35.138895 19575 raft_consensus.cc:374] T 1fdd104f1a28446cbc25d4ca16d03d7d P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:35.138962 19575 raft_consensus.cc:729] T 1fdd104f1a28446cbc25d4ca16d03d7d P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:35.139151 19575 consensus_queue.cc:260] T 1fdd104f1a28446cbc25d4ca16d03d7d P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:35.139626 19575 ts_tablet_manager.cc:1398] T 1fdd104f1a28446cbc25d4ca16d03d7d P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:53:35.139685 19577 raft_consensus.cc:348] T 52430bbf347a4c5191edb23e1aa5d385 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:35.139904 19577 raft_consensus.cc:374] T 52430bbf347a4c5191edb23e1aa5d385 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:35.139973 19577 raft_consensus.cc:729] T 52430bbf347a4c5191edb23e1aa5d385 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:35.140184 19577 consensus_queue.cc:260] T 52430bbf347a4c5191edb23e1aa5d385 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:35.142848 19577 ts_tablet_manager.cc:1398] T 52430bbf347a4c5191edb23e1aa5d385 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 01:53:35.143420 19577 tablet_bootstrap.cc:492] T 1fdd104f1a28446cbc25d4ca16d03d7d P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:35.145633 19577 tablet_bootstrap.cc:654] T 1fdd104f1a28446cbc25d4ca16d03d7d P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:35.148957 19577 tablet_bootstrap.cc:492] T 1fdd104f1a28446cbc25d4ca16d03d7d P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:35.149103 19577 ts_tablet_manager.cc:1367] T 1fdd104f1a28446cbc25d4ca16d03d7d P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240502 01:53:35.149703 19577 raft_consensus.cc:348] T 1fdd104f1a28446cbc25d4ca16d03d7d P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:35.149883 19577 raft_consensus.cc:374] T 1fdd104f1a28446cbc25d4ca16d03d7d P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:35.149982 19577 raft_consensus.cc:729] T 1fdd104f1a28446cbc25d4ca16d03d7d P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:35.150197 19577 consensus_queue.cc:260] T 1fdd104f1a28446cbc25d4ca16d03d7d P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:35.150702 19577 ts_tablet_manager.cc:1398] T 1fdd104f1a28446cbc25d4ca16d03d7d P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:53:35.208964 19426 raft_consensus.cc:482] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:35.209151 19426 raft_consensus.cc:504] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:35.209858 19426 leader_election.cc:288] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:53:35.210239 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aefc61cb95f4e949e7a23a1c677749e" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:53:35.210480 17735 raft_consensus.cc:2454] T 4aefc61cb95f4e949e7a23a1c677749e P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:53:35.210489 17865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aefc61cb95f4e949e7a23a1c677749e" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:53:35.210701 17865 raft_consensus.cc:2454] T 4aefc61cb95f4e949e7a23a1c677749e P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:53:35.210906 17542 leader_election.cc:302] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:35.211187 19426 raft_consensus.cc:2781] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:35.211306 19426 raft_consensus.cc:482] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:35.211400 19426 raft_consensus.cc:3037] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:35.213387 19426 raft_consensus.cc:504] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:35.213923 19426 leader_election.cc:288] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Requested vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:53:35.214385 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aefc61cb95f4e949e7a23a1c677749e" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:53:35.214401 17865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aefc61cb95f4e949e7a23a1c677749e" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:53:35.214583 17737 raft_consensus.cc:3037] T 4aefc61cb95f4e949e7a23a1c677749e P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:35.214586 17865 raft_consensus.cc:3037] T 4aefc61cb95f4e949e7a23a1c677749e P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:35.216681 17865 raft_consensus.cc:2454] T 4aefc61cb95f4e949e7a23a1c677749e P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:53:35.216804 17737 raft_consensus.cc:2454] T 4aefc61cb95f4e949e7a23a1c677749e P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:53:35.217128 17543 leader_election.cc:302] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:53:35.217550 19426 raft_consensus.cc:2781] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:35.217710 19426 raft_consensus.cc:686] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Becoming Leader. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Running, Role: LEADER
I20240502 01:53:35.217957 19426 consensus_queue.cc:237] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:35.218714 19511 raft_consensus.cc:482] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:35.218866 19511 raft_consensus.cc:504] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:35.219682 19511 leader_election.cc:288] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:53:35.219998 17865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdd104f1a28446cbc25d4ca16d03d7d" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:53:35.220219 17865 raft_consensus.cc:2454] T 1fdd104f1a28446cbc25d4ca16d03d7d P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:53:35.220393 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdd104f1a28446cbc25d4ca16d03d7d" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:53:35.220547 17543 leader_election.cc:302] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:53:35.220599 17737 raft_consensus.cc:2454] T 1fdd104f1a28446cbc25d4ca16d03d7d P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:53:35.220824 19511 raft_consensus.cc:2781] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:35.220916 19511 raft_consensus.cc:482] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:35.220999 19511 raft_consensus.cc:3037] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:35.221567 17335 catalog_manager.cc:5484] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66). New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:35.223392 19511 raft_consensus.cc:504] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:35.224112 19511 leader_election.cc:288] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Requested vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:53:35.224220 17865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdd104f1a28446cbc25d4ca16d03d7d" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:53:35.224432 17865 raft_consensus.cc:3037] T 1fdd104f1a28446cbc25d4ca16d03d7d P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:35.226347 17865 raft_consensus.cc:2454] T 1fdd104f1a28446cbc25d4ca16d03d7d P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:53:35.226683 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdd104f1a28446cbc25d4ca16d03d7d" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:53:35.226699 17543 leader_election.cc:302] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:53:35.226881 17737 raft_consensus.cc:3037] T 1fdd104f1a28446cbc25d4ca16d03d7d P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:35.226974 19511 raft_consensus.cc:2781] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:35.227114 19511 raft_consensus.cc:686] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Becoming Leader. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Running, Role: LEADER
I20240502 01:53:35.227545 19511 consensus_queue.cc:237] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:35.228868 17737 raft_consensus.cc:2454] T 1fdd104f1a28446cbc25d4ca16d03d7d P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:53:35.230564 17335 catalog_manager.cc:5484] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66). New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:35.347319 19349 raft_consensus.cc:482] T 52430bbf347a4c5191edb23e1aa5d385 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:35.347506 19349 raft_consensus.cc:504] T 52430bbf347a4c5191edb23e1aa5d385 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:35.348100 19349 leader_election.cc:288] T 52430bbf347a4c5191edb23e1aa5d385 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:35.348433 17474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52430bbf347a4c5191edb23e1aa5d385" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:35.348601 17605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52430bbf347a4c5191edb23e1aa5d385" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:53:35.348717 17474 raft_consensus.cc:2454] T 52430bbf347a4c5191edb23e1aa5d385 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:53:35.348837 17605 raft_consensus.cc:2454] T 52430bbf347a4c5191edb23e1aa5d385 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:53:35.349184 17672 leader_election.cc:302] T 52430bbf347a4c5191edb23e1aa5d385 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:35.349527 19349 raft_consensus.cc:2781] T 52430bbf347a4c5191edb23e1aa5d385 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:35.349645 19349 raft_consensus.cc:482] T 52430bbf347a4c5191edb23e1aa5d385 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:35.349773 19349 raft_consensus.cc:3037] T 52430bbf347a4c5191edb23e1aa5d385 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:35.351650 19349 raft_consensus.cc:504] T 52430bbf347a4c5191edb23e1aa5d385 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:35.352253 19349 leader_election.cc:288] T 52430bbf347a4c5191edb23e1aa5d385 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Requested vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:35.352504 17474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52430bbf347a4c5191edb23e1aa5d385" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:53:35.352677 17474 raft_consensus.cc:3037] T 52430bbf347a4c5191edb23e1aa5d385 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:35.352646 17605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52430bbf347a4c5191edb23e1aa5d385" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:53:35.352807 17605 raft_consensus.cc:3037] T 52430bbf347a4c5191edb23e1aa5d385 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:35.354626 17605 raft_consensus.cc:2454] T 52430bbf347a4c5191edb23e1aa5d385 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:53:35.354681 17474 raft_consensus.cc:2454] T 52430bbf347a4c5191edb23e1aa5d385 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:53:35.355018 17670 leader_election.cc:302] T 52430bbf347a4c5191edb23e1aa5d385 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:35.355278 19349 raft_consensus.cc:2781] T 52430bbf347a4c5191edb23e1aa5d385 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:35.355432 19349 raft_consensus.cc:686] T 52430bbf347a4c5191edb23e1aa5d385 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Becoming Leader. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Running, Role: LEADER
I20240502 01:53:35.355684 19349 consensus_queue.cc:237] T 52430bbf347a4c5191edb23e1aa5d385 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:35.358615 17335 catalog_manager.cc:5484] T 52430bbf347a4c5191edb23e1aa5d385 P dfa1152987644bd781b46c7e42a7a685 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfa1152987644bd781b46c7e42a7a685 (127.11.42.67). New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:35.605377 19511 consensus_queue.cc:1035] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:35.611174 19511 consensus_queue.cc:1035] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:35.627684 19426 consensus_queue.cc:1035] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:35.633199 19426 consensus_queue.cc:1035] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:35.730379 19591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:35.730646 19591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:35.738338 19591 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:53:35.750510 19437 consensus_queue.cc:1035] T 52430bbf347a4c5191edb23e1aa5d385 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:53:35.755805 19349 consensus_queue.cc:1035] T 52430bbf347a4c5191edb23e1aa5d385 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_13'
INSERT report
rows total: 2000
time total: 201.764 ms
time per row: 0.100882 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 01:53:36.964512 17335 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalance_test_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:36.978969 17456 tablet_service.cc:1467] Processing CreateTablet for tablet a12dd8d5f6164fb8a77d6a197aa0eb7e (DEFAULT_TABLE table=rebalance_test_table_14 [id=a07db144a5294a4dbfc503167983593e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:36.978998 17845 tablet_service.cc:1467] Processing CreateTablet for tablet a12dd8d5f6164fb8a77d6a197aa0eb7e (DEFAULT_TABLE table=rebalance_test_table_14 [id=a07db144a5294a4dbfc503167983593e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:36.979436 17456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a12dd8d5f6164fb8a77d6a197aa0eb7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:36.979467 17845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a12dd8d5f6164fb8a77d6a197aa0eb7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:36.979579 17585 tablet_service.cc:1467] Processing CreateTablet for tablet a12dd8d5f6164fb8a77d6a197aa0eb7e (DEFAULT_TABLE table=rebalance_test_table_14 [id=a07db144a5294a4dbfc503167983593e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:36.979959 17585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a12dd8d5f6164fb8a77d6a197aa0eb7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:36.984367 17454 tablet_service.cc:1467] Processing CreateTablet for tablet cf68f7ef20ba4064baee80ffddca53fe (DEFAULT_TABLE table=rebalance_test_table_14 [id=a07db144a5294a4dbfc503167983593e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:36.984707 17454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf68f7ef20ba4064baee80ffddca53fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:36.985623 17716 tablet_service.cc:1467] Processing CreateTablet for tablet cf68f7ef20ba4064baee80ffddca53fe (DEFAULT_TABLE table=rebalance_test_table_14 [id=a07db144a5294a4dbfc503167983593e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:36.986078 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf68f7ef20ba4064baee80ffddca53fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:36.989398 17846 tablet_service.cc:1467] Processing CreateTablet for tablet cf68f7ef20ba4064baee80ffddca53fe (DEFAULT_TABLE table=rebalance_test_table_14 [id=a07db144a5294a4dbfc503167983593e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:36.989753 17846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf68f7ef20ba4064baee80ffddca53fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:36.990280 17847 tablet_service.cc:1467] Processing CreateTablet for tablet 26ac703604664aff911ebe8fd7d7f101 (DEFAULT_TABLE table=rebalance_test_table_14 [id=a07db144a5294a4dbfc503167983593e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:36.990381 17455 tablet_service.cc:1467] Processing CreateTablet for tablet 26ac703604664aff911ebe8fd7d7f101 (DEFAULT_TABLE table=rebalance_test_table_14 [id=a07db144a5294a4dbfc503167983593e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:36.990679 17847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26ac703604664aff911ebe8fd7d7f101. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:36.990689 17455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26ac703604664aff911ebe8fd7d7f101. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:36.993602 17586 tablet_service.cc:1467] Processing CreateTablet for tablet 26ac703604664aff911ebe8fd7d7f101 (DEFAULT_TABLE table=rebalance_test_table_14 [id=a07db144a5294a4dbfc503167983593e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:36.993990 17586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26ac703604664aff911ebe8fd7d7f101. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:37.004700 19657 tablet_bootstrap.cc:492] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:37.007279 19657 tablet_bootstrap.cc:654] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:37.011343 19659 tablet_bootstrap.cc:492] T cf68f7ef20ba4064baee80ffddca53fe P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:37.012005 19658 tablet_bootstrap.cc:492] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:37.013787 19658 tablet_bootstrap.cc:654] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:37.014082 19656 tablet_bootstrap.cc:492] T a12dd8d5f6164fb8a77d6a197aa0eb7e P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:37.014855 19659 tablet_bootstrap.cc:654] T cf68f7ef20ba4064baee80ffddca53fe P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:37.015584 19656 tablet_bootstrap.cc:654] T a12dd8d5f6164fb8a77d6a197aa0eb7e P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:37.017556 19657 tablet_bootstrap.cc:492] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:37.017771 19657 ts_tablet_manager.cc:1367] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20240502 01:53:37.018594 19657 raft_consensus.cc:348] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:37.018961 19657 raft_consensus.cc:374] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:37.019098 19657 raft_consensus.cc:729] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:37.019403 19657 consensus_queue.cc:260] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:37.020404 19656 tablet_bootstrap.cc:492] T a12dd8d5f6164fb8a77d6a197aa0eb7e P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:37.020543 19656 ts_tablet_manager.cc:1367] T a12dd8d5f6164fb8a77d6a197aa0eb7e P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240502 01:53:37.021129 19657 ts_tablet_manager.cc:1398] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:37.021221 19659 tablet_bootstrap.cc:492] T cf68f7ef20ba4064baee80ffddca53fe P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:37.021229 19656 raft_consensus.cc:348] T a12dd8d5f6164fb8a77d6a197aa0eb7e P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:37.021349 19659 ts_tablet_manager.cc:1367] T cf68f7ef20ba4064baee80ffddca53fe P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20240502 01:53:37.021422 19656 raft_consensus.cc:374] T a12dd8d5f6164fb8a77d6a197aa0eb7e P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:37.021498 19657 tablet_bootstrap.cc:492] T 26ac703604664aff911ebe8fd7d7f101 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:37.021510 19656 raft_consensus.cc:729] T a12dd8d5f6164fb8a77d6a197aa0eb7e P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:37.021709 19656 consensus_queue.cc:260] T a12dd8d5f6164fb8a77d6a197aa0eb7e P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:37.022210 19656 ts_tablet_manager.cc:1398] T a12dd8d5f6164fb8a77d6a197aa0eb7e P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:37.022956 19657 tablet_bootstrap.cc:654] T 26ac703604664aff911ebe8fd7d7f101 P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:37.023367 19656 tablet_bootstrap.cc:492] T 26ac703604664aff911ebe8fd7d7f101 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:37.024829 19656 tablet_bootstrap.cc:654] T 26ac703604664aff911ebe8fd7d7f101 P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:37.026312 19658 tablet_bootstrap.cc:492] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:37.026685 19658 ts_tablet_manager.cc:1367] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.004s
I20240502 01:53:37.028466 19659 raft_consensus.cc:348] T cf68f7ef20ba4064baee80ffddca53fe P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:37.028735 19659 raft_consensus.cc:374] T cf68f7ef20ba4064baee80ffddca53fe P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:37.028864 19659 raft_consensus.cc:729] T cf68f7ef20ba4064baee80ffddca53fe P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:37.028671 19658 raft_consensus.cc:348] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:37.028995 19658 raft_consensus.cc:374] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:37.029176 19658 raft_consensus.cc:729] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:37.029174 19659 consensus_queue.cc:260] T cf68f7ef20ba4064baee80ffddca53fe P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:37.029460 19658 consensus_queue.cc:260] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:37.030153 19658 ts_tablet_manager.cc:1398] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:53:37.030467 19658 tablet_bootstrap.cc:492] T 26ac703604664aff911ebe8fd7d7f101 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:37.031817 19659 ts_tablet_manager.cc:1398] T cf68f7ef20ba4064baee80ffddca53fe P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240502 01:53:37.032213 19658 tablet_bootstrap.cc:654] T 26ac703604664aff911ebe8fd7d7f101 P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:37.033773 19656 tablet_bootstrap.cc:492] T 26ac703604664aff911ebe8fd7d7f101 P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:37.033845 19657 tablet_bootstrap.cc:492] T 26ac703604664aff911ebe8fd7d7f101 P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:37.033929 19656 ts_tablet_manager.cc:1367] T 26ac703604664aff911ebe8fd7d7f101 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240502 01:53:37.034018 19657 ts_tablet_manager.cc:1367] T 26ac703604664aff911ebe8fd7d7f101 P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20240502 01:53:37.034508 19656 raft_consensus.cc:348] T 26ac703604664aff911ebe8fd7d7f101 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:37.034690 19656 raft_consensus.cc:374] T 26ac703604664aff911ebe8fd7d7f101 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:37.034754 19656 raft_consensus.cc:729] T 26ac703604664aff911ebe8fd7d7f101 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:37.034741 19657 raft_consensus.cc:348] T 26ac703604664aff911ebe8fd7d7f101 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:37.034987 19657 raft_consensus.cc:374] T 26ac703604664aff911ebe8fd7d7f101 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:37.035099 19657 raft_consensus.cc:729] T 26ac703604664aff911ebe8fd7d7f101 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:37.035030 19656 consensus_queue.cc:260] T 26ac703604664aff911ebe8fd7d7f101 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:37.035593 19656 ts_tablet_manager.cc:1398] T 26ac703604664aff911ebe8fd7d7f101 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:37.035374 19657 consensus_queue.cc:260] T 26ac703604664aff911ebe8fd7d7f101 P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:37.036208 19657 ts_tablet_manager.cc:1398] T 26ac703604664aff911ebe8fd7d7f101 P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:37.036518 19657 tablet_bootstrap.cc:492] T cf68f7ef20ba4064baee80ffddca53fe P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:37.036669 19656 tablet_bootstrap.cc:492] T cf68f7ef20ba4064baee80ffddca53fe P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:37.037693 19657 tablet_bootstrap.cc:654] T cf68f7ef20ba4064baee80ffddca53fe P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:37.038089 19656 tablet_bootstrap.cc:654] T cf68f7ef20ba4064baee80ffddca53fe P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:37.045717 19658 tablet_bootstrap.cc:492] T 26ac703604664aff911ebe8fd7d7f101 P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:37.045985 19658 ts_tablet_manager.cc:1367] T 26ac703604664aff911ebe8fd7d7f101 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.003s
I20240502 01:53:37.046723 19658 raft_consensus.cc:348] T 26ac703604664aff911ebe8fd7d7f101 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:37.046883 19658 raft_consensus.cc:374] T 26ac703604664aff911ebe8fd7d7f101 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:37.046963 19658 raft_consensus.cc:729] T 26ac703604664aff911ebe8fd7d7f101 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:37.047150 19656 tablet_bootstrap.cc:492] T cf68f7ef20ba4064baee80ffddca53fe P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:37.047149 19658 consensus_queue.cc:260] T 26ac703604664aff911ebe8fd7d7f101 P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:37.047297 19656 ts_tablet_manager.cc:1367] T cf68f7ef20ba4064baee80ffddca53fe P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20240502 01:53:37.047638 19658 ts_tablet_manager.cc:1398] T 26ac703604664aff911ebe8fd7d7f101 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:53:37.047911 19656 raft_consensus.cc:348] T cf68f7ef20ba4064baee80ffddca53fe P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:37.048056 19656 raft_consensus.cc:374] T cf68f7ef20ba4064baee80ffddca53fe P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:37.048141 19656 raft_consensus.cc:729] T cf68f7ef20ba4064baee80ffddca53fe P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:37.048316 19651 raft_consensus.cc:482] T 26ac703604664aff911ebe8fd7d7f101 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:37.048331 19656 consensus_queue.cc:260] T cf68f7ef20ba4064baee80ffddca53fe P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:37.048442 19651 raft_consensus.cc:504] T 26ac703604664aff911ebe8fd7d7f101 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:37.048804 19656 ts_tablet_manager.cc:1398] T cf68f7ef20ba4064baee80ffddca53fe P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:53:37.049232 19651 leader_election.cc:288] T 26ac703604664aff911ebe8fd7d7f101 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:37.049886 17474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ac703604664aff911ebe8fd7d7f101" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:37.050025 17865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ac703604664aff911ebe8fd7d7f101" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:53:37.050098 17474 raft_consensus.cc:2454] T 26ac703604664aff911ebe8fd7d7f101 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:53:37.050209 17865 raft_consensus.cc:2454] T 26ac703604664aff911ebe8fd7d7f101 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:53:37.050540 19657 tablet_bootstrap.cc:492] T cf68f7ef20ba4064baee80ffddca53fe P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:37.050676 19657 ts_tablet_manager.cc:1367] T cf68f7ef20ba4064baee80ffddca53fe P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.007s
I20240502 01:53:37.050765 17542 leader_election.cc:302] T 26ac703604664aff911ebe8fd7d7f101 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, 4da690c0fa13407a85c0689ce8de9ade; no voters:
I20240502 01:53:37.051379 19657 raft_consensus.cc:348] T cf68f7ef20ba4064baee80ffddca53fe P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:37.051579 19657 raft_consensus.cc:374] T cf68f7ef20ba4064baee80ffddca53fe P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:37.051671 19657 raft_consensus.cc:729] T cf68f7ef20ba4064baee80ffddca53fe P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:37.052237 19651 raft_consensus.cc:2781] T 26ac703604664aff911ebe8fd7d7f101 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:37.052349 19651 raft_consensus.cc:482] T 26ac703604664aff911ebe8fd7d7f101 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:37.052428 19651 raft_consensus.cc:3037] T 26ac703604664aff911ebe8fd7d7f101 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:37.052775 19657 consensus_queue.cc:260] T cf68f7ef20ba4064baee80ffddca53fe P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:37.053319 19657 ts_tablet_manager.cc:1398] T cf68f7ef20ba4064baee80ffddca53fe P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:37.054936 19651 raft_consensus.cc:504] T 26ac703604664aff911ebe8fd7d7f101 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:37.055917 19651 leader_election.cc:288] T 26ac703604664aff911ebe8fd7d7f101 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Requested vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:37.056007 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ac703604664aff911ebe8fd7d7f101" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:53:37.056008 17865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ac703604664aff911ebe8fd7d7f101" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:53:37.056167 17475 raft_consensus.cc:3037] T 26ac703604664aff911ebe8fd7d7f101 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:37.056185 17865 raft_consensus.cc:3037] T 26ac703604664aff911ebe8fd7d7f101 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:37.058427 17475 raft_consensus.cc:2454] T 26ac703604664aff911ebe8fd7d7f101 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:53:37.058460 17865 raft_consensus.cc:2454] T 26ac703604664aff911ebe8fd7d7f101 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:53:37.058804 17542 leader_election.cc:302] T 26ac703604664aff911ebe8fd7d7f101 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, 4da690c0fa13407a85c0689ce8de9ade; no voters:
I20240502 01:53:37.059283 19651 raft_consensus.cc:2781] T 26ac703604664aff911ebe8fd7d7f101 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:37.059420 19651 raft_consensus.cc:686] T 26ac703604664aff911ebe8fd7d7f101 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Becoming Leader. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Running, Role: LEADER
I20240502 01:53:37.059649 19651 consensus_queue.cc:237] T 26ac703604664aff911ebe8fd7d7f101 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:37.062654 17334 catalog_manager.cc:5484] T 26ac703604664aff911ebe8fd7d7f101 P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66). New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } }
I20240502 01:53:37.156447 19651 raft_consensus.cc:482] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:37.156631 19651 raft_consensus.cc:504] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:37.157415 19651 leader_election.cc:288] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:53:37.157822 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12dd8d5f6164fb8a77d6a197aa0eb7e" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:37.157862 17865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12dd8d5f6164fb8a77d6a197aa0eb7e" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:53:37.158039 17475 raft_consensus.cc:2454] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:53:37.158062 17865 raft_consensus.cc:2454] T a12dd8d5f6164fb8a77d6a197aa0eb7e P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:53:37.158403 17543 leader_election.cc:302] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, 4da690c0fa13407a85c0689ce8de9ade, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:53:37.158677 19651 raft_consensus.cc:2781] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:37.158771 19651 raft_consensus.cc:482] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:37.158885 19651 raft_consensus.cc:3037] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:37.160652 19651 raft_consensus.cc:504] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:37.161176 19651 leader_election.cc:288] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Requested vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:53:37.161470 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12dd8d5f6164fb8a77d6a197aa0eb7e" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:53:37.161630 17475 raft_consensus.cc:3037] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:37.161609 17865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a12dd8d5f6164fb8a77d6a197aa0eb7e" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:53:37.161787 17865 raft_consensus.cc:3037] T a12dd8d5f6164fb8a77d6a197aa0eb7e P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:37.163439 17475 raft_consensus.cc:2454] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:53:37.163620 17865 raft_consensus.cc:2454] T a12dd8d5f6164fb8a77d6a197aa0eb7e P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:53:37.163797 17542 leader_election.cc:302] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, 4da690c0fa13407a85c0689ce8de9ade; no voters:
I20240502 01:53:37.164086 19651 raft_consensus.cc:2781] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:37.164228 19651 raft_consensus.cc:686] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Becoming Leader. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Running, Role: LEADER
I20240502 01:53:37.164507 19651 consensus_queue.cc:237] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:37.167758 17334 catalog_manager.cc:5484] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66). New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:37.326061 19660 raft_consensus.cc:482] T cf68f7ef20ba4064baee80ffddca53fe P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:37.326246 19660 raft_consensus.cc:504] T cf68f7ef20ba4064baee80ffddca53fe P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:37.326922 19660 leader_election.cc:288] T cf68f7ef20ba4064baee80ffddca53fe P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:53:37.327215 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf68f7ef20ba4064baee80ffddca53fe" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:37.327318 17865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf68f7ef20ba4064baee80ffddca53fe" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:53:37.327432 17475 raft_consensus.cc:2454] T cf68f7ef20ba4064baee80ffddca53fe P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:53:37.327533 17865 raft_consensus.cc:2454] T cf68f7ef20ba4064baee80ffddca53fe P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:53:37.327845 17672 leader_election.cc:302] T cf68f7ef20ba4064baee80ffddca53fe P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:37.328119 19660 raft_consensus.cc:2781] T cf68f7ef20ba4064baee80ffddca53fe P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:37.328250 19660 raft_consensus.cc:482] T cf68f7ef20ba4064baee80ffddca53fe P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:37.328348 19660 raft_consensus.cc:3037] T cf68f7ef20ba4064baee80ffddca53fe P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:37.330441 19660 raft_consensus.cc:504] T cf68f7ef20ba4064baee80ffddca53fe P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:37.330904 19660 leader_election.cc:288] T cf68f7ef20ba4064baee80ffddca53fe P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Requested vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:53:37.331300 17865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf68f7ef20ba4064baee80ffddca53fe" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:53:37.331332 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf68f7ef20ba4064baee80ffddca53fe" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:53:37.331446 17865 raft_consensus.cc:3037] T cf68f7ef20ba4064baee80ffddca53fe P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:37.331492 17475 raft_consensus.cc:3037] T cf68f7ef20ba4064baee80ffddca53fe P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:37.333318 17865 raft_consensus.cc:2454] T cf68f7ef20ba4064baee80ffddca53fe P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:53:37.333439 17475 raft_consensus.cc:2454] T cf68f7ef20ba4064baee80ffddca53fe P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:53:37.333667 17673 leader_election.cc:302] T cf68f7ef20ba4064baee80ffddca53fe P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb31bceba76d466db24b9341c04900c3, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:37.333909 19660 raft_consensus.cc:2781] T cf68f7ef20ba4064baee80ffddca53fe P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:37.334043 19660 raft_consensus.cc:686] T cf68f7ef20ba4064baee80ffddca53fe P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Becoming Leader. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Running, Role: LEADER
I20240502 01:53:37.334326 19660 consensus_queue.cc:237] T cf68f7ef20ba4064baee80ffddca53fe P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:37.337395 17334 catalog_manager.cc:5484] T cf68f7ef20ba4064baee80ffddca53fe P dfa1152987644bd781b46c7e42a7a685 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfa1152987644bd781b46c7e42a7a685 (127.11.42.67). New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } }
I20240502 01:53:37.500847 19643 consensus_queue.cc:1035] T 26ac703604664aff911ebe8fd7d7f101 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:37.508574 19651 consensus_queue.cc:1035] T 26ac703604664aff911ebe8fd7d7f101 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:37.563757 19643 consensus_queue.cc:1035] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:37.569506 19643 consensus_queue.cc:1035] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:37.738313 19674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:37.738754 19674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:37.753201 19674 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:53:37.760617 19660 consensus_queue.cc:1035] T cf68f7ef20ba4064baee80ffddca53fe P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:37.772521 19650 consensus_queue.cc:1035] T cf68f7ef20ba4064baee80ffddca53fe P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_14'
INSERT report
rows total: 2000
time total: 242.283 ms
time per row: 0.121141 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 01:53:39.062090 17333 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalance_test_table_15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:39.071492 17456 tablet_service.cc:1467] Processing CreateTablet for tablet 5b092dd1cc004c26a33e17c28af4d3ba (DEFAULT_TABLE table=rebalance_test_table_15 [id=21fe1df2eeed4efd801d10edf592ecce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:39.071965 17456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b092dd1cc004c26a33e17c28af4d3ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:39.073459 17585 tablet_service.cc:1467] Processing CreateTablet for tablet 5b092dd1cc004c26a33e17c28af4d3ba (DEFAULT_TABLE table=rebalance_test_table_15 [id=21fe1df2eeed4efd801d10edf592ecce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:39.073910 17585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b092dd1cc004c26a33e17c28af4d3ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:39.073815 17586 tablet_service.cc:1467] Processing CreateTablet for tablet 6aacf7c4f19546f581c9a54ad171916b (DEFAULT_TABLE table=rebalance_test_table_15 [id=21fe1df2eeed4efd801d10edf592ecce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:39.073963 17715 tablet_service.cc:1467] Processing CreateTablet for tablet 0618761d30f34cfe9650d13c9275cdcb (DEFAULT_TABLE table=rebalance_test_table_15 [id=21fe1df2eeed4efd801d10edf592ecce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:39.074211 17586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6aacf7c4f19546f581c9a54ad171916b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:39.074352 17715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0618761d30f34cfe9650d13c9275cdcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:39.077528 17716 tablet_service.cc:1467] Processing CreateTablet for tablet 5b092dd1cc004c26a33e17c28af4d3ba (DEFAULT_TABLE table=rebalance_test_table_15 [id=21fe1df2eeed4efd801d10edf592ecce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:39.077524 17845 tablet_service.cc:1467] Processing CreateTablet for tablet 0618761d30f34cfe9650d13c9275cdcb (DEFAULT_TABLE table=rebalance_test_table_15 [id=21fe1df2eeed4efd801d10edf592ecce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:39.077898 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b092dd1cc004c26a33e17c28af4d3ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:39.077913 17845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0618761d30f34cfe9650d13c9275cdcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:39.078589 17717 tablet_service.cc:1467] Processing CreateTablet for tablet 6aacf7c4f19546f581c9a54ad171916b (DEFAULT_TABLE table=rebalance_test_table_15 [id=21fe1df2eeed4efd801d10edf592ecce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:39.078907 17717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6aacf7c4f19546f581c9a54ad171916b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:39.081364 17846 tablet_service.cc:1467] Processing CreateTablet for tablet 6aacf7c4f19546f581c9a54ad171916b (DEFAULT_TABLE table=rebalance_test_table_15 [id=21fe1df2eeed4efd801d10edf592ecce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:39.081724 17846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6aacf7c4f19546f581c9a54ad171916b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:39.084524 17454 tablet_service.cc:1467] Processing CreateTablet for tablet 0618761d30f34cfe9650d13c9275cdcb (DEFAULT_TABLE table=rebalance_test_table_15 [id=21fe1df2eeed4efd801d10edf592ecce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:39.084956 17454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0618761d30f34cfe9650d13c9275cdcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:39.102353 19747 tablet_bootstrap.cc:492] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:39.102921 19746 tablet_bootstrap.cc:492] T 6aacf7c4f19546f581c9a54ad171916b P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:39.105185 19746 tablet_bootstrap.cc:654] T 6aacf7c4f19546f581c9a54ad171916b P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:39.106614 19745 tablet_bootstrap.cc:492] T 5b092dd1cc004c26a33e17c28af4d3ba P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:39.106634 19744 tablet_bootstrap.cc:492] T 5b092dd1cc004c26a33e17c28af4d3ba P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:39.108202 19744 tablet_bootstrap.cc:654] T 5b092dd1cc004c26a33e17c28af4d3ba P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:39.108204 19745 tablet_bootstrap.cc:654] T 5b092dd1cc004c26a33e17c28af4d3ba P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:39.108218 19747 tablet_bootstrap.cc:654] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:39.109874 19747 tablet_bootstrap.cc:492] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:39.110045 19747 ts_tablet_manager.cc:1367] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20240502 01:53:39.110421 19746 tablet_bootstrap.cc:492] T 6aacf7c4f19546f581c9a54ad171916b P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:39.110421 19745 tablet_bootstrap.cc:492] T 5b092dd1cc004c26a33e17c28af4d3ba P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:39.110566 19746 ts_tablet_manager.cc:1367] T 6aacf7c4f19546f581c9a54ad171916b P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.003s
I20240502 01:53:39.110648 19745 ts_tablet_manager.cc:1367] T 5b092dd1cc004c26a33e17c28af4d3ba P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:53:39.110659 19747 raft_consensus.cc:348] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:39.110854 19747 raft_consensus.cc:374] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:39.110946 19747 raft_consensus.cc:729] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:39.111164 19747 consensus_queue.cc:260] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:39.111322 19746 raft_consensus.cc:348] T 6aacf7c4f19546f581c9a54ad171916b P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:39.111399 19745 raft_consensus.cc:348] T 5b092dd1cc004c26a33e17c28af4d3ba P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:39.111588 19746 raft_consensus.cc:374] T 6aacf7c4f19546f581c9a54ad171916b P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:39.111677 19745 raft_consensus.cc:374] T 5b092dd1cc004c26a33e17c28af4d3ba P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:39.111722 19746 raft_consensus.cc:729] T 6aacf7c4f19546f581c9a54ad171916b P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:39.111757 19747 ts_tablet_manager.cc:1398] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:53:39.111827 19745 raft_consensus.cc:729] T 5b092dd1cc004c26a33e17c28af4d3ba P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:39.111974 19746 consensus_queue.cc:260] T 6aacf7c4f19546f581c9a54ad171916b P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:39.112144 19747 tablet_bootstrap.cc:492] T 6aacf7c4f19546f581c9a54ad171916b P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:39.112111 19745 consensus_queue.cc:260] T 5b092dd1cc004c26a33e17c28af4d3ba P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:39.112789 19745 ts_tablet_manager.cc:1398] T 5b092dd1cc004c26a33e17c28af4d3ba P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:39.113823 19746 ts_tablet_manager.cc:1398] T 6aacf7c4f19546f581c9a54ad171916b P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:53:39.114140 19746 tablet_bootstrap.cc:492] T 0618761d30f34cfe9650d13c9275cdcb P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:39.113144 19745 tablet_bootstrap.cc:492] T 6aacf7c4f19546f581c9a54ad171916b P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:39.117154 19745 tablet_bootstrap.cc:654] T 6aacf7c4f19546f581c9a54ad171916b P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:39.117201 19747 tablet_bootstrap.cc:654] T 6aacf7c4f19546f581c9a54ad171916b P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:39.119349 19746 tablet_bootstrap.cc:654] T 0618761d30f34cfe9650d13c9275cdcb P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:39.121145 19744 tablet_bootstrap.cc:492] T 5b092dd1cc004c26a33e17c28af4d3ba P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:39.121289 19744 ts_tablet_manager.cc:1367] T 5b092dd1cc004c26a33e17c28af4d3ba P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.003s
I20240502 01:53:39.122009 19744 raft_consensus.cc:348] T 5b092dd1cc004c26a33e17c28af4d3ba P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:39.122200 19744 raft_consensus.cc:374] T 5b092dd1cc004c26a33e17c28af4d3ba P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:39.122280 19744 raft_consensus.cc:729] T 5b092dd1cc004c26a33e17c28af4d3ba P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:39.122484 19744 consensus_queue.cc:260] T 5b092dd1cc004c26a33e17c28af4d3ba P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:39.123008 19744 ts_tablet_manager.cc:1398] T 5b092dd1cc004c26a33e17c28af4d3ba P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:39.123416 19744 tablet_bootstrap.cc:492] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:39.124965 19744 tablet_bootstrap.cc:654] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:39.126750 19746 tablet_bootstrap.cc:492] T 0618761d30f34cfe9650d13c9275cdcb P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:39.126952 19746 ts_tablet_manager.cc:1367] T 0618761d30f34cfe9650d13c9275cdcb P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.002s
I20240502 01:53:39.127775 19736 raft_consensus.cc:482] T 5b092dd1cc004c26a33e17c28af4d3ba P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:39.127928 19736 raft_consensus.cc:504] T 5b092dd1cc004c26a33e17c28af4d3ba P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:39.128800 19736 leader_election.cc:288] T 5b092dd1cc004c26a33e17c28af4d3ba P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:39.129360 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b092dd1cc004c26a33e17c28af4d3ba" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
W20240502 01:53:39.130563 17411 leader_election.cc:341] T 5b092dd1cc004c26a33e17c28af4d3ba P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:53:39.130620 19747 tablet_bootstrap.cc:492] T 6aacf7c4f19546f581c9a54ad171916b P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:39.130750 19747 ts_tablet_manager.cc:1367] T 6aacf7c4f19546f581c9a54ad171916b P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.002s
I20240502 01:53:39.131379 17607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b092dd1cc004c26a33e17c28af4d3ba" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:53:39.131592 17607 raft_consensus.cc:2454] T 5b092dd1cc004c26a33e17c28af4d3ba P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 0.
I20240502 01:53:39.132014 17409 leader_election.cc:302] T 5b092dd1cc004c26a33e17c28af4d3ba P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, 4da690c0fa13407a85c0689ce8de9ade; no voters: dfa1152987644bd781b46c7e42a7a685
I20240502 01:53:39.132417 19736 raft_consensus.cc:2781] T 5b092dd1cc004c26a33e17c28af4d3ba P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:39.132532 19736 raft_consensus.cc:482] T 5b092dd1cc004c26a33e17c28af4d3ba P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:39.132632 19736 raft_consensus.cc:3037] T 5b092dd1cc004c26a33e17c28af4d3ba P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:39.131469 19747 raft_consensus.cc:348] T 6aacf7c4f19546f581c9a54ad171916b P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:39.132959 19747 raft_consensus.cc:374] T 6aacf7c4f19546f581c9a54ad171916b P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:39.133132 19747 raft_consensus.cc:729] T 6aacf7c4f19546f581c9a54ad171916b P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:39.133397 19747 consensus_queue.cc:260] T 6aacf7c4f19546f581c9a54ad171916b P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:39.134058 19747 ts_tablet_manager.cc:1398] T 6aacf7c4f19546f581c9a54ad171916b P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:53:39.135421 19736 raft_consensus.cc:504] T 5b092dd1cc004c26a33e17c28af4d3ba P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:39.137538 19736 leader_election.cc:288] T 5b092dd1cc004c26a33e17c28af4d3ba P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Requested vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:39.136986 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b092dd1cc004c26a33e17c28af4d3ba" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:53:39.138406 17607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b092dd1cc004c26a33e17c28af4d3ba" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:53:39.138581 17607 raft_consensus.cc:3037] T 5b092dd1cc004c26a33e17c28af4d3ba P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:53:39.138749 17411 leader_election.cc:341] T 5b092dd1cc004c26a33e17c28af4d3ba P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:53:39.131758 19746 raft_consensus.cc:348] T 0618761d30f34cfe9650d13c9275cdcb P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:39.139508 19746 raft_consensus.cc:374] T 0618761d30f34cfe9650d13c9275cdcb P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:39.139575 19746 raft_consensus.cc:729] T 0618761d30f34cfe9650d13c9275cdcb P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:39.139819 19746 consensus_queue.cc:260] T 0618761d30f34cfe9650d13c9275cdcb P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:39.140846 19746 ts_tablet_manager.cc:1398] T 0618761d30f34cfe9650d13c9275cdcb P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240502 01:53:39.140915 19744 tablet_bootstrap.cc:492] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:39.141088 19744 ts_tablet_manager.cc:1367] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.005s
I20240502 01:53:39.141158 19746 tablet_bootstrap.cc:492] T 5b092dd1cc004c26a33e17c28af4d3ba P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:39.141837 19744 raft_consensus.cc:348] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:39.142040 19744 raft_consensus.cc:374] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:39.142102 19744 raft_consensus.cc:729] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:39.142333 19744 consensus_queue.cc:260] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:39.143966 19744 ts_tablet_manager.cc:1398] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:53:39.144500 17607 raft_consensus.cc:2454] T 5b092dd1cc004c26a33e17c28af4d3ba P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
I20240502 01:53:39.146236 17409 leader_election.cc:302] T 5b092dd1cc004c26a33e17c28af4d3ba P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, 4da690c0fa13407a85c0689ce8de9ade; no voters: dfa1152987644bd781b46c7e42a7a685
I20240502 01:53:39.146891 19756 raft_consensus.cc:2781] T 5b092dd1cc004c26a33e17c28af4d3ba P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:39.147063 19756 raft_consensus.cc:686] T 5b092dd1cc004c26a33e17c28af4d3ba P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Becoming Leader. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Running, Role: LEADER
I20240502 01:53:39.149699 19746 tablet_bootstrap.cc:654] T 5b092dd1cc004c26a33e17c28af4d3ba P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:39.147384 19756 consensus_queue.cc:237] T 5b092dd1cc004c26a33e17c28af4d3ba P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:39.151747 19745 tablet_bootstrap.cc:492] T 6aacf7c4f19546f581c9a54ad171916b P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:39.151922 19745 ts_tablet_manager.cc:1367] T 6aacf7c4f19546f581c9a54ad171916b P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.039s user 0.009s sys 0.011s
I20240502 01:53:39.152671 19745 raft_consensus.cc:348] T 6aacf7c4f19546f581c9a54ad171916b P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:39.152895 19745 raft_consensus.cc:374] T 6aacf7c4f19546f581c9a54ad171916b P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:39.152972 19745 raft_consensus.cc:729] T 6aacf7c4f19546f581c9a54ad171916b P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:39.153226 19745 consensus_queue.cc:260] T 6aacf7c4f19546f581c9a54ad171916b P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:39.154739 19745 ts_tablet_manager.cc:1398] T 6aacf7c4f19546f581c9a54ad171916b P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:39.154983 17327 catalog_manager.cc:5484] T 5b092dd1cc004c26a33e17c28af4d3ba P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: term changed from 0 to 1, leader changed from <none> to 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65). New cstate: current_term: 1 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:39.159750 19746 tablet_bootstrap.cc:492] T 5b092dd1cc004c26a33e17c28af4d3ba P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:39.159883 19746 ts_tablet_manager.cc:1367] T 5b092dd1cc004c26a33e17c28af4d3ba P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.002s
I20240502 01:53:39.160434 19746 raft_consensus.cc:348] T 5b092dd1cc004c26a33e17c28af4d3ba P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:39.160619 19746 raft_consensus.cc:374] T 5b092dd1cc004c26a33e17c28af4d3ba P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:39.160709 19746 raft_consensus.cc:729] T 5b092dd1cc004c26a33e17c28af4d3ba P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:39.160898 19746 consensus_queue.cc:260] T 5b092dd1cc004c26a33e17c28af4d3ba P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:39.161317 19746 ts_tablet_manager.cc:1398] T 5b092dd1cc004c26a33e17c28af4d3ba P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 01:53:39.162748 19756 raft_consensus.cc:482] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:39.162909 19756 raft_consensus.cc:504] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:39.163606 19756 leader_election.cc:288] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:53:39.163823 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0618761d30f34cfe9650d13c9275cdcb" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:53:39.163884 17866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0618761d30f34cfe9650d13c9275cdcb" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:53:39.164043 17735 raft_consensus.cc:2454] T 0618761d30f34cfe9650d13c9275cdcb P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 0.
I20240502 01:53:39.164098 17866 raft_consensus.cc:2454] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 0.
I20240502 01:53:39.164430 17411 leader_election.cc:302] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:39.164698 19756 raft_consensus.cc:2781] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:39.164784 19756 raft_consensus.cc:482] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:39.164883 19756 raft_consensus.cc:3037] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:39.167299 19756 raft_consensus.cc:504] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:39.167773 19756 leader_election.cc:288] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Requested vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:53:39.168123 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0618761d30f34cfe9650d13c9275cdcb" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:53:39.168212 17866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0618761d30f34cfe9650d13c9275cdcb" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:53:39.168291 17735 raft_consensus.cc:3037] T 0618761d30f34cfe9650d13c9275cdcb P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:39.168426 17866 raft_consensus.cc:3037] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:39.170110 17735 raft_consensus.cc:2454] T 0618761d30f34cfe9650d13c9275cdcb P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
I20240502 01:53:39.171067 17866 raft_consensus.cc:2454] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
I20240502 01:53:39.171447 17412 leader_election.cc:302] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:53:39.171676 19756 raft_consensus.cc:2781] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:39.171844 19756 raft_consensus.cc:686] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Becoming Leader. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Running, Role: LEADER
I20240502 01:53:39.172089 19756 consensus_queue.cc:237] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:39.174978 17326 catalog_manager.cc:5484] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: term changed from 0 to 1, leader changed from <none> to 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65). New cstate: current_term: 1 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } }
I20240502 01:53:39.355193 19739 raft_consensus.cc:482] T 6aacf7c4f19546f581c9a54ad171916b P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:39.355373 19739 raft_consensus.cc:504] T 6aacf7c4f19546f581c9a54ad171916b P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:39.356040 19739 leader_election.cc:288] T 6aacf7c4f19546f581c9a54ad171916b P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:53:39.356390 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aacf7c4f19546f581c9a54ad171916b" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:53:39.356415 17866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aacf7c4f19546f581c9a54ad171916b" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:53:39.356628 17866 raft_consensus.cc:2454] T 6aacf7c4f19546f581c9a54ad171916b P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:53:39.356679 17735 raft_consensus.cc:2454] T 6aacf7c4f19546f581c9a54ad171916b P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:53:39.357055 17543 leader_election.cc:302] T 6aacf7c4f19546f581c9a54ad171916b P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:53:39.357345 19739 raft_consensus.cc:2781] T 6aacf7c4f19546f581c9a54ad171916b P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:39.357453 19739 raft_consensus.cc:482] T 6aacf7c4f19546f581c9a54ad171916b P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:39.357537 19739 raft_consensus.cc:3037] T 6aacf7c4f19546f581c9a54ad171916b P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:39.359393 19739 raft_consensus.cc:504] T 6aacf7c4f19546f581c9a54ad171916b P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:39.359951 19739 leader_election.cc:288] T 6aacf7c4f19546f581c9a54ad171916b P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Requested vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:53:39.360281 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aacf7c4f19546f581c9a54ad171916b" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:53:39.360448 17735 raft_consensus.cc:3037] T 6aacf7c4f19546f581c9a54ad171916b P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:39.360383 17866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aacf7c4f19546f581c9a54ad171916b" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:53:39.360553 17866 raft_consensus.cc:3037] T 6aacf7c4f19546f581c9a54ad171916b P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:39.362488 17735 raft_consensus.cc:2454] T 6aacf7c4f19546f581c9a54ad171916b P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:53:39.362648 17866 raft_consensus.cc:2454] T 6aacf7c4f19546f581c9a54ad171916b P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:53:39.362898 17542 leader_election.cc:302] T 6aacf7c4f19546f581c9a54ad171916b P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:39.363592 19739 raft_consensus.cc:2781] T 6aacf7c4f19546f581c9a54ad171916b P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:39.363967 19739 raft_consensus.cc:686] T 6aacf7c4f19546f581c9a54ad171916b P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Becoming Leader. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Running, Role: LEADER
I20240502 01:53:39.364235 19739 consensus_queue.cc:237] T 6aacf7c4f19546f581c9a54ad171916b P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:39.367919 17326 catalog_manager.cc:5484] T 6aacf7c4f19546f581c9a54ad171916b P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66). New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:39.418881 19716 raft_consensus.cc:482] T 5b092dd1cc004c26a33e17c28af4d3ba P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:39.419056 19716 raft_consensus.cc:504] T 5b092dd1cc004c26a33e17c28af4d3ba P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:39.419692 19716 leader_election.cc:288] T 5b092dd1cc004c26a33e17c28af4d3ba P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:39.420099 17476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b092dd1cc004c26a33e17c28af4d3ba" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:39.420176 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b092dd1cc004c26a33e17c28af4d3ba" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:53:39.420446 17604 raft_consensus.cc:2379] T 5b092dd1cc004c26a33e17c28af4d3ba P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dfa1152987644bd781b46c7e42a7a685 in current term 1: Already voted for candidate 4da690c0fa13407a85c0689ce8de9ade in this term.
I20240502 01:53:39.420861 17670 leader_election.cc:302] T 5b092dd1cc004c26a33e17c28af4d3ba P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dfa1152987644bd781b46c7e42a7a685; no voters: 3fceccefba274b1dbbd8216af8b0bc29, 4da690c0fa13407a85c0689ce8de9ade
I20240502 01:53:39.421155 19716 raft_consensus.cc:3037] T 5b092dd1cc004c26a33e17c28af4d3ba P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:39.423892 19716 raft_consensus.cc:2726] T 5b092dd1cc004c26a33e17c28af4d3ba P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:53:39.546648 19756 consensus_queue.cc:1035] T 5b092dd1cc004c26a33e17c28af4d3ba P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:39.554996 19756 consensus_queue.cc:1035] T 5b092dd1cc004c26a33e17c28af4d3ba P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:39.636557 19756 consensus_queue.cc:1035] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:39.641902 19736 consensus_queue.cc:1035] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:39.730760 19766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:39.731156 19766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:39.739089 19766 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_15'
I20240502 01:53:39.807938 19739 consensus_queue.cc:1035] T 6aacf7c4f19546f581c9a54ad171916b P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:39.826500 19739 consensus_queue.cc:1035] T 6aacf7c4f19546f581c9a54ad171916b P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 238.464 ms
time per row: 0.119232 ms
I20240502 01:53:40.129525 17330 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalance_test_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:40.139117 17585 tablet_service.cc:1467] Processing CreateTablet for tablet fa21373c6f4a43df9fd2a5ec2b8d6242 (DEFAULT_TABLE table=rebalance_test_table_16 [id=f27ebc659702427c996fda2492c7e5d0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:40.139593 17585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa21373c6f4a43df9fd2a5ec2b8d6242. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:40.139772 17586 tablet_service.cc:1467] Processing CreateTablet for tablet 27815f771d5b4bbaa7a70d5ea30b279b (DEFAULT_TABLE table=rebalance_test_table_16 [id=f27ebc659702427c996fda2492c7e5d0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:40.140154 17586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27815f771d5b4bbaa7a70d5ea30b279b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:40.145967 17456 tablet_service.cc:1467] Processing CreateTablet for tablet fa21373c6f4a43df9fd2a5ec2b8d6242 (DEFAULT_TABLE table=rebalance_test_table_16 [id=f27ebc659702427c996fda2492c7e5d0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:40.146128 17717 tablet_service.cc:1467] Processing CreateTablet for tablet fa21373c6f4a43df9fd2a5ec2b8d6242 (DEFAULT_TABLE table=rebalance_test_table_16 [id=f27ebc659702427c996fda2492c7e5d0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:40.146432 17456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa21373c6f4a43df9fd2a5ec2b8d6242. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:40.146554 17717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa21373c6f4a43df9fd2a5ec2b8d6242. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:40.150005 17845 tablet_service.cc:1467] Processing CreateTablet for tablet 27815f771d5b4bbaa7a70d5ea30b279b (DEFAULT_TABLE table=rebalance_test_table_16 [id=f27ebc659702427c996fda2492c7e5d0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:40.150435 17845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27815f771d5b4bbaa7a70d5ea30b279b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:40.152982 17716 tablet_service.cc:1467] Processing CreateTablet for tablet 377e804dbeed4913a7d435cfdd1e2d8e (DEFAULT_TABLE table=rebalance_test_table_16 [id=f27ebc659702427c996fda2492c7e5d0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:40.153594 17454 tablet_service.cc:1467] Processing CreateTablet for tablet 27815f771d5b4bbaa7a70d5ea30b279b (DEFAULT_TABLE table=rebalance_test_table_16 [id=f27ebc659702427c996fda2492c7e5d0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:40.153766 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 377e804dbeed4913a7d435cfdd1e2d8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:40.153771 17455 tablet_service.cc:1467] Processing CreateTablet for tablet 377e804dbeed4913a7d435cfdd1e2d8e (DEFAULT_TABLE table=rebalance_test_table_16 [id=f27ebc659702427c996fda2492c7e5d0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:40.153970 17454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27815f771d5b4bbaa7a70d5ea30b279b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:40.154167 17455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 377e804dbeed4913a7d435cfdd1e2d8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:40.158217 17846 tablet_service.cc:1467] Processing CreateTablet for tablet 377e804dbeed4913a7d435cfdd1e2d8e (DEFAULT_TABLE table=rebalance_test_table_16 [id=f27ebc659702427c996fda2492c7e5d0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:40.158588 17846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 377e804dbeed4913a7d435cfdd1e2d8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:40.164011 19824 tablet_bootstrap.cc:492] T fa21373c6f4a43df9fd2a5ec2b8d6242 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:40.166280 19825 tablet_bootstrap.cc:492] T 377e804dbeed4913a7d435cfdd1e2d8e P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:40.166376 19824 tablet_bootstrap.cc:654] T fa21373c6f4a43df9fd2a5ec2b8d6242 P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:40.168215 19825 tablet_bootstrap.cc:654] T 377e804dbeed4913a7d435cfdd1e2d8e P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:40.169730 19823 tablet_bootstrap.cc:492] T fa21373c6f4a43df9fd2a5ec2b8d6242 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:40.172447 19823 tablet_bootstrap.cc:654] T fa21373c6f4a43df9fd2a5ec2b8d6242 P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:40.179082 19825 tablet_bootstrap.cc:492] T 377e804dbeed4913a7d435cfdd1e2d8e P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:40.179251 19825 ts_tablet_manager.cc:1367] T 377e804dbeed4913a7d435cfdd1e2d8e P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20240502 01:53:40.179260 19824 tablet_bootstrap.cc:492] T fa21373c6f4a43df9fd2a5ec2b8d6242 P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:40.179394 19824 ts_tablet_manager.cc:1367] T fa21373c6f4a43df9fd2a5ec2b8d6242 P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.011s
I20240502 01:53:40.179733 19823 tablet_bootstrap.cc:492] T fa21373c6f4a43df9fd2a5ec2b8d6242 P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:40.179950 19823 ts_tablet_manager.cc:1367] T fa21373c6f4a43df9fd2a5ec2b8d6242 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.002s
I20240502 01:53:40.180775 19823 raft_consensus.cc:348] T fa21373c6f4a43df9fd2a5ec2b8d6242 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:40.180910 19825 raft_consensus.cc:348] T 377e804dbeed4913a7d435cfdd1e2d8e P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:40.181051 19823 raft_consensus.cc:374] T fa21373c6f4a43df9fd2a5ec2b8d6242 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:40.181155 19823 raft_consensus.cc:729] T fa21373c6f4a43df9fd2a5ec2b8d6242 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:40.181226 19825 raft_consensus.cc:374] T 377e804dbeed4913a7d435cfdd1e2d8e P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:40.181378 19825 raft_consensus.cc:729] T 377e804dbeed4913a7d435cfdd1e2d8e P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:40.181437 19824 raft_consensus.cc:348] T fa21373c6f4a43df9fd2a5ec2b8d6242 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:40.181701 19824 raft_consensus.cc:374] T fa21373c6f4a43df9fd2a5ec2b8d6242 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:40.181618 19823 consensus_queue.cc:260] T fa21373c6f4a43df9fd2a5ec2b8d6242 P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:40.181641 19825 consensus_queue.cc:260] T 377e804dbeed4913a7d435cfdd1e2d8e P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:40.182015 19824 raft_consensus.cc:729] T fa21373c6f4a43df9fd2a5ec2b8d6242 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:40.182149 19825 ts_tablet_manager.cc:1398] T 377e804dbeed4913a7d435cfdd1e2d8e P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:40.182346 19824 consensus_queue.cc:260] T fa21373c6f4a43df9fd2a5ec2b8d6242 P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:40.182974 19824 ts_tablet_manager.cc:1398] T fa21373c6f4a43df9fd2a5ec2b8d6242 P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 01:53:40.182988 19827 tablet_bootstrap.cc:492] T 27815f771d5b4bbaa7a70d5ea30b279b P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:40.183300 19824 tablet_bootstrap.cc:492] T 377e804dbeed4913a7d435cfdd1e2d8e P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:40.184150 19825 tablet_bootstrap.cc:492] T fa21373c6f4a43df9fd2a5ec2b8d6242 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:40.184871 19824 tablet_bootstrap.cc:654] T 377e804dbeed4913a7d435cfdd1e2d8e P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:40.188689 19827 tablet_bootstrap.cc:654] T 27815f771d5b4bbaa7a70d5ea30b279b P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:40.189757 19823 ts_tablet_manager.cc:1398] T fa21373c6f4a43df9fd2a5ec2b8d6242 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.010s user 0.000s sys 0.001s
I20240502 01:53:40.190078 19823 tablet_bootstrap.cc:492] T 27815f771d5b4bbaa7a70d5ea30b279b P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:40.192032 19825 tablet_bootstrap.cc:654] T fa21373c6f4a43df9fd2a5ec2b8d6242 P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:40.192962 19824 tablet_bootstrap.cc:492] T 377e804dbeed4913a7d435cfdd1e2d8e P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:40.193145 19824 ts_tablet_manager.cc:1367] T 377e804dbeed4913a7d435cfdd1e2d8e P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.001s
I20240502 01:53:40.193351 19823 tablet_bootstrap.cc:654] T 27815f771d5b4bbaa7a70d5ea30b279b P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:40.193935 19824 raft_consensus.cc:348] T 377e804dbeed4913a7d435cfdd1e2d8e P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:40.194144 19824 raft_consensus.cc:374] T 377e804dbeed4913a7d435cfdd1e2d8e P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:40.194229 19824 raft_consensus.cc:729] T 377e804dbeed4913a7d435cfdd1e2d8e P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:40.194522 19824 consensus_queue.cc:260] T 377e804dbeed4913a7d435cfdd1e2d8e P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:40.195654 19824 ts_tablet_manager.cc:1398] T 377e804dbeed4913a7d435cfdd1e2d8e P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:53:40.196024 19824 tablet_bootstrap.cc:492] T 27815f771d5b4bbaa7a70d5ea30b279b P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:40.197587 19824 tablet_bootstrap.cc:654] T 27815f771d5b4bbaa7a70d5ea30b279b P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:40.205874 19824 tablet_bootstrap.cc:492] T 27815f771d5b4bbaa7a70d5ea30b279b P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:40.206019 19824 ts_tablet_manager.cc:1367] T 27815f771d5b4bbaa7a70d5ea30b279b P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20240502 01:53:40.206521 19827 tablet_bootstrap.cc:492] T 27815f771d5b4bbaa7a70d5ea30b279b P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:40.206655 19827 ts_tablet_manager.cc:1367] T 27815f771d5b4bbaa7a70d5ea30b279b P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.000s
I20240502 01:53:40.206760 19824 raft_consensus.cc:348] T 27815f771d5b4bbaa7a70d5ea30b279b P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:40.206961 19824 raft_consensus.cc:374] T 27815f771d5b4bbaa7a70d5ea30b279b P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:40.207049 19824 raft_consensus.cc:729] T 27815f771d5b4bbaa7a70d5ea30b279b P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:40.207283 19824 consensus_queue.cc:260] T 27815f771d5b4bbaa7a70d5ea30b279b P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:40.207350 19827 raft_consensus.cc:348] T 27815f771d5b4bbaa7a70d5ea30b279b P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:40.207566 19827 raft_consensus.cc:374] T 27815f771d5b4bbaa7a70d5ea30b279b P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:40.207679 19827 raft_consensus.cc:729] T 27815f771d5b4bbaa7a70d5ea30b279b P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:40.207862 19824 ts_tablet_manager.cc:1398] T 27815f771d5b4bbaa7a70d5ea30b279b P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:53:40.207906 19827 consensus_queue.cc:260] T 27815f771d5b4bbaa7a70d5ea30b279b P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:40.208461 19827 ts_tablet_manager.cc:1398] T 27815f771d5b4bbaa7a70d5ea30b279b P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:40.208755 19827 tablet_bootstrap.cc:492] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:40.210602 19827 tablet_bootstrap.cc:654] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:40.211962 19823 tablet_bootstrap.cc:492] T 27815f771d5b4bbaa7a70d5ea30b279b P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:40.212131 19823 ts_tablet_manager.cc:1367] T 27815f771d5b4bbaa7a70d5ea30b279b P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.006s
I20240502 01:53:40.212858 19823 raft_consensus.cc:348] T 27815f771d5b4bbaa7a70d5ea30b279b P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:40.213099 19823 raft_consensus.cc:374] T 27815f771d5b4bbaa7a70d5ea30b279b P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:40.213173 19823 raft_consensus.cc:729] T 27815f771d5b4bbaa7a70d5ea30b279b P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:40.213414 19823 consensus_queue.cc:260] T 27815f771d5b4bbaa7a70d5ea30b279b P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:40.213960 19823 ts_tablet_manager.cc:1398] T 27815f771d5b4bbaa7a70d5ea30b279b P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:40.215399 19825 tablet_bootstrap.cc:492] T fa21373c6f4a43df9fd2a5ec2b8d6242 P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:40.215564 19825 ts_tablet_manager.cc:1367] T fa21373c6f4a43df9fd2a5ec2b8d6242 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.007s
I20240502 01:53:40.216230 19825 raft_consensus.cc:348] T fa21373c6f4a43df9fd2a5ec2b8d6242 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:40.216411 19825 raft_consensus.cc:374] T fa21373c6f4a43df9fd2a5ec2b8d6242 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:40.216485 19825 raft_consensus.cc:729] T fa21373c6f4a43df9fd2a5ec2b8d6242 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:40.216722 19825 consensus_queue.cc:260] T fa21373c6f4a43df9fd2a5ec2b8d6242 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:40.217275 19825 ts_tablet_manager.cc:1398] T fa21373c6f4a43df9fd2a5ec2b8d6242 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:53:40.217373 19762 raft_consensus.cc:482] T 27815f771d5b4bbaa7a70d5ea30b279b P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:40.217497 19762 raft_consensus.cc:504] T 27815f771d5b4bbaa7a70d5ea30b279b P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:40.218070 19762 leader_election.cc:288] T 27815f771d5b4bbaa7a70d5ea30b279b P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:40.218554 19827 tablet_bootstrap.cc:492] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:40.218482 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27815f771d5b4bbaa7a70d5ea30b279b" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:53:40.218679 17604 raft_consensus.cc:2454] T 27815f771d5b4bbaa7a70d5ea30b279b P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 0.
I20240502 01:53:40.218705 19827 ts_tablet_manager.cc:1367] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240502 01:53:40.219095 17800 leader_election.cc:302] T 27815f771d5b4bbaa7a70d5ea30b279b P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:53:40.219381 19762 raft_consensus.cc:2781] T 27815f771d5b4bbaa7a70d5ea30b279b P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:40.219321 17476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27815f771d5b4bbaa7a70d5ea30b279b" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:40.219501 19762 raft_consensus.cc:482] T 27815f771d5b4bbaa7a70d5ea30b279b P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:40.219537 17476 raft_consensus.cc:2454] T 27815f771d5b4bbaa7a70d5ea30b279b P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 0.
I20240502 01:53:40.219473 19827 raft_consensus.cc:348] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:40.219596 19762 raft_consensus.cc:3037] T 27815f771d5b4bbaa7a70d5ea30b279b P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:40.219677 19827 raft_consensus.cc:374] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:40.219770 19827 raft_consensus.cc:729] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:40.220011 19827 consensus_queue.cc:260] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:40.220589 19827 ts_tablet_manager.cc:1398] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:53:40.222309 19762 raft_consensus.cc:504] T 27815f771d5b4bbaa7a70d5ea30b279b P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:40.223312 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27815f771d5b4bbaa7a70d5ea30b279b" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:53:40.223498 17604 raft_consensus.cc:3037] T 27815f771d5b4bbaa7a70d5ea30b279b P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:40.225976 17604 raft_consensus.cc:2454] T 27815f771d5b4bbaa7a70d5ea30b279b P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 1.
I20240502 01:53:40.226339 17800 leader_election.cc:302] T 27815f771d5b4bbaa7a70d5ea30b279b P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:53:40.226610 19738 raft_consensus.cc:2781] T 27815f771d5b4bbaa7a70d5ea30b279b P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:40.226881 19762 leader_election.cc:288] T 27815f771d5b4bbaa7a70d5ea30b279b P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 election: Requested vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:40.227099 17476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27815f771d5b4bbaa7a70d5ea30b279b" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:53:40.227331 17476 raft_consensus.cc:3037] T 27815f771d5b4bbaa7a70d5ea30b279b P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:40.229108 19738 raft_consensus.cc:686] T 27815f771d5b4bbaa7a70d5ea30b279b P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Becoming Leader. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Running, Role: LEADER
I20240502 01:53:40.229146 17476 raft_consensus.cc:2454] T 27815f771d5b4bbaa7a70d5ea30b279b P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 1.
I20240502 01:53:40.229430 19738 consensus_queue.cc:237] T 27815f771d5b4bbaa7a70d5ea30b279b P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:40.232270 17335 catalog_manager.cc:5484] T 27815f771d5b4bbaa7a70d5ea30b279b P bb31bceba76d466db24b9341c04900c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb31bceba76d466db24b9341c04900c3 (127.11.42.68). New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:40.299294 19838 raft_consensus.cc:482] T fa21373c6f4a43df9fd2a5ec2b8d6242 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:40.299489 19838 raft_consensus.cc:504] T fa21373c6f4a43df9fd2a5ec2b8d6242 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:40.300115 19838 leader_election.cc:288] T fa21373c6f4a43df9fd2a5ec2b8d6242 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:40.300485 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa21373c6f4a43df9fd2a5ec2b8d6242" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:53:40.300524 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa21373c6f4a43df9fd2a5ec2b8d6242" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:40.300720 17604 raft_consensus.cc:2454] T fa21373c6f4a43df9fd2a5ec2b8d6242 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:53:40.300760 17475 raft_consensus.cc:2454] T fa21373c6f4a43df9fd2a5ec2b8d6242 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:53:40.301108 17670 leader_election.cc:302] T fa21373c6f4a43df9fd2a5ec2b8d6242 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:40.301364 19838 raft_consensus.cc:2781] T fa21373c6f4a43df9fd2a5ec2b8d6242 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:40.301479 19838 raft_consensus.cc:482] T fa21373c6f4a43df9fd2a5ec2b8d6242 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:40.301550 19838 raft_consensus.cc:3037] T fa21373c6f4a43df9fd2a5ec2b8d6242 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:40.303805 19838 raft_consensus.cc:504] T fa21373c6f4a43df9fd2a5ec2b8d6242 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:40.304302 19838 leader_election.cc:288] T fa21373c6f4a43df9fd2a5ec2b8d6242 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Requested vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:40.304678 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa21373c6f4a43df9fd2a5ec2b8d6242" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:53:40.304803 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa21373c6f4a43df9fd2a5ec2b8d6242" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:53:40.304921 17604 raft_consensus.cc:3037] T fa21373c6f4a43df9fd2a5ec2b8d6242 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:40.305052 17475 raft_consensus.cc:3037] T fa21373c6f4a43df9fd2a5ec2b8d6242 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:40.307103 17475 raft_consensus.cc:2454] T fa21373c6f4a43df9fd2a5ec2b8d6242 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:53:40.307291 17604 raft_consensus.cc:2454] T fa21373c6f4a43df9fd2a5ec2b8d6242 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:53:40.307492 17672 leader_election.cc:302] T fa21373c6f4a43df9fd2a5ec2b8d6242 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:40.307842 19838 raft_consensus.cc:2781] T fa21373c6f4a43df9fd2a5ec2b8d6242 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:40.307991 19838 raft_consensus.cc:686] T fa21373c6f4a43df9fd2a5ec2b8d6242 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Becoming Leader. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Running, Role: LEADER
I20240502 01:53:40.308491 19838 consensus_queue.cc:237] T fa21373c6f4a43df9fd2a5ec2b8d6242 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:40.311688 17335 catalog_manager.cc:5484] T fa21373c6f4a43df9fd2a5ec2b8d6242 P dfa1152987644bd781b46c7e42a7a685 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfa1152987644bd781b46c7e42a7a685 (127.11.42.67). New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } }
I20240502 01:53:40.377674 19738 raft_consensus.cc:482] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:40.377841 19738 raft_consensus.cc:504] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:40.378535 19738 leader_election.cc:288] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:40.378904 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377e804dbeed4913a7d435cfdd1e2d8e" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:53:40.378988 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377e804dbeed4913a7d435cfdd1e2d8e" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:40.379122 17736 raft_consensus.cc:2454] T 377e804dbeed4913a7d435cfdd1e2d8e P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 0.
I20240502 01:53:40.379181 17475 raft_consensus.cc:2454] T 377e804dbeed4913a7d435cfdd1e2d8e P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 0.
I20240502 01:53:40.379529 17802 leader_election.cc:302] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb31bceba76d466db24b9341c04900c3, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:40.379787 19738 raft_consensus.cc:2781] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:40.379904 19738 raft_consensus.cc:482] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:40.380023 19738 raft_consensus.cc:3037] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:40.382438 19738 raft_consensus.cc:504] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:40.383064 19738 leader_election.cc:288] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 election: Requested vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:40.383697 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377e804dbeed4913a7d435cfdd1e2d8e" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:53:40.383751 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377e804dbeed4913a7d435cfdd1e2d8e" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:53:40.383873 17475 raft_consensus.cc:3037] T 377e804dbeed4913a7d435cfdd1e2d8e P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:40.383970 17736 raft_consensus.cc:3037] T 377e804dbeed4913a7d435cfdd1e2d8e P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:40.385792 17736 raft_consensus.cc:2454] T 377e804dbeed4913a7d435cfdd1e2d8e P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 1.
I20240502 01:53:40.385810 17475 raft_consensus.cc:2454] T 377e804dbeed4913a7d435cfdd1e2d8e P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 1.
I20240502 01:53:40.386132 17802 leader_election.cc:302] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb31bceba76d466db24b9341c04900c3, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:40.386359 19738 raft_consensus.cc:2781] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:40.386538 19738 raft_consensus.cc:686] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Becoming Leader. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Running, Role: LEADER
I20240502 01:53:40.386776 19738 consensus_queue.cc:237] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:40.389827 17335 catalog_manager.cc:5484] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb31bceba76d466db24b9341c04900c3 (127.11.42.68). New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:40.688949 19738 consensus_queue.cc:1035] T 27815f771d5b4bbaa7a70d5ea30b279b P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:40.706179 19762 consensus_queue.cc:1035] T 27815f771d5b4bbaa7a70d5ea30b279b P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:40.733825 19716 consensus_queue.cc:1035] T fa21373c6f4a43df9fd2a5ec2b8d6242 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:40.738864 19841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:40.739262 19841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 01:53:40.747140 19716 consensus_queue.cc:1035] T fa21373c6f4a43df9fd2a5ec2b8d6242 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:40.748607 19841 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_16'
I20240502 01:53:40.888252 17475 raft_consensus.cc:1261] T 377e804dbeed4913a7d435cfdd1e2d8e P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:40.889195 19836 consensus_queue.cc:1035] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:40.928333 17736 raft_consensus.cc:1261] T 377e804dbeed4913a7d435cfdd1e2d8e P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:40.932812 19836 consensus_queue.cc:1035] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 269.56 ms
time per row: 0.13478 ms
I20240502 01:53:41.160107 17335 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalance_test_table_17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:41.169191 17585 tablet_service.cc:1467] Processing CreateTablet for tablet 4bb42883b1a0475b863d008bd01fa70a (DEFAULT_TABLE table=rebalance_test_table_17 [id=c6474d0c5256497eb6fa5844f3530b62]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:41.169421 17716 tablet_service.cc:1467] Processing CreateTablet for tablet 4bb42883b1a0475b863d008bd01fa70a (DEFAULT_TABLE table=rebalance_test_table_17 [id=c6474d0c5256497eb6fa5844f3530b62]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:41.169677 17585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bb42883b1a0475b863d008bd01fa70a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:41.169822 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bb42883b1a0475b863d008bd01fa70a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:41.170975 17717 tablet_service.cc:1467] Processing CreateTablet for tablet 65bdab2c9fba48a6828a864278319cc0 (DEFAULT_TABLE table=rebalance_test_table_17 [id=c6474d0c5256497eb6fa5844f3530b62]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:41.171335 17717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65bdab2c9fba48a6828a864278319cc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:41.173482 17454 tablet_service.cc:1467] Processing CreateTablet for tablet 30f43088bd0f48ee86907df16194b9be (DEFAULT_TABLE table=rebalance_test_table_17 [id=c6474d0c5256497eb6fa5844f3530b62]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:41.173895 17454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30f43088bd0f48ee86907df16194b9be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:41.177474 17715 tablet_service.cc:1467] Processing CreateTablet for tablet 30f43088bd0f48ee86907df16194b9be (DEFAULT_TABLE table=rebalance_test_table_17 [id=c6474d0c5256497eb6fa5844f3530b62]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:41.177834 17715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30f43088bd0f48ee86907df16194b9be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:41.178719 17586 tablet_service.cc:1467] Processing CreateTablet for tablet 65bdab2c9fba48a6828a864278319cc0 (DEFAULT_TABLE table=rebalance_test_table_17 [id=c6474d0c5256497eb6fa5844f3530b62]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:41.178809 17587 tablet_service.cc:1467] Processing CreateTablet for tablet 30f43088bd0f48ee86907df16194b9be (DEFAULT_TABLE table=rebalance_test_table_17 [id=c6474d0c5256497eb6fa5844f3530b62]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:41.179119 17586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65bdab2c9fba48a6828a864278319cc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:41.179342 17587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30f43088bd0f48ee86907df16194b9be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:41.181964 17845 tablet_service.cc:1467] Processing CreateTablet for tablet 4bb42883b1a0475b863d008bd01fa70a (DEFAULT_TABLE table=rebalance_test_table_17 [id=c6474d0c5256497eb6fa5844f3530b62]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:41.182379 17845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bb42883b1a0475b863d008bd01fa70a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:41.185379 17846 tablet_service.cc:1467] Processing CreateTablet for tablet 65bdab2c9fba48a6828a864278319cc0 (DEFAULT_TABLE table=rebalance_test_table_17 [id=c6474d0c5256497eb6fa5844f3530b62]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:41.185716 17846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65bdab2c9fba48a6828a864278319cc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:41.188093 19910 tablet_bootstrap.cc:492] T 30f43088bd0f48ee86907df16194b9be P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:41.190696 19910 tablet_bootstrap.cc:654] T 30f43088bd0f48ee86907df16194b9be P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:41.195029 19909 tablet_bootstrap.cc:492] T 4bb42883b1a0475b863d008bd01fa70a P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:41.198964 19909 tablet_bootstrap.cc:654] T 4bb42883b1a0475b863d008bd01fa70a P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:41.202890 19913 tablet_bootstrap.cc:492] T 65bdab2c9fba48a6828a864278319cc0 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:41.205091 19913 tablet_bootstrap.cc:654] T 65bdab2c9fba48a6828a864278319cc0 P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:41.205737 19911 tablet_bootstrap.cc:492] T 4bb42883b1a0475b863d008bd01fa70a P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:41.209107 19911 tablet_bootstrap.cc:654] T 4bb42883b1a0475b863d008bd01fa70a P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:41.210081 19909 tablet_bootstrap.cc:492] T 4bb42883b1a0475b863d008bd01fa70a P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:41.210089 19910 tablet_bootstrap.cc:492] T 30f43088bd0f48ee86907df16194b9be P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:41.210233 19910 ts_tablet_manager.cc:1367] T 30f43088bd0f48ee86907df16194b9be P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.006s
I20240502 01:53:41.210299 19909 ts_tablet_manager.cc:1367] T 4bb42883b1a0475b863d008bd01fa70a P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.008s
I20240502 01:53:41.211074 19910 raft_consensus.cc:348] T 30f43088bd0f48ee86907df16194b9be P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:41.211365 19910 raft_consensus.cc:374] T 30f43088bd0f48ee86907df16194b9be P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:41.211088 19909 raft_consensus.cc:348] T 4bb42883b1a0475b863d008bd01fa70a P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:41.211473 19910 raft_consensus.cc:729] T 30f43088bd0f48ee86907df16194b9be P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:41.211601 19909 raft_consensus.cc:374] T 4bb42883b1a0475b863d008bd01fa70a P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:41.211755 19909 raft_consensus.cc:729] T 4bb42883b1a0475b863d008bd01fa70a P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:41.211727 19910 consensus_queue.cc:260] T 30f43088bd0f48ee86907df16194b9be P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:41.212076 19909 consensus_queue.cc:260] T 4bb42883b1a0475b863d008bd01fa70a P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:41.212364 19910 ts_tablet_manager.cc:1398] T 30f43088bd0f48ee86907df16194b9be P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:53:41.212859 19909 ts_tablet_manager.cc:1398] T 4bb42883b1a0475b863d008bd01fa70a P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:53:41.213295 19909 tablet_bootstrap.cc:492] T 65bdab2c9fba48a6828a864278319cc0 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:41.214830 19909 tablet_bootstrap.cc:654] T 65bdab2c9fba48a6828a864278319cc0 P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:41.216140 19913 tablet_bootstrap.cc:492] T 65bdab2c9fba48a6828a864278319cc0 P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:41.216291 19913 ts_tablet_manager.cc:1367] T 65bdab2c9fba48a6828a864278319cc0 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.004s
I20240502 01:53:41.217139 19913 raft_consensus.cc:348] T 65bdab2c9fba48a6828a864278319cc0 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:41.217357 19913 raft_consensus.cc:374] T 65bdab2c9fba48a6828a864278319cc0 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:41.217438 19913 raft_consensus.cc:729] T 65bdab2c9fba48a6828a864278319cc0 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:41.217638 19913 consensus_queue.cc:260] T 65bdab2c9fba48a6828a864278319cc0 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:41.218552 19911 tablet_bootstrap.cc:492] T 4bb42883b1a0475b863d008bd01fa70a P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:41.218711 19911 ts_tablet_manager.cc:1367] T 4bb42883b1a0475b863d008bd01fa70a P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.004s
I20240502 01:53:41.219511 19911 raft_consensus.cc:348] T 4bb42883b1a0475b863d008bd01fa70a P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:41.219739 19911 raft_consensus.cc:374] T 4bb42883b1a0475b863d008bd01fa70a P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:41.219842 19911 raft_consensus.cc:729] T 4bb42883b1a0475b863d008bd01fa70a P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:41.220090 19911 consensus_queue.cc:260] T 4bb42883b1a0475b863d008bd01fa70a P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:41.220700 19911 ts_tablet_manager.cc:1398] T 4bb42883b1a0475b863d008bd01fa70a P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:53:41.220999 19911 tablet_bootstrap.cc:492] T 65bdab2c9fba48a6828a864278319cc0 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:41.221630 19913 ts_tablet_manager.cc:1398] T 65bdab2c9fba48a6828a864278319cc0 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:53:41.222003 19913 tablet_bootstrap.cc:492] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:41.224098 19913 tablet_bootstrap.cc:654] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:41.225070 19909 tablet_bootstrap.cc:492] T 65bdab2c9fba48a6828a864278319cc0 P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:41.225220 19909 ts_tablet_manager.cc:1367] T 65bdab2c9fba48a6828a864278319cc0 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.002s
I20240502 01:53:41.225772 19911 tablet_bootstrap.cc:654] T 65bdab2c9fba48a6828a864278319cc0 P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:41.226285 19909 raft_consensus.cc:348] T 65bdab2c9fba48a6828a864278319cc0 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:41.226719 19909 raft_consensus.cc:374] T 65bdab2c9fba48a6828a864278319cc0 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:41.226826 19909 raft_consensus.cc:729] T 65bdab2c9fba48a6828a864278319cc0 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:41.227041 19909 consensus_queue.cc:260] T 65bdab2c9fba48a6828a864278319cc0 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:41.227651 19909 ts_tablet_manager.cc:1398] T 65bdab2c9fba48a6828a864278319cc0 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:41.228077 19909 tablet_bootstrap.cc:492] T 30f43088bd0f48ee86907df16194b9be P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:41.229719 19909 tablet_bootstrap.cc:654] T 30f43088bd0f48ee86907df16194b9be P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:41.229763 19911 tablet_bootstrap.cc:492] T 65bdab2c9fba48a6828a864278319cc0 P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:41.229874 19913 tablet_bootstrap.cc:492] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:41.229924 19911 ts_tablet_manager.cc:1367] T 65bdab2c9fba48a6828a864278319cc0 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240502 01:53:41.230541 19911 raft_consensus.cc:348] T 65bdab2c9fba48a6828a864278319cc0 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:41.230751 19911 raft_consensus.cc:374] T 65bdab2c9fba48a6828a864278319cc0 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:41.230818 19911 raft_consensus.cc:729] T 65bdab2c9fba48a6828a864278319cc0 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:41.231133 19913 ts_tablet_manager.cc:1367] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.001s
I20240502 01:53:41.231047 19911 consensus_queue.cc:260] T 65bdab2c9fba48a6828a864278319cc0 P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:41.231601 19909 tablet_bootstrap.cc:492] T 30f43088bd0f48ee86907df16194b9be P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:41.231596 19911 ts_tablet_manager.cc:1398] T 65bdab2c9fba48a6828a864278319cc0 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:53:41.231736 19909 ts_tablet_manager.cc:1367] T 30f43088bd0f48ee86907df16194b9be P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:53:41.231868 19911 tablet_bootstrap.cc:492] T 30f43088bd0f48ee86907df16194b9be P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:41.232204 19913 raft_consensus.cc:348] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:41.232483 19913 raft_consensus.cc:374] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:41.232409 19909 raft_consensus.cc:348] T 30f43088bd0f48ee86907df16194b9be P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:41.232584 19909 raft_consensus.cc:374] T 30f43088bd0f48ee86907df16194b9be P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:41.232627 19913 raft_consensus.cc:729] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:41.232685 19909 raft_consensus.cc:729] T 30f43088bd0f48ee86907df16194b9be P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:41.232947 19909 consensus_queue.cc:260] T 30f43088bd0f48ee86907df16194b9be P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:41.233549 19909 ts_tablet_manager.cc:1398] T 30f43088bd0f48ee86907df16194b9be P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:41.233603 19911 tablet_bootstrap.cc:654] T 30f43088bd0f48ee86907df16194b9be P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:41.234285 19913 consensus_queue.cc:260] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:41.234970 19913 ts_tablet_manager.cc:1398] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:53:41.236614 19911 tablet_bootstrap.cc:492] T 30f43088bd0f48ee86907df16194b9be P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:41.236738 19911 ts_tablet_manager.cc:1367] T 30f43088bd0f48ee86907df16194b9be P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240502 01:53:41.237444 19911 raft_consensus.cc:348] T 30f43088bd0f48ee86907df16194b9be P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:41.237623 19911 raft_consensus.cc:374] T 30f43088bd0f48ee86907df16194b9be P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:41.237712 19911 raft_consensus.cc:729] T 30f43088bd0f48ee86907df16194b9be P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:41.237931 19911 consensus_queue.cc:260] T 30f43088bd0f48ee86907df16194b9be P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:41.238422 19911 ts_tablet_manager.cc:1398] T 30f43088bd0f48ee86907df16194b9be P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:41.259987 19739 raft_consensus.cc:482] T 30f43088bd0f48ee86907df16194b9be P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:41.260145 19739 raft_consensus.cc:504] T 30f43088bd0f48ee86907df16194b9be P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:41.260664 19739 leader_election.cc:288] T 30f43088bd0f48ee86907df16194b9be P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:41.261128 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30f43088bd0f48ee86907df16194b9be" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:53:41.261171 17476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30f43088bd0f48ee86907df16194b9be" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:41.261340 17736 raft_consensus.cc:2454] T 30f43088bd0f48ee86907df16194b9be P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:53:41.261395 17476 raft_consensus.cc:2454] T 30f43088bd0f48ee86907df16194b9be P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:53:41.261723 17542 leader_election.cc:302] T 30f43088bd0f48ee86907df16194b9be P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:41.261996 19739 raft_consensus.cc:2781] T 30f43088bd0f48ee86907df16194b9be P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:41.262086 19739 raft_consensus.cc:482] T 30f43088bd0f48ee86907df16194b9be P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:41.262187 19739 raft_consensus.cc:3037] T 30f43088bd0f48ee86907df16194b9be P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:41.264017 19739 raft_consensus.cc:504] T 30f43088bd0f48ee86907df16194b9be P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:41.264489 19739 leader_election.cc:288] T 30f43088bd0f48ee86907df16194b9be P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Requested vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:41.264869 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30f43088bd0f48ee86907df16194b9be" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:53:41.264956 17476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30f43088bd0f48ee86907df16194b9be" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:53:41.265050 17736 raft_consensus.cc:3037] T 30f43088bd0f48ee86907df16194b9be P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:41.265153 17476 raft_consensus.cc:3037] T 30f43088bd0f48ee86907df16194b9be P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:41.267254 17736 raft_consensus.cc:2454] T 30f43088bd0f48ee86907df16194b9be P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:53:41.267424 17476 raft_consensus.cc:2454] T 30f43088bd0f48ee86907df16194b9be P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:53:41.267602 17542 leader_election.cc:302] T 30f43088bd0f48ee86907df16194b9be P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:41.267859 19739 raft_consensus.cc:2781] T 30f43088bd0f48ee86907df16194b9be P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:41.268024 19739 raft_consensus.cc:686] T 30f43088bd0f48ee86907df16194b9be P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Becoming Leader. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Running, Role: LEADER
I20240502 01:53:41.268255 19739 consensus_queue.cc:237] T 30f43088bd0f48ee86907df16194b9be P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:41.271303 17333 catalog_manager.cc:5484] T 30f43088bd0f48ee86907df16194b9be P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66). New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } }
I20240502 01:53:41.272246 19738 raft_consensus.cc:482] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:41.272397 19738 raft_consensus.cc:504] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:41.273005 19738 leader_election.cc:288] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:53:41.273360 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bb42883b1a0475b863d008bd01fa70a" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:53:41.273415 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bb42883b1a0475b863d008bd01fa70a" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:53:41.273557 17736 raft_consensus.cc:2454] T 4bb42883b1a0475b863d008bd01fa70a P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 0.
I20240502 01:53:41.273542 17604 raft_consensus.cc:2454] T 4bb42883b1a0475b863d008bd01fa70a P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 0.
I20240502 01:53:41.273917 17800 leader_election.cc:302] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:53:41.274168 19738 raft_consensus.cc:2781] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:41.274266 19738 raft_consensus.cc:482] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:41.274372 19738 raft_consensus.cc:3037] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:41.276082 19738 raft_consensus.cc:504] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:41.276778 19738 leader_election.cc:288] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 election: Requested vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:53:41.276921 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bb42883b1a0475b863d008bd01fa70a" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:53:41.276944 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bb42883b1a0475b863d008bd01fa70a" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:53:41.277127 17604 raft_consensus.cc:3037] T 4bb42883b1a0475b863d008bd01fa70a P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:41.277164 17736 raft_consensus.cc:3037] T 4bb42883b1a0475b863d008bd01fa70a P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:41.278959 17736 raft_consensus.cc:2454] T 4bb42883b1a0475b863d008bd01fa70a P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 1.
I20240502 01:53:41.279047 17604 raft_consensus.cc:2454] T 4bb42883b1a0475b863d008bd01fa70a P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 1.
I20240502 01:53:41.279314 17802 leader_election.cc:302] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb31bceba76d466db24b9341c04900c3, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:41.279570 19738 raft_consensus.cc:2781] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:41.279726 19738 raft_consensus.cc:686] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Becoming Leader. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Running, Role: LEADER
I20240502 01:53:41.279938 19738 consensus_queue.cc:237] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:41.282886 17333 catalog_manager.cc:5484] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb31bceba76d466db24b9341c04900c3 (127.11.42.68). New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } }
I20240502 01:53:41.515087 19738 raft_consensus.cc:482] T 65bdab2c9fba48a6828a864278319cc0 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:41.515244 19738 raft_consensus.cc:504] T 65bdab2c9fba48a6828a864278319cc0 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:41.515901 19738 leader_election.cc:288] T 65bdab2c9fba48a6828a864278319cc0 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:41.516296 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65bdab2c9fba48a6828a864278319cc0" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:53:41.516332 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65bdab2c9fba48a6828a864278319cc0" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:53:41.516510 17736 raft_consensus.cc:2454] T 65bdab2c9fba48a6828a864278319cc0 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 0.
I20240502 01:53:41.516538 17604 raft_consensus.cc:2454] T 65bdab2c9fba48a6828a864278319cc0 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 0.
I20240502 01:53:41.516940 17800 leader_election.cc:302] T 65bdab2c9fba48a6828a864278319cc0 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:53:41.517251 19738 raft_consensus.cc:2781] T 65bdab2c9fba48a6828a864278319cc0 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:41.517360 19738 raft_consensus.cc:482] T 65bdab2c9fba48a6828a864278319cc0 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:41.517441 19738 raft_consensus.cc:3037] T 65bdab2c9fba48a6828a864278319cc0 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:41.519273 19738 raft_consensus.cc:504] T 65bdab2c9fba48a6828a864278319cc0 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:41.519796 19738 leader_election.cc:288] T 65bdab2c9fba48a6828a864278319cc0 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 election: Requested vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:41.520149 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65bdab2c9fba48a6828a864278319cc0" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:53:41.520231 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65bdab2c9fba48a6828a864278319cc0" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:53:41.520319 17736 raft_consensus.cc:3037] T 65bdab2c9fba48a6828a864278319cc0 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:41.520428 17604 raft_consensus.cc:3037] T 65bdab2c9fba48a6828a864278319cc0 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:41.522146 17736 raft_consensus.cc:2454] T 65bdab2c9fba48a6828a864278319cc0 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 1.
I20240502 01:53:41.522294 17604 raft_consensus.cc:2454] T 65bdab2c9fba48a6828a864278319cc0 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 1.
I20240502 01:53:41.522524 17802 leader_election.cc:302] T 65bdab2c9fba48a6828a864278319cc0 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb31bceba76d466db24b9341c04900c3, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:41.522773 19738 raft_consensus.cc:2781] T 65bdab2c9fba48a6828a864278319cc0 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:41.522926 19738 raft_consensus.cc:686] T 65bdab2c9fba48a6828a864278319cc0 P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Becoming Leader. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Running, Role: LEADER
I20240502 01:53:41.523160 19738 consensus_queue.cc:237] T 65bdab2c9fba48a6828a864278319cc0 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:41.526657 17333 catalog_manager.cc:5484] T 65bdab2c9fba48a6828a864278319cc0 P bb31bceba76d466db24b9341c04900c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb31bceba76d466db24b9341c04900c3 (127.11.42.68). New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } }
I20240502 01:53:41.736534 19739 consensus_queue.cc:1035] T 30f43088bd0f48ee86907df16194b9be P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:41.746357 19842 consensus_queue.cc:1035] T 30f43088bd0f48ee86907df16194b9be P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:41.795143 19836 consensus_queue.cc:1035] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:41.808085 19836 consensus_queue.cc:1035] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:41.907399 19925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:41.907678 19925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:41.915581 19925 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_17'
I20240502 01:53:42.006469 19738 consensus_queue.cc:1035] T 65bdab2c9fba48a6828a864278319cc0 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:42.016427 19936 consensus_queue.cc:1035] T 65bdab2c9fba48a6828a864278319cc0 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 205.88 ms
time per row: 0.10294 ms
I20240502 01:53:42.275619 17335 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalance_test_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:42.284987 17846 tablet_service.cc:1467] Processing CreateTablet for tablet 5c56837c0a2e49f69fab0aaf01d113bd (DEFAULT_TABLE table=rebalance_test_table_18 [id=fa75b6d8fec44d1998a1c84ded9c9998]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:42.285378 17585 tablet_service.cc:1467] Processing CreateTablet for tablet 5c56837c0a2e49f69fab0aaf01d113bd (DEFAULT_TABLE table=rebalance_test_table_18 [id=fa75b6d8fec44d1998a1c84ded9c9998]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:42.285526 17846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c56837c0a2e49f69fab0aaf01d113bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:42.285822 17585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c56837c0a2e49f69fab0aaf01d113bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:42.290553 19990 tablet_bootstrap.cc:492] T 5c56837c0a2e49f69fab0aaf01d113bd P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:42.290484 17454 tablet_service.cc:1467] Processing CreateTablet for tablet 5c56837c0a2e49f69fab0aaf01d113bd (DEFAULT_TABLE table=rebalance_test_table_18 [id=fa75b6d8fec44d1998a1c84ded9c9998]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:42.290926 17454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c56837c0a2e49f69fab0aaf01d113bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:42.292268 19990 tablet_bootstrap.cc:654] T 5c56837c0a2e49f69fab0aaf01d113bd P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:42.293278 17845 tablet_service.cc:1467] Processing CreateTablet for tablet 9fe583045886494ca8cf656ad2f83189 (DEFAULT_TABLE table=rebalance_test_table_18 [id=fa75b6d8fec44d1998a1c84ded9c9998]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:42.293669 17845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fe583045886494ca8cf656ad2f83189. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:42.294296 19990 tablet_bootstrap.cc:492] T 5c56837c0a2e49f69fab0aaf01d113bd P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:42.294435 19990 ts_tablet_manager.cc:1367] T 5c56837c0a2e49f69fab0aaf01d113bd P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 01:53:42.295150 19990 raft_consensus.cc:348] T 5c56837c0a2e49f69fab0aaf01d113bd P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:42.295361 19990 raft_consensus.cc:374] T 5c56837c0a2e49f69fab0aaf01d113bd P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:42.295444 19990 raft_consensus.cc:729] T 5c56837c0a2e49f69fab0aaf01d113bd P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:42.295639 19990 consensus_queue.cc:260] T 5c56837c0a2e49f69fab0aaf01d113bd P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:42.297900 17456 tablet_service.cc:1467] Processing CreateTablet for tablet 9fe583045886494ca8cf656ad2f83189 (DEFAULT_TABLE table=rebalance_test_table_18 [id=fa75b6d8fec44d1998a1c84ded9c9998]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:42.298302 17456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fe583045886494ca8cf656ad2f83189. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:42.298357 17715 tablet_service.cc:1467] Processing CreateTablet for tablet 9fe583045886494ca8cf656ad2f83189 (DEFAULT_TABLE table=rebalance_test_table_18 [id=fa75b6d8fec44d1998a1c84ded9c9998]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:42.298754 17715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fe583045886494ca8cf656ad2f83189. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:42.298909 19990 ts_tablet_manager.cc:1398] T 5c56837c0a2e49f69fab0aaf01d113bd P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240502 01:53:42.300699 19989 tablet_bootstrap.cc:492] T 5c56837c0a2e49f69fab0aaf01d113bd P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:42.302386 19989 tablet_bootstrap.cc:654] T 5c56837c0a2e49f69fab0aaf01d113bd P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:42.305330 17456 tablet_service.cc:1467] Processing CreateTablet for tablet 1d4efb3b1e3e4fa3bde6305ac993596e (DEFAULT_TABLE table=rebalance_test_table_18 [id=fa75b6d8fec44d1998a1c84ded9c9998]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:42.305399 17585 tablet_service.cc:1467] Processing CreateTablet for tablet 1d4efb3b1e3e4fa3bde6305ac993596e (DEFAULT_TABLE table=rebalance_test_table_18 [id=fa75b6d8fec44d1998a1c84ded9c9998]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:42.305672 17456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d4efb3b1e3e4fa3bde6305ac993596e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:42.305735 17585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d4efb3b1e3e4fa3bde6305ac993596e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:42.309572 17846 tablet_service.cc:1467] Processing CreateTablet for tablet 1d4efb3b1e3e4fa3bde6305ac993596e (DEFAULT_TABLE table=rebalance_test_table_18 [id=fa75b6d8fec44d1998a1c84ded9c9998]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:42.309895 17846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d4efb3b1e3e4fa3bde6305ac993596e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:42.316138 19990 tablet_bootstrap.cc:492] T 1d4efb3b1e3e4fa3bde6305ac993596e P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:42.316267 19992 tablet_bootstrap.cc:492] T 5c56837c0a2e49f69fab0aaf01d113bd P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:42.317665 19990 tablet_bootstrap.cc:654] T 1d4efb3b1e3e4fa3bde6305ac993596e P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:42.317801 19992 tablet_bootstrap.cc:654] T 5c56837c0a2e49f69fab0aaf01d113bd P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:42.319175 19994 tablet_bootstrap.cc:492] T 9fe583045886494ca8cf656ad2f83189 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:42.319725 19989 tablet_bootstrap.cc:492] T 5c56837c0a2e49f69fab0aaf01d113bd P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:42.319866 19989 ts_tablet_manager.cc:1367] T 5c56837c0a2e49f69fab0aaf01d113bd P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20240502 01:53:42.320487 19994 tablet_bootstrap.cc:654] T 9fe583045886494ca8cf656ad2f83189 P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:42.320526 19989 raft_consensus.cc:348] T 5c56837c0a2e49f69fab0aaf01d113bd P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:42.320724 19989 raft_consensus.cc:374] T 5c56837c0a2e49f69fab0aaf01d113bd P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:42.320870 19989 raft_consensus.cc:729] T 5c56837c0a2e49f69fab0aaf01d113bd P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:42.321218 19989 consensus_queue.cc:260] T 5c56837c0a2e49f69fab0aaf01d113bd P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:42.322124 19992 tablet_bootstrap.cc:492] T 5c56837c0a2e49f69fab0aaf01d113bd P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:42.322292 19992 ts_tablet_manager.cc:1367] T 5c56837c0a2e49f69fab0aaf01d113bd P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240502 01:53:42.323006 19989 ts_tablet_manager.cc:1398] T 5c56837c0a2e49f69fab0aaf01d113bd P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:42.323107 19992 raft_consensus.cc:348] T 5c56837c0a2e49f69fab0aaf01d113bd P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:42.323346 19989 tablet_bootstrap.cc:492] T 9fe583045886494ca8cf656ad2f83189 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:42.323375 19992 raft_consensus.cc:374] T 5c56837c0a2e49f69fab0aaf01d113bd P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:42.323523 19992 raft_consensus.cc:729] T 5c56837c0a2e49f69fab0aaf01d113bd P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:42.323796 19992 consensus_queue.cc:260] T 5c56837c0a2e49f69fab0aaf01d113bd P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:42.324405 19992 ts_tablet_manager.cc:1398] T 5c56837c0a2e49f69fab0aaf01d113bd P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:53:42.324688 19992 tablet_bootstrap.cc:492] T 9fe583045886494ca8cf656ad2f83189 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:42.326090 19992 tablet_bootstrap.cc:654] T 9fe583045886494ca8cf656ad2f83189 P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:42.326572 19989 tablet_bootstrap.cc:654] T 9fe583045886494ca8cf656ad2f83189 P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:42.328178 19990 tablet_bootstrap.cc:492] T 1d4efb3b1e3e4fa3bde6305ac993596e P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:42.328187 19994 tablet_bootstrap.cc:492] T 9fe583045886494ca8cf656ad2f83189 P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:42.328315 19994 ts_tablet_manager.cc:1367] T 9fe583045886494ca8cf656ad2f83189 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.003s
I20240502 01:53:42.328341 19990 ts_tablet_manager.cc:1367] T 1d4efb3b1e3e4fa3bde6305ac993596e P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.003s
I20240502 01:53:42.329037 19990 raft_consensus.cc:348] T 1d4efb3b1e3e4fa3bde6305ac993596e P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:42.329070 19994 raft_consensus.cc:348] T 9fe583045886494ca8cf656ad2f83189 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:42.329262 19994 raft_consensus.cc:374] T 9fe583045886494ca8cf656ad2f83189 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:42.329257 19990 raft_consensus.cc:374] T 1d4efb3b1e3e4fa3bde6305ac993596e P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:42.329324 19990 raft_consensus.cc:729] T 1d4efb3b1e3e4fa3bde6305ac993596e P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:42.329339 19994 raft_consensus.cc:729] T 9fe583045886494ca8cf656ad2f83189 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:42.329548 19994 consensus_queue.cc:260] T 9fe583045886494ca8cf656ad2f83189 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:42.329548 19990 consensus_queue.cc:260] T 1d4efb3b1e3e4fa3bde6305ac993596e P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:42.330075 19990 ts_tablet_manager.cc:1398] T 1d4efb3b1e3e4fa3bde6305ac993596e P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:53:42.330524 19994 ts_tablet_manager.cc:1398] T 9fe583045886494ca8cf656ad2f83189 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:53:42.336935 19989 tablet_bootstrap.cc:492] T 9fe583045886494ca8cf656ad2f83189 P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:42.337122 19989 ts_tablet_manager.cc:1367] T 9fe583045886494ca8cf656ad2f83189 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.000s
I20240502 01:53:42.338088 19989 raft_consensus.cc:348] T 9fe583045886494ca8cf656ad2f83189 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:42.338266 19989 raft_consensus.cc:374] T 9fe583045886494ca8cf656ad2f83189 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:42.338346 19989 raft_consensus.cc:729] T 9fe583045886494ca8cf656ad2f83189 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:42.338554 19989 consensus_queue.cc:260] T 9fe583045886494ca8cf656ad2f83189 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:42.338809 19992 tablet_bootstrap.cc:492] T 9fe583045886494ca8cf656ad2f83189 P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:42.339021 19992 ts_tablet_manager.cc:1367] T 9fe583045886494ca8cf656ad2f83189 P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.003s
I20240502 01:53:42.339046 19989 ts_tablet_manager.cc:1398] T 9fe583045886494ca8cf656ad2f83189 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:42.339324 19989 tablet_bootstrap.cc:492] T 1d4efb3b1e3e4fa3bde6305ac993596e P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:42.340407 19992 raft_consensus.cc:348] T 9fe583045886494ca8cf656ad2f83189 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:42.340687 19992 raft_consensus.cc:374] T 9fe583045886494ca8cf656ad2f83189 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:42.340809 19989 tablet_bootstrap.cc:654] T 1d4efb3b1e3e4fa3bde6305ac993596e P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:42.340806 19992 raft_consensus.cc:729] T 9fe583045886494ca8cf656ad2f83189 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:42.341048 19992 consensus_queue.cc:260] T 9fe583045886494ca8cf656ad2f83189 P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:42.341622 19992 ts_tablet_manager.cc:1398] T 9fe583045886494ca8cf656ad2f83189 P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:53:42.341945 19992 tablet_bootstrap.cc:492] T 1d4efb3b1e3e4fa3bde6305ac993596e P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:42.343575 19992 tablet_bootstrap.cc:654] T 1d4efb3b1e3e4fa3bde6305ac993596e P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:42.346702 19989 tablet_bootstrap.cc:492] T 1d4efb3b1e3e4fa3bde6305ac993596e P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:42.346940 19989 ts_tablet_manager.cc:1367] T 1d4efb3b1e3e4fa3bde6305ac993596e P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240502 01:53:42.347690 19989 raft_consensus.cc:348] T 1d4efb3b1e3e4fa3bde6305ac993596e P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:42.348086 19989 raft_consensus.cc:374] T 1d4efb3b1e3e4fa3bde6305ac993596e P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:42.348188 19989 raft_consensus.cc:729] T 1d4efb3b1e3e4fa3bde6305ac993596e P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:42.348381 19989 consensus_queue.cc:260] T 1d4efb3b1e3e4fa3bde6305ac993596e P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:42.348778 19989 ts_tablet_manager.cc:1398] T 1d4efb3b1e3e4fa3bde6305ac993596e P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:42.350170 19992 tablet_bootstrap.cc:492] T 1d4efb3b1e3e4fa3bde6305ac993596e P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:42.350490 19992 ts_tablet_manager.cc:1367] T 1d4efb3b1e3e4fa3bde6305ac993596e P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20240502 01:53:42.351197 19992 raft_consensus.cc:348] T 1d4efb3b1e3e4fa3bde6305ac993596e P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:42.351342 19992 raft_consensus.cc:374] T 1d4efb3b1e3e4fa3bde6305ac993596e P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:42.351430 19992 raft_consensus.cc:729] T 1d4efb3b1e3e4fa3bde6305ac993596e P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:42.351610 19992 consensus_queue.cc:260] T 1d4efb3b1e3e4fa3bde6305ac993596e P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:42.352100 19992 ts_tablet_manager.cc:1398] T 1d4efb3b1e3e4fa3bde6305ac993596e P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 01:53:42.355887 19983 raft_consensus.cc:482] T 9fe583045886494ca8cf656ad2f83189 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:42.356005 19983 raft_consensus.cc:504] T 9fe583045886494ca8cf656ad2f83189 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:42.356637 19983 leader_election.cc:288] T 9fe583045886494ca8cf656ad2f83189 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:42.357051 17865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe583045886494ca8cf656ad2f83189" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:53:42.357267 17865 raft_consensus.cc:2454] T 9fe583045886494ca8cf656ad2f83189 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:53:42.357221 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe583045886494ca8cf656ad2f83189" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:42.357420 17475 raft_consensus.cc:2454] T 9fe583045886494ca8cf656ad2f83189 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:53:42.357774 17673 leader_election.cc:302] T 9fe583045886494ca8cf656ad2f83189 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb31bceba76d466db24b9341c04900c3, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:42.358093 19983 raft_consensus.cc:2781] T 9fe583045886494ca8cf656ad2f83189 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:42.358215 19983 raft_consensus.cc:482] T 9fe583045886494ca8cf656ad2f83189 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:42.358307 19983 raft_consensus.cc:3037] T 9fe583045886494ca8cf656ad2f83189 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:42.360422 19983 raft_consensus.cc:504] T 9fe583045886494ca8cf656ad2f83189 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:42.360960 19983 leader_election.cc:288] T 9fe583045886494ca8cf656ad2f83189 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Requested vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:42.361346 17865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe583045886494ca8cf656ad2f83189" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:53:42.361393 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe583045886494ca8cf656ad2f83189" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:53:42.361572 17865 raft_consensus.cc:3037] T 9fe583045886494ca8cf656ad2f83189 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:42.361575 17475 raft_consensus.cc:3037] T 9fe583045886494ca8cf656ad2f83189 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:42.364318 17475 raft_consensus.cc:2454] T 9fe583045886494ca8cf656ad2f83189 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:53:42.364336 17865 raft_consensus.cc:2454] T 9fe583045886494ca8cf656ad2f83189 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:53:42.364713 17672 leader_election.cc:302] T 9fe583045886494ca8cf656ad2f83189 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:42.364997 19983 raft_consensus.cc:2781] T 9fe583045886494ca8cf656ad2f83189 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:42.365214 19983 raft_consensus.cc:686] T 9fe583045886494ca8cf656ad2f83189 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Becoming Leader. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Running, Role: LEADER
I20240502 01:53:42.365545 19983 consensus_queue.cc:237] T 9fe583045886494ca8cf656ad2f83189 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:42.368569 17333 catalog_manager.cc:5484] T 9fe583045886494ca8cf656ad2f83189 P dfa1152987644bd781b46c7e42a7a685 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfa1152987644bd781b46c7e42a7a685 (127.11.42.67). New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } }
I20240502 01:53:42.435442 19738 raft_consensus.cc:482] T 1d4efb3b1e3e4fa3bde6305ac993596e P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:42.435617 19738 raft_consensus.cc:504] T 1d4efb3b1e3e4fa3bde6305ac993596e P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:42.436208 19738 leader_election.cc:288] T 1d4efb3b1e3e4fa3bde6305ac993596e P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:42.436542 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d4efb3b1e3e4fa3bde6305ac993596e" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:42.436717 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d4efb3b1e3e4fa3bde6305ac993596e" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:53:42.436793 17475 raft_consensus.cc:2454] T 1d4efb3b1e3e4fa3bde6305ac993596e P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 0.
I20240502 01:53:42.436929 17604 raft_consensus.cc:2454] T 1d4efb3b1e3e4fa3bde6305ac993596e P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 0.
I20240502 01:53:42.437224 17802 leader_election.cc:302] T 1d4efb3b1e3e4fa3bde6305ac993596e P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:53:42.437487 19738 raft_consensus.cc:2781] T 1d4efb3b1e3e4fa3bde6305ac993596e P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:42.437585 19738 raft_consensus.cc:482] T 1d4efb3b1e3e4fa3bde6305ac993596e P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:42.437654 19738 raft_consensus.cc:3037] T 1d4efb3b1e3e4fa3bde6305ac993596e P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:42.439472 19738 raft_consensus.cc:504] T 1d4efb3b1e3e4fa3bde6305ac993596e P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:42.439962 19738 leader_election.cc:288] T 1d4efb3b1e3e4fa3bde6305ac993596e P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 election: Requested vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:42.440356 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d4efb3b1e3e4fa3bde6305ac993596e" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:53:42.440457 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d4efb3b1e3e4fa3bde6305ac993596e" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:53:42.440587 17475 raft_consensus.cc:3037] T 1d4efb3b1e3e4fa3bde6305ac993596e P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:42.440636 17604 raft_consensus.cc:3037] T 1d4efb3b1e3e4fa3bde6305ac993596e P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:42.443408 17604 raft_consensus.cc:2454] T 1d4efb3b1e3e4fa3bde6305ac993596e P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 1.
I20240502 01:53:42.443400 17475 raft_consensus.cc:2454] T 1d4efb3b1e3e4fa3bde6305ac993596e P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 1.
I20240502 01:53:42.443778 17800 leader_election.cc:302] T 1d4efb3b1e3e4fa3bde6305ac993596e P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:53:42.444054 19738 raft_consensus.cc:2781] T 1d4efb3b1e3e4fa3bde6305ac993596e P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:42.444202 19738 raft_consensus.cc:686] T 1d4efb3b1e3e4fa3bde6305ac993596e P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Becoming Leader. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Running, Role: LEADER
I20240502 01:53:42.444473 19738 consensus_queue.cc:237] T 1d4efb3b1e3e4fa3bde6305ac993596e P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:42.447748 17333 catalog_manager.cc:5484] T 1d4efb3b1e3e4fa3bde6305ac993596e P bb31bceba76d466db24b9341c04900c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb31bceba76d466db24b9341c04900c3 (127.11.42.68). New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } }
I20240502 01:53:42.466476 19842 raft_consensus.cc:482] T 5c56837c0a2e49f69fab0aaf01d113bd P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:42.466660 19842 raft_consensus.cc:504] T 5c56837c0a2e49f69fab0aaf01d113bd P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:42.467276 19842 leader_election.cc:288] T 5c56837c0a2e49f69fab0aaf01d113bd P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:42.467667 17865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c56837c0a2e49f69fab0aaf01d113bd" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:53:42.467680 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c56837c0a2e49f69fab0aaf01d113bd" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:42.467885 17475 raft_consensus.cc:2454] T 5c56837c0a2e49f69fab0aaf01d113bd P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:53:42.467880 17865 raft_consensus.cc:2454] T 5c56837c0a2e49f69fab0aaf01d113bd P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:53:42.468237 17542 leader_election.cc:302] T 5c56837c0a2e49f69fab0aaf01d113bd P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, 4da690c0fa13407a85c0689ce8de9ade; no voters:
I20240502 01:53:42.468525 19842 raft_consensus.cc:2781] T 5c56837c0a2e49f69fab0aaf01d113bd P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:42.468605 19842 raft_consensus.cc:482] T 5c56837c0a2e49f69fab0aaf01d113bd P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:42.468698 19842 raft_consensus.cc:3037] T 5c56837c0a2e49f69fab0aaf01d113bd P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:42.470868 19842 raft_consensus.cc:504] T 5c56837c0a2e49f69fab0aaf01d113bd P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:42.471493 19842 leader_election.cc:288] T 5c56837c0a2e49f69fab0aaf01d113bd P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Requested vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:42.471930 17865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c56837c0a2e49f69fab0aaf01d113bd" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:53:42.472116 17865 raft_consensus.cc:3037] T 5c56837c0a2e49f69fab0aaf01d113bd P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:42.472124 17476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c56837c0a2e49f69fab0aaf01d113bd" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:53:42.472347 17476 raft_consensus.cc:3037] T 5c56837c0a2e49f69fab0aaf01d113bd P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:42.474815 17865 raft_consensus.cc:2454] T 5c56837c0a2e49f69fab0aaf01d113bd P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:53:42.474984 17476 raft_consensus.cc:2454] T 5c56837c0a2e49f69fab0aaf01d113bd P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:53:42.475337 17542 leader_election.cc:302] T 5c56837c0a2e49f69fab0aaf01d113bd P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, 4da690c0fa13407a85c0689ce8de9ade; no voters:
I20240502 01:53:42.475625 19842 raft_consensus.cc:2781] T 5c56837c0a2e49f69fab0aaf01d113bd P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:42.475839 19842 raft_consensus.cc:686] T 5c56837c0a2e49f69fab0aaf01d113bd P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Becoming Leader. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Running, Role: LEADER
I20240502 01:53:42.476373 19842 consensus_queue.cc:237] T 5c56837c0a2e49f69fab0aaf01d113bd P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:42.480221 17333 catalog_manager.cc:5484] T 5c56837c0a2e49f69fab0aaf01d113bd P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66). New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } }
W20240502 01:53:42.853067 20005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:42.853353 20005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:42.860885 20005 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_18'
I20240502 01:53:42.902163 19983 consensus_queue.cc:1035] T 9fe583045886494ca8cf656ad2f83189 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:42.912837 20028 consensus_queue.cc:1035] T 9fe583045886494ca8cf656ad2f83189 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:42.951220 19738 consensus_queue.cc:1035] T 1d4efb3b1e3e4fa3bde6305ac993596e P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:42.965798 19836 consensus_queue.cc:1035] T 1d4efb3b1e3e4fa3bde6305ac993596e P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:43.075640 19930 consensus_queue.cc:1035] T 5c56837c0a2e49f69fab0aaf01d113bd P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:43.103322 19842 consensus_queue.cc:1035] T 5c56837c0a2e49f69fab0aaf01d113bd P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 291.356 ms
time per row: 0.145678 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 01:53:44.188648 17326 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalance_test_table_19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:44.199165 17715 tablet_service.cc:1467] Processing CreateTablet for tablet 4099f4b2ae164020839faab53957750f (DEFAULT_TABLE table=rebalance_test_table_19 [id=b69eb497d10340fea8df98ad03a1e465]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:44.199373 17585 tablet_service.cc:1467] Processing CreateTablet for tablet 4099f4b2ae164020839faab53957750f (DEFAULT_TABLE table=rebalance_test_table_19 [id=b69eb497d10340fea8df98ad03a1e465]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:44.199678 17715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4099f4b2ae164020839faab53957750f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:44.199854 17585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4099f4b2ae164020839faab53957750f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:44.200341 17454 tablet_service.cc:1467] Processing CreateTablet for tablet 4099f4b2ae164020839faab53957750f (DEFAULT_TABLE table=rebalance_test_table_19 [id=b69eb497d10340fea8df98ad03a1e465]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:44.200764 17454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4099f4b2ae164020839faab53957750f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:44.205392 17846 tablet_service.cc:1467] Processing CreateTablet for tablet 71cbc012efe14fdb937302796a9f84e5 (DEFAULT_TABLE table=rebalance_test_table_19 [id=b69eb497d10340fea8df98ad03a1e465]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:44.205540 17587 tablet_service.cc:1467] Processing CreateTablet for tablet 59c288e7279f4119afb6c3bf860e6dc3 (DEFAULT_TABLE table=rebalance_test_table_19 [id=b69eb497d10340fea8df98ad03a1e465]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:44.205694 17845 tablet_service.cc:1467] Processing CreateTablet for tablet 59c288e7279f4119afb6c3bf860e6dc3 (DEFAULT_TABLE table=rebalance_test_table_19 [id=b69eb497d10340fea8df98ad03a1e465]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:44.205811 17846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71cbc012efe14fdb937302796a9f84e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:44.205909 17587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59c288e7279f4119afb6c3bf860e6dc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:44.206068 17845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59c288e7279f4119afb6c3bf860e6dc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:44.209658 17456 tablet_service.cc:1467] Processing CreateTablet for tablet 71cbc012efe14fdb937302796a9f84e5 (DEFAULT_TABLE table=rebalance_test_table_19 [id=b69eb497d10340fea8df98ad03a1e465]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:44.210043 17456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71cbc012efe14fdb937302796a9f84e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:44.212204 17716 tablet_service.cc:1467] Processing CreateTablet for tablet 71cbc012efe14fdb937302796a9f84e5 (DEFAULT_TABLE table=rebalance_test_table_19 [id=b69eb497d10340fea8df98ad03a1e465]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:44.212613 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71cbc012efe14fdb937302796a9f84e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:44.213511 17717 tablet_service.cc:1467] Processing CreateTablet for tablet 59c288e7279f4119afb6c3bf860e6dc3 (DEFAULT_TABLE table=rebalance_test_table_19 [id=b69eb497d10340fea8df98ad03a1e465]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:44.213900 17717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59c288e7279f4119afb6c3bf860e6dc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:44.220458 20074 tablet_bootstrap.cc:492] T 4099f4b2ae164020839faab53957750f P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:44.222123 20074 tablet_bootstrap.cc:654] T 4099f4b2ae164020839faab53957750f P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:44.227283 20074 tablet_bootstrap.cc:492] T 4099f4b2ae164020839faab53957750f P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:44.227515 20074 ts_tablet_manager.cc:1367] T 4099f4b2ae164020839faab53957750f P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240502 01:53:44.228273 20074 raft_consensus.cc:348] T 4099f4b2ae164020839faab53957750f P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:44.228478 20074 raft_consensus.cc:374] T 4099f4b2ae164020839faab53957750f P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:44.228571 20074 raft_consensus.cc:729] T 4099f4b2ae164020839faab53957750f P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:44.228811 20074 consensus_queue.cc:260] T 4099f4b2ae164020839faab53957750f P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:44.229377 20074 ts_tablet_manager.cc:1398] T 4099f4b2ae164020839faab53957750f P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:44.229655 20074 tablet_bootstrap.cc:492] T 59c288e7279f4119afb6c3bf860e6dc3 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:44.231156 20074 tablet_bootstrap.cc:654] T 59c288e7279f4119afb6c3bf860e6dc3 P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:44.240015 20077 tablet_bootstrap.cc:492] T 59c288e7279f4119afb6c3bf860e6dc3 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:44.241969 20077 tablet_bootstrap.cc:654] T 59c288e7279f4119afb6c3bf860e6dc3 P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:44.242383 20073 tablet_bootstrap.cc:492] T 4099f4b2ae164020839faab53957750f P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:44.243870 20072 tablet_bootstrap.cc:492] T 4099f4b2ae164020839faab53957750f P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:44.245086 20072 tablet_bootstrap.cc:654] T 4099f4b2ae164020839faab53957750f P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:44.245234 20073 tablet_bootstrap.cc:654] T 4099f4b2ae164020839faab53957750f P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:44.246546 20074 tablet_bootstrap.cc:492] T 59c288e7279f4119afb6c3bf860e6dc3 P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:44.246773 20074 ts_tablet_manager.cc:1367] T 59c288e7279f4119afb6c3bf860e6dc3 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.007s
I20240502 01:53:44.248072 20074 raft_consensus.cc:348] T 59c288e7279f4119afb6c3bf860e6dc3 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:44.248322 20074 raft_consensus.cc:374] T 59c288e7279f4119afb6c3bf860e6dc3 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:44.248395 20074 raft_consensus.cc:729] T 59c288e7279f4119afb6c3bf860e6dc3 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:44.248623 20074 consensus_queue.cc:260] T 59c288e7279f4119afb6c3bf860e6dc3 P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:44.248911 20077 tablet_bootstrap.cc:492] T 59c288e7279f4119afb6c3bf860e6dc3 P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:44.249109 20077 ts_tablet_manager.cc:1367] T 59c288e7279f4119afb6c3bf860e6dc3 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240502 01:53:44.249318 20074 ts_tablet_manager.cc:1398] T 59c288e7279f4119afb6c3bf860e6dc3 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:53:44.249933 20077 raft_consensus.cc:348] T 59c288e7279f4119afb6c3bf860e6dc3 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:44.250169 20077 raft_consensus.cc:374] T 59c288e7279f4119afb6c3bf860e6dc3 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:44.250299 20077 raft_consensus.cc:729] T 59c288e7279f4119afb6c3bf860e6dc3 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:44.250532 20077 consensus_queue.cc:260] T 59c288e7279f4119afb6c3bf860e6dc3 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:44.252169 20077 ts_tablet_manager.cc:1398] T 59c288e7279f4119afb6c3bf860e6dc3 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:44.252493 20077 tablet_bootstrap.cc:492] T 71cbc012efe14fdb937302796a9f84e5 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:44.254048 20077 tablet_bootstrap.cc:654] T 71cbc012efe14fdb937302796a9f84e5 P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:44.255337 20072 tablet_bootstrap.cc:492] T 4099f4b2ae164020839faab53957750f P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:44.255476 20072 ts_tablet_manager.cc:1367] T 4099f4b2ae164020839faab53957750f P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.008s
I20240502 01:53:44.256369 20072 raft_consensus.cc:348] T 4099f4b2ae164020839faab53957750f P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:44.256762 20072 raft_consensus.cc:374] T 4099f4b2ae164020839faab53957750f P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:44.256947 20072 raft_consensus.cc:729] T 4099f4b2ae164020839faab53957750f P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:44.257242 20072 consensus_queue.cc:260] T 4099f4b2ae164020839faab53957750f P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:44.257870 20072 ts_tablet_manager.cc:1398] T 4099f4b2ae164020839faab53957750f P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:44.258358 20072 tablet_bootstrap.cc:492] T 59c288e7279f4119afb6c3bf860e6dc3 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:44.260007 20072 tablet_bootstrap.cc:654] T 59c288e7279f4119afb6c3bf860e6dc3 P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:44.261756 20073 tablet_bootstrap.cc:492] T 4099f4b2ae164020839faab53957750f P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:44.261970 20073 ts_tablet_manager.cc:1367] T 4099f4b2ae164020839faab53957750f P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.007s
I20240502 01:53:44.262817 20073 raft_consensus.cc:348] T 4099f4b2ae164020839faab53957750f P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:44.263108 20073 raft_consensus.cc:374] T 4099f4b2ae164020839faab53957750f P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:44.263250 20073 raft_consensus.cc:729] T 4099f4b2ae164020839faab53957750f P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:44.263511 20073 consensus_queue.cc:260] T 4099f4b2ae164020839faab53957750f P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:44.264180 20073 ts_tablet_manager.cc:1398] T 4099f4b2ae164020839faab53957750f P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:53:44.265358 20073 tablet_bootstrap.cc:492] T 71cbc012efe14fdb937302796a9f84e5 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:44.266010 20072 tablet_bootstrap.cc:492] T 59c288e7279f4119afb6c3bf860e6dc3 P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:44.266206 20072 ts_tablet_manager.cc:1367] T 59c288e7279f4119afb6c3bf860e6dc3 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20240502 01:53:44.266935 20073 tablet_bootstrap.cc:654] T 71cbc012efe14fdb937302796a9f84e5 P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:44.266958 20072 raft_consensus.cc:348] T 59c288e7279f4119afb6c3bf860e6dc3 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:44.267194 20072 raft_consensus.cc:374] T 59c288e7279f4119afb6c3bf860e6dc3 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:44.267298 20072 raft_consensus.cc:729] T 59c288e7279f4119afb6c3bf860e6dc3 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:44.267531 20072 consensus_queue.cc:260] T 59c288e7279f4119afb6c3bf860e6dc3 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:44.268111 20072 ts_tablet_manager.cc:1398] T 59c288e7279f4119afb6c3bf860e6dc3 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:53:44.269315 20072 tablet_bootstrap.cc:492] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:44.271065 20072 tablet_bootstrap.cc:654] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:44.273766 20077 tablet_bootstrap.cc:492] T 71cbc012efe14fdb937302796a9f84e5 P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:44.273892 20077 ts_tablet_manager.cc:1367] T 71cbc012efe14fdb937302796a9f84e5 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.000s
I20240502 01:53:44.274531 20077 raft_consensus.cc:348] T 71cbc012efe14fdb937302796a9f84e5 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:44.274746 20077 raft_consensus.cc:374] T 71cbc012efe14fdb937302796a9f84e5 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:44.274821 20077 raft_consensus.cc:729] T 71cbc012efe14fdb937302796a9f84e5 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:44.274982 20072 tablet_bootstrap.cc:492] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:44.274910 20073 tablet_bootstrap.cc:492] T 71cbc012efe14fdb937302796a9f84e5 P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:44.275103 20072 ts_tablet_manager.cc:1367] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20240502 01:53:44.275148 20073 ts_tablet_manager.cc:1367] T 71cbc012efe14fdb937302796a9f84e5 P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20240502 01:53:44.275063 20077 consensus_queue.cc:260] T 71cbc012efe14fdb937302796a9f84e5 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:44.275575 20077 ts_tablet_manager.cc:1398] T 71cbc012efe14fdb937302796a9f84e5 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:44.275779 20072 raft_consensus.cc:348] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:44.275866 20073 raft_consensus.cc:348] T 71cbc012efe14fdb937302796a9f84e5 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:44.275981 20072 raft_consensus.cc:374] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:44.276067 20072 raft_consensus.cc:729] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:44.276068 20073 raft_consensus.cc:374] T 71cbc012efe14fdb937302796a9f84e5 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:44.276137 20073 raft_consensus.cc:729] T 71cbc012efe14fdb937302796a9f84e5 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:44.276302 20072 consensus_queue.cc:260] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:44.276362 20073 consensus_queue.cc:260] T 71cbc012efe14fdb937302796a9f84e5 P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:44.276953 20073 ts_tablet_manager.cc:1398] T 71cbc012efe14fdb937302796a9f84e5 P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:53:44.277391 20072 ts_tablet_manager.cc:1398] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:53:44.308496 20028 raft_consensus.cc:482] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:44.308670 20028 raft_consensus.cc:504] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:44.309428 20028 leader_election.cc:288] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:44.309751 17866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71cbc012efe14fdb937302796a9f84e5" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:53:44.309988 17866 raft_consensus.cc:2454] T 71cbc012efe14fdb937302796a9f84e5 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:53:44.309958 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71cbc012efe14fdb937302796a9f84e5" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:44.310159 17475 raft_consensus.cc:2454] T 71cbc012efe14fdb937302796a9f84e5 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:53:44.310431 17673 leader_election.cc:302] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb31bceba76d466db24b9341c04900c3, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:44.310783 20028 raft_consensus.cc:2781] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:44.310863 20028 raft_consensus.cc:482] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:44.310935 20028 raft_consensus.cc:3037] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:44.312799 20028 raft_consensus.cc:504] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:44.313397 20028 leader_election.cc:288] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Requested vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:44.313721 17866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71cbc012efe14fdb937302796a9f84e5" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:53:44.313788 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71cbc012efe14fdb937302796a9f84e5" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:53:44.313890 17866 raft_consensus.cc:3037] T 71cbc012efe14fdb937302796a9f84e5 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:44.313975 17475 raft_consensus.cc:3037] T 71cbc012efe14fdb937302796a9f84e5 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:44.315668 17866 raft_consensus.cc:2454] T 71cbc012efe14fdb937302796a9f84e5 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:53:44.315881 17475 raft_consensus.cc:2454] T 71cbc012efe14fdb937302796a9f84e5 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:53:44.316063 17673 leader_election.cc:302] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb31bceba76d466db24b9341c04900c3, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:44.316416 20028 raft_consensus.cc:2781] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:44.316556 20028 raft_consensus.cc:686] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Becoming Leader. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Running, Role: LEADER
I20240502 01:53:44.316823 20028 consensus_queue.cc:237] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:44.320154 17335 catalog_manager.cc:5484] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfa1152987644bd781b46c7e42a7a685 (127.11.42.67). New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:44.385979 20028 raft_consensus.cc:482] T 4099f4b2ae164020839faab53957750f P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:44.386122 20028 raft_consensus.cc:504] T 4099f4b2ae164020839faab53957750f P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:44.386804 20028 leader_election.cc:288] T 4099f4b2ae164020839faab53957750f P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:44.387292 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4099f4b2ae164020839faab53957750f" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:53:44.387326 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4099f4b2ae164020839faab53957750f" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:44.387545 17604 raft_consensus.cc:2454] T 4099f4b2ae164020839faab53957750f P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:53:44.387579 17475 raft_consensus.cc:2454] T 4099f4b2ae164020839faab53957750f P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:53:44.387984 17672 leader_election.cc:302] T 4099f4b2ae164020839faab53957750f P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:44.388260 20028 raft_consensus.cc:2781] T 4099f4b2ae164020839faab53957750f P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:44.388382 20028 raft_consensus.cc:482] T 4099f4b2ae164020839faab53957750f P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:44.388474 20028 raft_consensus.cc:3037] T 4099f4b2ae164020839faab53957750f P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:44.390365 20028 raft_consensus.cc:504] T 4099f4b2ae164020839faab53957750f P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:44.390877 20028 leader_election.cc:288] T 4099f4b2ae164020839faab53957750f P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Requested vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:44.391237 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4099f4b2ae164020839faab53957750f" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:53:44.391261 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4099f4b2ae164020839faab53957750f" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:53:44.391427 17475 raft_consensus.cc:3037] T 4099f4b2ae164020839faab53957750f P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:44.391438 17604 raft_consensus.cc:3037] T 4099f4b2ae164020839faab53957750f P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:44.393345 17604 raft_consensus.cc:2454] T 4099f4b2ae164020839faab53957750f P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:53:44.393404 17475 raft_consensus.cc:2454] T 4099f4b2ae164020839faab53957750f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:53:44.393709 17670 leader_election.cc:302] T 4099f4b2ae164020839faab53957750f P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:44.393992 20028 raft_consensus.cc:2781] T 4099f4b2ae164020839faab53957750f P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:44.394138 20028 raft_consensus.cc:686] T 4099f4b2ae164020839faab53957750f P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Becoming Leader. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Running, Role: LEADER
I20240502 01:53:44.394404 20028 consensus_queue.cc:237] T 4099f4b2ae164020839faab53957750f P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:44.397533 17335 catalog_manager.cc:5484] T 4099f4b2ae164020839faab53957750f P dfa1152987644bd781b46c7e42a7a685 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfa1152987644bd781b46c7e42a7a685 (127.11.42.67). New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:44.566231 19836 raft_consensus.cc:482] T 59c288e7279f4119afb6c3bf860e6dc3 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:44.566404 19836 raft_consensus.cc:504] T 59c288e7279f4119afb6c3bf860e6dc3 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:44.566951 19836 leader_election.cc:288] T 59c288e7279f4119afb6c3bf860e6dc3 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:44.567294 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59c288e7279f4119afb6c3bf860e6dc3" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:53:44.567364 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59c288e7279f4119afb6c3bf860e6dc3" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:53:44.567512 17736 raft_consensus.cc:2454] T 59c288e7279f4119afb6c3bf860e6dc3 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 0.
I20240502 01:53:44.567564 17604 raft_consensus.cc:2454] T 59c288e7279f4119afb6c3bf860e6dc3 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 0.
I20240502 01:53:44.567905 17802 leader_election.cc:302] T 59c288e7279f4119afb6c3bf860e6dc3 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb31bceba76d466db24b9341c04900c3, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:44.568173 19836 raft_consensus.cc:2781] T 59c288e7279f4119afb6c3bf860e6dc3 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:44.568253 19836 raft_consensus.cc:482] T 59c288e7279f4119afb6c3bf860e6dc3 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:44.568328 19836 raft_consensus.cc:3037] T 59c288e7279f4119afb6c3bf860e6dc3 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:44.570303 19836 raft_consensus.cc:504] T 59c288e7279f4119afb6c3bf860e6dc3 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:44.570820 19836 leader_election.cc:288] T 59c288e7279f4119afb6c3bf860e6dc3 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 election: Requested vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:44.571107 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59c288e7279f4119afb6c3bf860e6dc3" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:53:44.571280 17736 raft_consensus.cc:3037] T 59c288e7279f4119afb6c3bf860e6dc3 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:44.571295 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59c288e7279f4119afb6c3bf860e6dc3" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:53:44.571472 17604 raft_consensus.cc:3037] T 59c288e7279f4119afb6c3bf860e6dc3 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:44.573200 17736 raft_consensus.cc:2454] T 59c288e7279f4119afb6c3bf860e6dc3 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 1.
I20240502 01:53:44.573289 17604 raft_consensus.cc:2454] T 59c288e7279f4119afb6c3bf860e6dc3 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 1.
I20240502 01:53:44.573614 17802 leader_election.cc:302] T 59c288e7279f4119afb6c3bf860e6dc3 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb31bceba76d466db24b9341c04900c3, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:44.573884 19836 raft_consensus.cc:2781] T 59c288e7279f4119afb6c3bf860e6dc3 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:44.574054 19836 raft_consensus.cc:686] T 59c288e7279f4119afb6c3bf860e6dc3 P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Becoming Leader. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Running, Role: LEADER
I20240502 01:53:44.574307 19836 consensus_queue.cc:237] T 59c288e7279f4119afb6c3bf860e6dc3 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:44.577509 17335 catalog_manager.cc:5484] T 59c288e7279f4119afb6c3bf860e6dc3 P bb31bceba76d466db24b9341c04900c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb31bceba76d466db24b9341c04900c3 (127.11.42.68). New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:44.754009 20028 consensus_queue.cc:1035] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:44.762750 19983 consensus_queue.cc:1035] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:44.805311 20028 consensus_queue.cc:1035] T 4099f4b2ae164020839faab53957750f P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:44.813199 20028 consensus_queue.cc:1035] T 4099f4b2ae164020839faab53957750f P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:44.933933 20089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:44.934234 20089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:44.942512 20089 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_19'
I20240502 01:53:44.984972 20063 consensus_queue.cc:1035] T 59c288e7279f4119afb6c3bf860e6dc3 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:53:44.991523 19836 consensus_queue.cc:1035] T 59c288e7279f4119afb6c3bf860e6dc3 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 216.775 ms
time per row: 0.108387 ms
I20240502 01:53:45.317112 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.69:34339
--local_ip_for_outbound_sockets=127.11.42.69
--tserver_master_addrs=127.11.42.126:39453
--webserver_port=37073
--webserver_interface=127.11.42.69
--builtin_ntp_servers=127.11.42.84:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 01:53:45.651620 20150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:45.651901 20150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:45.652022 20150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:45.660086 20150 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:53:45.660207 20150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:45.660369 20150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.69
I20240502 01:53:45.669581 20150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.69:34339
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.11.42.69
--webserver_port=37073
--tserver_master_addrs=127.11.42.126:39453
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.69
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:45.670904 20150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:45.672562 20150 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:45.684520 20155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:45.684885 20156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:45.685257 20158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:45.685794 20150 server_base.cc:1034] running on GCE node
I20240502 01:53:45.686436 20150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:45.687459 20150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:45.688701 20150 hybrid_clock.cc:648] HybridClock initialized: now 1714614825688644 us; error 50 us; skew 500 ppm
I20240502 01:53:45.689044 20150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:45.691316 20150 webserver.cc:460] Webserver started at http://127.11.42.69:37073/ using document root <none> and password file <none>
I20240502 01:53:45.692014 20150 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:45.692144 20150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:45.697543 20150 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20240502 01:53:45.700598 20164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:45.702154 20150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:53:45.702313 20150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "fdc2ac372492454c85387589a91f63af"
format_stamp: "Formatted at 2024-05-02 01:53:11 on dist-test-slave-xvk5"
I20240502 01:53:45.703334 20150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:45.734818 20150 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:45.735651 20150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:45.735919 20150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:45.736893 20150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:45.738754 20150 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:45.738830 20150 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:45.738943 20150 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:45.738991 20150 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:45.768362 20150 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.69:34339
I20240502 01:53:45.768402 20277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.69:34339 every 8 connection(s)
I20240502 01:53:45.769945 20150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:53:45.778115 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 20150
I20240502 01:53:45.778695 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.70:39155
--local_ip_for_outbound_sockets=127.11.42.70
--tserver_master_addrs=127.11.42.126:39453
--webserver_port=45687
--webserver_interface=127.11.42.70
--builtin_ntp_servers=127.11.42.84:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:53:45.783140 20278 heartbeater.cc:347] Connected to a master server at 127.11.42.126:39453
I20240502 01:53:45.783457 20278 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:45.784436 20278 heartbeater.cc:511] Master 127.11.42.126:39453 requested a full tablet report, sending...
I20240502 01:53:45.786217 17333 ts_manager.cc:194] Re-registered known tserver with Master: fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:53:45.787027 17333 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.69:35451
W20240502 01:53:46.113869 20282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:46.114151 20282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:46.114287 20282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:46.122898 20282 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:53:46.123024 20282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:46.123193 20282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.70
I20240502 01:53:46.137229 20282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.70:39155
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.11.42.70
--webserver_port=45687
--tserver_master_addrs=127.11.42.126:39453
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.70
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:46.138679 20282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:46.140753 20282 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:46.154274 20289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:46.155098 20290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:46.158833 20292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:46.159667 20282 server_base.cc:1034] running on GCE node
I20240502 01:53:46.160270 20282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:46.161155 20282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:46.162353 20282 hybrid_clock.cc:648] HybridClock initialized: now 1714614826162295 us; error 50 us; skew 500 ppm
I20240502 01:53:46.162636 20282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:46.164811 20282 webserver.cc:460] Webserver started at http://127.11.42.70:45687/ using document root <none> and password file <none>
I20240502 01:53:46.165581 20282 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:46.165699 20282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:46.170914 20282 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 01:53:46.176362 20298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:46.177667 20282 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20240502 01:53:46.177814 20282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
format_stamp: "Formatted at 2024-05-02 01:53:11 on dist-test-slave-xvk5"
I20240502 01:53:46.178649 20282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:46.203722 20282 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:46.204572 20282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:46.204815 20282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:46.205754 20282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:46.207278 20282 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:46.207350 20282 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:46.207444 20282 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:46.207487 20282 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:46.238133 20282 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.70:39155
I20240502 01:53:46.238163 20411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.70:39155 every 8 connection(s)
I20240502 01:53:46.239534 20282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 01:53:46.242027 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 20282
I20240502 01:53:46.242699 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.71:38623
--local_ip_for_outbound_sockets=127.11.42.71
--tserver_master_addrs=127.11.42.126:39453
--webserver_port=36651
--webserver_interface=127.11.42.71
--builtin_ntp_servers=127.11.42.84:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:53:46.252131 20412 heartbeater.cc:347] Connected to a master server at 127.11.42.126:39453
I20240502 01:53:46.252395 20412 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:46.253145 20412 heartbeater.cc:511] Master 127.11.42.126:39453 requested a full tablet report, sending...
I20240502 01:53:46.254470 17333 ts_manager.cc:194] Re-registered known tserver with Master: 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:53:46.255357 17333 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.70:59593
W20240502 01:53:46.576403 20415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:46.576685 20415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:46.576813 20415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:46.584960 20415 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:53:46.585111 20415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:46.585286 20415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.71
I20240502 01:53:46.594753 20415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.71:38623
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.11.42.71
--webserver_port=36651
--tserver_master_addrs=127.11.42.126:39453
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.71
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:46.595978 20415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:46.598055 20415 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:46.610461 20421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:46.610939 20424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:46.611315 20422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:46.611821 20415 server_base.cc:1034] running on GCE node
I20240502 01:53:46.612365 20415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:46.613201 20415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:46.614404 20415 hybrid_clock.cc:648] HybridClock initialized: now 1714614826614348 us; error 37 us; skew 500 ppm
I20240502 01:53:46.614732 20415 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:46.616997 20415 webserver.cc:460] Webserver started at http://127.11.42.71:36651/ using document root <none> and password file <none>
I20240502 01:53:46.617740 20415 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:46.617887 20415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:46.622678 20415 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20240502 01:53:46.625341 20430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:46.626621 20415 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 01:53:46.626775 20415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "2f306549407b4bb281ccf2994035e4f1"
format_stamp: "Formatted at 2024-05-02 01:53:12 on dist-test-slave-xvk5"
I20240502 01:53:46.627566 20415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:46.666483 20415 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:46.667273 20415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:46.667526 20415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:46.668576 20415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:46.670198 20415 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:46.670289 20415 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:46.670411 20415 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:46.670459 20415 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:46.701509 20415 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.71:38623
I20240502 01:53:46.701598 20542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.71:38623 every 8 connection(s)
I20240502 01:53:46.703035 20415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 01:53:46.712956 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 20415
I20240502 01:53:46.713688 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.72:42411
--local_ip_for_outbound_sockets=127.11.42.72
--tserver_master_addrs=127.11.42.126:39453
--webserver_port=46559
--webserver_interface=127.11.42.72
--builtin_ntp_servers=127.11.42.84:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:53:46.716167 20543 heartbeater.cc:347] Connected to a master server at 127.11.42.126:39453
I20240502 01:53:46.716454 20543 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:46.717371 20543 heartbeater.cc:511] Master 127.11.42.126:39453 requested a full tablet report, sending...
I20240502 01:53:46.718909 17333 ts_manager.cc:194] Re-registered known tserver with Master: 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:53:46.719946 17333 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.71:43015
I20240502 01:53:46.789644 20278 heartbeater.cc:503] Master 127.11.42.126:39453 was elected leader, sending a full tablet report...
W20240502 01:53:47.066237 20546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:47.066486 20546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:47.066587 20546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:47.073966 20546 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:53:47.074077 20546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:47.074230 20546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.72
I20240502 01:53:47.083905 20546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.72:42411
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.11.42.72
--webserver_port=46559
--tserver_master_addrs=127.11.42.126:39453
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.72
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:47.085175 20546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:47.086898 20546 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:47.100018 20551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:47.101402 20555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:47.101836 20546 server_base.cc:1034] running on GCE node
W20240502 01:53:47.102130 20552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:47.102741 20546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:47.103885 20546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:47.105108 20546 hybrid_clock.cc:648] HybridClock initialized: now 1714614827105048 us; error 47 us; skew 500 ppm
I20240502 01:53:47.105576 20546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:47.107685 20546 webserver.cc:460] Webserver started at http://127.11.42.72:46559/ using document root <none> and password file <none>
I20240502 01:53:47.108443 20546 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:47.108593 20546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:47.114051 20546 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20240502 01:53:47.116573 20561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:47.117769 20546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 01:53:47.117899 20546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "f19cc6cb1e76460aa6711e49a5775091"
format_stamp: "Formatted at 2024-05-02 01:53:12 on dist-test-slave-xvk5"
I20240502 01:53:47.118651 20546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:47.143963 20546 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:47.144744 20546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:47.145061 20546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:47.145979 20546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:47.147536 20546 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:47.147611 20546 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:47.147732 20546 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:47.147789 20546 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:47.190250 20546 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.72:42411
I20240502 01:53:47.190371 20674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.72:42411 every 8 connection(s)
I20240502 01:53:47.191726 20546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 01:53:47.196595 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 20546
I20240502 01:53:47.205116 20675 heartbeater.cc:347] Connected to a master server at 127.11.42.126:39453
I20240502 01:53:47.205389 20675 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:47.206223 20675 heartbeater.cc:511] Master 127.11.42.126:39453 requested a full tablet report, sending...
I20240502 01:53:47.207667 17333 ts_manager.cc:194] Re-registered known tserver with Master: f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:53:47.208503 17333 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.72:60095
I20240502 01:53:47.210143 17333 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:53:47.220507 17585 tablet_service.cc:1467] Processing CreateTablet for tablet 9f47b05574944b2e9d35b0d90f95a8d1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=250c3fd291e940b5a1f68621cb9e98dc]), partition=
I20240502 01:53:47.220947 17585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f47b05574944b2e9d35b0d90f95a8d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:47.250680 20691 tablet_bootstrap.cc:492] T 9f47b05574944b2e9d35b0d90f95a8d1 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:47.253113 20691 tablet_bootstrap.cc:654] T 9f47b05574944b2e9d35b0d90f95a8d1 P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:47.253135 20478 tablet_service.cc:1467] Processing CreateTablet for tablet 9f47b05574944b2e9d35b0d90f95a8d1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=250c3fd291e940b5a1f68621cb9e98dc]), partition=
I20240502 01:53:47.254225 20478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f47b05574944b2e9d35b0d90f95a8d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:47.257906 20412 heartbeater.cc:503] Master 127.11.42.126:39453 was elected leader, sending a full tablet report...
I20240502 01:53:47.260674 20610 tablet_service.cc:1467] Processing CreateTablet for tablet 9f47b05574944b2e9d35b0d90f95a8d1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=250c3fd291e940b5a1f68621cb9e98dc]), partition=
I20240502 01:53:47.262046 20610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f47b05574944b2e9d35b0d90f95a8d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:47.264142 20691 tablet_bootstrap.cc:492] T 9f47b05574944b2e9d35b0d90f95a8d1 P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:47.264281 20691 ts_tablet_manager.cc:1367] T 9f47b05574944b2e9d35b0d90f95a8d1 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.008s
I20240502 01:53:47.264866 20691 raft_consensus.cc:348] T 9f47b05574944b2e9d35b0d90f95a8d1 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:47.265204 20691 raft_consensus.cc:374] T 9f47b05574944b2e9d35b0d90f95a8d1 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:47.265323 20691 raft_consensus.cc:729] T 9f47b05574944b2e9d35b0d90f95a8d1 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:47.265566 20691 consensus_queue.cc:260] T 9f47b05574944b2e9d35b0d90f95a8d1 P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:47.266201 20691 ts_tablet_manager.cc:1398] T 9f47b05574944b2e9d35b0d90f95a8d1 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:53:47.281455 20693 tablet_bootstrap.cc:492] T 9f47b05574944b2e9d35b0d90f95a8d1 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:53:47.286180 20693 tablet_bootstrap.cc:654] T 9f47b05574944b2e9d35b0d90f95a8d1 P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:47.289243 20693 log.cc:824] T 9f47b05574944b2e9d35b0d90f95a8d1 P 2f306549407b4bb281ccf2994035e4f1: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:47.290402 20694 tablet_bootstrap.cc:492] T 9f47b05574944b2e9d35b0d90f95a8d1 P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:53:47.293478 20693 tablet_bootstrap.cc:492] T 9f47b05574944b2e9d35b0d90f95a8d1 P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:53:47.294027 20694 tablet_bootstrap.cc:654] T 9f47b05574944b2e9d35b0d90f95a8d1 P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:47.294605 20693 ts_tablet_manager.cc:1367] T 9f47b05574944b2e9d35b0d90f95a8d1 P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.002s
I20240502 01:53:47.295042 20694 log.cc:824] T 9f47b05574944b2e9d35b0d90f95a8d1 P f19cc6cb1e76460aa6711e49a5775091: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:47.297151 20694 tablet_bootstrap.cc:492] T 9f47b05574944b2e9d35b0d90f95a8d1 P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:53:47.297379 20694 ts_tablet_manager.cc:1367] T 9f47b05574944b2e9d35b0d90f95a8d1 P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240502 01:53:47.303658 20693 raft_consensus.cc:348] T 9f47b05574944b2e9d35b0d90f95a8d1 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:47.304054 20693 raft_consensus.cc:374] T 9f47b05574944b2e9d35b0d90f95a8d1 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:47.304144 20693 raft_consensus.cc:729] T 9f47b05574944b2e9d35b0d90f95a8d1 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:53:47.304273 20694 raft_consensus.cc:348] T 9f47b05574944b2e9d35b0d90f95a8d1 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:47.304656 20694 raft_consensus.cc:374] T 9f47b05574944b2e9d35b0d90f95a8d1 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:47.304747 20694 raft_consensus.cc:729] T 9f47b05574944b2e9d35b0d90f95a8d1 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:53:47.304872 20693 consensus_queue.cc:260] T 9f47b05574944b2e9d35b0d90f95a8d1 P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:47.305429 20694 consensus_queue.cc:260] T 9f47b05574944b2e9d35b0d90f95a8d1 P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:47.309640 20543 heartbeater.cc:503] Master 127.11.42.126:39453 was elected leader, sending a full tablet report...
I20240502 01:53:47.310326 20693 ts_tablet_manager.cc:1398] T 9f47b05574944b2e9d35b0d90f95a8d1 P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20240502 01:53:47.311192 20694 ts_tablet_manager.cc:1398] T 9f47b05574944b2e9d35b0d90f95a8d1 P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20240502 01:53:47.311515 20675 heartbeater.cc:503] Master 127.11.42.126:39453 was elected leader, sending a full tablet report...
W20240502 01:53:47.356093 17653 tablet.cc:2369] T 9f47b05574944b2e9d35b0d90f95a8d1 P 3fceccefba274b1dbbd8216af8b0bc29: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:53:47.403434 20087 raft_consensus.cc:482] T 9f47b05574944b2e9d35b0d90f95a8d1 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:47.403621 20087 raft_consensus.cc:504] T 9f47b05574944b2e9d35b0d90f95a8d1 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:47.404475 20087 leader_election.cc:288] T 9f47b05574944b2e9d35b0d90f95a8d1 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:53:47.420840 20498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f47b05574944b2e9d35b0d90f95a8d1" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
I20240502 01:53:47.421335 20498 raft_consensus.cc:2454] T 9f47b05574944b2e9d35b0d90f95a8d1 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:53:47.421831 17543 leader_election.cc:302] T 9f47b05574944b2e9d35b0d90f95a8d1 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 3fceccefba274b1dbbd8216af8b0bc29; no voters:
I20240502 01:53:47.422077 20087 raft_consensus.cc:2781] T 9f47b05574944b2e9d35b0d90f95a8d1 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:47.422194 20087 raft_consensus.cc:482] T 9f47b05574944b2e9d35b0d90f95a8d1 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:47.422279 20087 raft_consensus.cc:3037] T 9f47b05574944b2e9d35b0d90f95a8d1 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:47.422523 20630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f47b05574944b2e9d35b0d90f95a8d1" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091" is_pre_election: true
I20240502 01:53:47.423004 20630 raft_consensus.cc:2454] T 9f47b05574944b2e9d35b0d90f95a8d1 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:53:47.424826 20087 raft_consensus.cc:504] T 9f47b05574944b2e9d35b0d90f95a8d1 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:47.425444 20087 leader_election.cc:288] T 9f47b05574944b2e9d35b0d90f95a8d1 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Requested vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:53:47.425901 20498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f47b05574944b2e9d35b0d90f95a8d1" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
I20240502 01:53:47.425959 20630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f47b05574944b2e9d35b0d90f95a8d1" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091"
I20240502 01:53:47.426050 20498 raft_consensus.cc:3037] T 9f47b05574944b2e9d35b0d90f95a8d1 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:47.426164 20630 raft_consensus.cc:3037] T 9f47b05574944b2e9d35b0d90f95a8d1 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:47.428165 20498 raft_consensus.cc:2454] T 9f47b05574944b2e9d35b0d90f95a8d1 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:53:47.428525 17543 leader_election.cc:302] T 9f47b05574944b2e9d35b0d90f95a8d1 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 3fceccefba274b1dbbd8216af8b0bc29; no voters:
I20240502 01:53:47.428778 20087 raft_consensus.cc:2781] T 9f47b05574944b2e9d35b0d90f95a8d1 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:47.428934 20087 raft_consensus.cc:686] T 9f47b05574944b2e9d35b0d90f95a8d1 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Becoming Leader. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Running, Role: LEADER
I20240502 01:53:47.429411 20630 raft_consensus.cc:2454] T 9f47b05574944b2e9d35b0d90f95a8d1 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:53:47.429394 20087 consensus_queue.cc:237] T 9f47b05574944b2e9d35b0d90f95a8d1 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:47.433523 17333 catalog_manager.cc:5484] T 9f47b05574944b2e9d35b0d90f95a8d1 P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66). New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } }
W20240502 01:53:47.443454 20676 tablet.cc:2369] T 9f47b05574944b2e9d35b0d90f95a8d1 P f19cc6cb1e76460aa6711e49a5775091: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:53:47.455054 20544 tablet.cc:2369] T 9f47b05574944b2e9d35b0d90f95a8d1 P 2f306549407b4bb281ccf2994035e4f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:53:47.560204 17333 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:53:47.569552 20630 raft_consensus.cc:1261] T 9f47b05574944b2e9d35b0d90f95a8d1 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:47.570466 20703 consensus_queue.cc:1035] T 9f47b05574944b2e9d35b0d90f95a8d1 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:47.571633 20498 raft_consensus.cc:1261] T 9f47b05574944b2e9d35b0d90f95a8d1 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:47.576045 20703 consensus_queue.cc:1035] T 9f47b05574944b2e9d35b0d90f95a8d1 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:47.590935 17326 catalog_manager.cc:5576] TS 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623) does not have the latest schema for tablet 9f47b05574944b2e9d35b0d90f95a8d1 (table rebalancer_dynamic_table [id=250c3fd291e940b5a1f68621cb9e98dc]). Expected version 1 got 0
I20240502 01:53:47.593688 17333 catalog_manager.cc:5576] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411) does not have the latest schema for tablet 9f47b05574944b2e9d35b0d90f95a8d1 (table rebalancer_dynamic_table [id=250c3fd291e940b5a1f68621cb9e98dc]). Expected version 1 got 0
I20240502 01:53:47.594763 20704 mvcc.cc:205] Tried to move back new op lower bound from 7023062333714132992 to 7023062333155078144. Current Snapshot: MvccSnapshot[applied={T|T < 7023062333714132992}]
I20240502 01:53:47.597568 17333 catalog_manager.cc:5576] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411) does not have the latest schema for tablet 9f47b05574944b2e9d35b0d90f95a8d1 (table rebalancer_dynamic_table [id=250c3fd291e940b5a1f68621cb9e98dc]). Expected version 1 got 0
I20240502 01:53:47.599735 20711 tablet.cc:1722] T 9f47b05574944b2e9d35b0d90f95a8d1 P 2f306549407b4bb281ccf2994035e4f1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:53:47.600742 17333 catalog_manager.cc:5576] TS 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623) does not have the latest schema for tablet 9f47b05574944b2e9d35b0d90f95a8d1 (table rebalancer_dynamic_table [id=250c3fd291e940b5a1f68621cb9e98dc]). Expected version 1 got 0
I20240502 01:53:47.600872 20710 tablet.cc:1722] T 9f47b05574944b2e9d35b0d90f95a8d1 P f19cc6cb1e76460aa6711e49a5775091: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:53:47.601099 20711 tablet.cc:1620] T 9f47b05574944b2e9d35b0d90f95a8d1 P 2f306549407b4bb281ccf2994035e4f1: MemRowSet was empty: no flush needed.
I20240502 01:53:47.604559 20710 tablet.cc:1620] T 9f47b05574944b2e9d35b0d90f95a8d1 P f19cc6cb1e76460aa6711e49a5775091: MemRowSet was empty: no flush needed.
I20240502 01:53:47.616950 20707 mvcc.cc:205] Tried to move back new op lower bound from 7023062333714132992 to 7023062333155078144. Current Snapshot: MvccSnapshot[applied={T|T < 7023062333714132992}]
I20240502 01:53:47.617398 17326 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=250c3fd291e940b5a1f68621cb9e98dc] alter complete (version 1)
I20240502 01:53:47.617587 20709 tablet.cc:1722] T 9f47b05574944b2e9d35b0d90f95a8d1 P 3fceccefba274b1dbbd8216af8b0bc29: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:53:47.618991 20709 tablet.cc:1620] T 9f47b05574944b2e9d35b0d90f95a8d1 P 3fceccefba274b1dbbd8216af8b0bc29: MemRowSet was empty: no flush needed.
I20240502 01:53:47.698984 17326 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240502 01:53:47.699105 17333 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_extra_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:47.699285 17326 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240502 01:53:47.708595 17326 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet bc2c55fdea5c45caaff73a0937a31779
I20240502 01:53:47.711629 17715 tablet_service.cc:1514] Processing DeleteTablet for tablet bc2c55fdea5c45caaff73a0937a31779 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:47 UTC) from {username='slave'} at 127.0.0.1:52198
I20240502 01:53:47.717340 20609 tablet_service.cc:1467] Processing CreateTablet for tablet 31aa6a26f64f49368ad43cc97c66f2f1 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=2da12dd1e8bc4159bc3e1040e3db33fe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:47.718014 20609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31aa6a26f64f49368ad43cc97c66f2f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:47.718039 17454 tablet_service.cc:1467] Processing CreateTablet for tablet 975f253ff03447ff96fe84169c05097e (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=2da12dd1e8bc4159bc3e1040e3db33fe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:47.718415 17454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 975f253ff03447ff96fe84169c05097e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:47.717517 20610 tablet_service.cc:1467] Processing CreateTablet for tablet 975f253ff03447ff96fe84169c05097e (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=2da12dd1e8bc4159bc3e1040e3db33fe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:47.721963 20610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 975f253ff03447ff96fe84169c05097e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:47.722023 17716 tablet_service.cc:1467] Processing CreateTablet for tablet ebb9bca9f1bd41b0857939909b1110e4 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=2da12dd1e8bc4159bc3e1040e3db33fe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:47.722437 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebb9bca9f1bd41b0857939909b1110e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:47.727818 17333 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:53:47.732575 20694 tablet_bootstrap.cc:492] T 975f253ff03447ff96fe84169c05097e P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:53:47.734282 20694 tablet_bootstrap.cc:654] T 975f253ff03447ff96fe84169c05097e P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:47.739882 20709 tablet.cc:1722] T 9f47b05574944b2e9d35b0d90f95a8d1 P 3fceccefba274b1dbbd8216af8b0bc29: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:53:47.740111 20694 tablet_bootstrap.cc:492] T 975f253ff03447ff96fe84169c05097e P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:53:47.740228 20731 tablet_bootstrap.cc:492] T 975f253ff03447ff96fe84169c05097e P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:47.740422 20709 tablet.cc:1620] T 9f47b05574944b2e9d35b0d90f95a8d1 P 3fceccefba274b1dbbd8216af8b0bc29: MemRowSet was empty: no flush needed.
I20240502 01:53:47.740885 20694 ts_tablet_manager.cc:1367] T 975f253ff03447ff96fe84169c05097e P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20240502 01:53:47.741887 20731 tablet_bootstrap.cc:654] T 975f253ff03447ff96fe84169c05097e P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:47.741889 20694 raft_consensus.cc:348] T 975f253ff03447ff96fe84169c05097e P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:47.742081 20694 raft_consensus.cc:374] T 975f253ff03447ff96fe84169c05097e P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:47.742179 20694 raft_consensus.cc:729] T 975f253ff03447ff96fe84169c05097e P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:53:47.742513 20694 consensus_queue.cc:260] T 975f253ff03447ff96fe84169c05097e P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:47.744138 20710 tablet.cc:1722] T 9f47b05574944b2e9d35b0d90f95a8d1 P f19cc6cb1e76460aa6711e49a5775091: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:53:47.744592 20710 tablet.cc:1620] T 9f47b05574944b2e9d35b0d90f95a8d1 P f19cc6cb1e76460aa6711e49a5775091: MemRowSet was empty: no flush needed.
I20240502 01:53:47.747853 20711 tablet.cc:1722] T 9f47b05574944b2e9d35b0d90f95a8d1 P 2f306549407b4bb281ccf2994035e4f1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:53:47.748301 20711 tablet.cc:1620] T 9f47b05574944b2e9d35b0d90f95a8d1 P 2f306549407b4bb281ccf2994035e4f1: MemRowSet was empty: no flush needed.
I20240502 01:53:47.756150 17330 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=250c3fd291e940b5a1f68621cb9e98dc] alter complete (version 2)
I20240502 01:53:47.759485 17326 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet 8f7ccf4d86204a59bcced6ec414fe23b
I20240502 01:53:47.760411 17585 tablet_service.cc:1514] Processing DeleteTablet for tablet bc2c55fdea5c45caaff73a0937a31779 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:47 UTC) from {username='slave'} at 127.0.0.1:55232
I20240502 01:53:47.760588 17326 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet 89213ec8d6f5415abde90e807a1d4684
I20240502 01:53:47.760947 17587 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f7ccf4d86204a59bcced6ec414fe23b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:47 UTC) from {username='slave'} at 127.0.0.1:55232
I20240502 01:53:47.761221 17717 tablet_service.cc:1514] Processing DeleteTablet for tablet 89213ec8d6f5415abde90e807a1d4684 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:47 UTC) from {username='slave'} at 127.0.0.1:52198
I20240502 01:53:47.762099 17845 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f7ccf4d86204a59bcced6ec414fe23b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:47 UTC) from {username='slave'} at 127.0.0.1:47226
I20240502 01:53:47.766305 20694 ts_tablet_manager.cc:1398] T 975f253ff03447ff96fe84169c05097e P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.025s user 0.000s sys 0.002s
I20240502 01:53:47.770084 20694 tablet_bootstrap.cc:492] T 31aa6a26f64f49368ad43cc97c66f2f1 P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:53:47.771731 20694 tablet_bootstrap.cc:654] T 31aa6a26f64f49368ad43cc97c66f2f1 P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:47.771709 17846 tablet_service.cc:1514] Processing DeleteTablet for tablet 89213ec8d6f5415abde90e807a1d4684 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:47 UTC) from {username='slave'} at 127.0.0.1:47226
I20240502 01:53:47.775173 20737 tablet_replica.cc:330] T bc2c55fdea5c45caaff73a0937a31779 P 3fceccefba274b1dbbd8216af8b0bc29: stopping tablet replica
I20240502 01:53:47.775771 20737 raft_consensus.cc:2229] T bc2c55fdea5c45caaff73a0937a31779 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:47.776139 20737 raft_consensus.cc:2258] T bc2c55fdea5c45caaff73a0937a31779 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:47.777529 20737 ts_tablet_manager.cc:1875] T bc2c55fdea5c45caaff73a0937a31779 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:47.788790 20737 ts_tablet_manager.cc:1888] T bc2c55fdea5c45caaff73a0937a31779 P 3fceccefba274b1dbbd8216af8b0bc29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:47.790939 20738 tablet_replica.cc:330] T 8f7ccf4d86204a59bcced6ec414fe23b P bb31bceba76d466db24b9341c04900c3: stopping tablet replica
I20240502 01:53:47.791180 20694 tablet_bootstrap.cc:492] T 31aa6a26f64f49368ad43cc97c66f2f1 P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:53:47.791244 20737 log.cc:1196] T bc2c55fdea5c45caaff73a0937a31779 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/wals/bc2c55fdea5c45caaff73a0937a31779
I20240502 01:53:47.791369 20694 ts_tablet_manager.cc:1367] T 31aa6a26f64f49368ad43cc97c66f2f1 P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.005s
I20240502 01:53:47.792102 20737 ts_tablet_manager.cc:1909] T bc2c55fdea5c45caaff73a0937a31779 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting consensus metadata
I20240502 01:53:47.793762 17454 tablet_service.cc:1514] Processing DeleteTablet for tablet bc2c55fdea5c45caaff73a0937a31779 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:47 UTC) from {username='slave'} at 127.0.0.1:54520
I20240502 01:53:47.794875 20738 raft_consensus.cc:2229] T 8f7ccf4d86204a59bcced6ec414fe23b P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:47.795598 20737 tablet_replica.cc:330] T 8f7ccf4d86204a59bcced6ec414fe23b P 3fceccefba274b1dbbd8216af8b0bc29: stopping tablet replica
I20240502 01:53:47.795711 17319 catalog_manager.cc:4830] TS 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): tablet bc2c55fdea5c45caaff73a0937a31779 (table rebalance_test_table_0 [id=c58427a7a33942e1a0e24f2c6c4f0ea1]) successfully deleted
I20240502 01:53:47.796097 20738 raft_consensus.cc:2258] T 8f7ccf4d86204a59bcced6ec414fe23b P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:47.796432 17456 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f7ccf4d86204a59bcced6ec414fe23b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:47 UTC) from {username='slave'} at 127.0.0.1:54520
I20240502 01:53:47.796798 20737 raft_consensus.cc:2229] T 8f7ccf4d86204a59bcced6ec414fe23b P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:47.797219 17456 tablet_service.cc:1514] Processing DeleteTablet for tablet 89213ec8d6f5415abde90e807a1d4684 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:47 UTC) from {username='slave'} at 127.0.0.1:54520
I20240502 01:53:47.797299 20728 tablet_replica.cc:330] T bc2c55fdea5c45caaff73a0937a31779 P dfa1152987644bd781b46c7e42a7a685: stopping tablet replica
I20240502 01:53:47.797325 20737 raft_consensus.cc:2258] T 8f7ccf4d86204a59bcced6ec414fe23b P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:47.797729 20728 raft_consensus.cc:2229] T bc2c55fdea5c45caaff73a0937a31779 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:47.798342 20728 raft_consensus.cc:2258] T bc2c55fdea5c45caaff73a0937a31779 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:47.798764 20694 raft_consensus.cc:348] T 31aa6a26f64f49368ad43cc97c66f2f1 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:47.799041 20694 raft_consensus.cc:374] T 31aa6a26f64f49368ad43cc97c66f2f1 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:47.799228 20737 ts_tablet_manager.cc:1875] T 8f7ccf4d86204a59bcced6ec414fe23b P 3fceccefba274b1dbbd8216af8b0bc29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:47.800923 20694 raft_consensus.cc:729] T 31aa6a26f64f49368ad43cc97c66f2f1 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:53:47.801800 20738 ts_tablet_manager.cc:1875] T 8f7ccf4d86204a59bcced6ec414fe23b P bb31bceba76d466db24b9341c04900c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:47.802214 20732 tablet_bootstrap.cc:492] T ebb9bca9f1bd41b0857939909b1110e4 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:47.803886 20732 tablet_bootstrap.cc:654] T ebb9bca9f1bd41b0857939909b1110e4 P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:47.801725 20694 consensus_queue.cc:260] T 31aa6a26f64f49368ad43cc97c66f2f1 P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:47.805898 20694 ts_tablet_manager.cc:1398] T 31aa6a26f64f49368ad43cc97c66f2f1 P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.014s user 0.000s sys 0.003s
I20240502 01:53:47.806201 20728 ts_tablet_manager.cc:1875] T bc2c55fdea5c45caaff73a0937a31779 P dfa1152987644bd781b46c7e42a7a685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:47.810310 20741 tablet_replica.cc:330] T bc2c55fdea5c45caaff73a0937a31779 P 4da690c0fa13407a85c0689ce8de9ade: stopping tablet replica
I20240502 01:53:47.810658 20737 ts_tablet_manager.cc:1888] T 8f7ccf4d86204a59bcced6ec414fe23b P 3fceccefba274b1dbbd8216af8b0bc29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:47.811277 20741 raft_consensus.cc:2229] T bc2c55fdea5c45caaff73a0937a31779 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:47.811702 20741 raft_consensus.cc:2258] T bc2c55fdea5c45caaff73a0937a31779 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:47.811810 20737 log.cc:1196] T 8f7ccf4d86204a59bcced6ec414fe23b P 3fceccefba274b1dbbd8216af8b0bc29: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/wals/8f7ccf4d86204a59bcced6ec414fe23b
I20240502 01:53:47.812788 20737 ts_tablet_manager.cc:1909] T 8f7ccf4d86204a59bcced6ec414fe23b P 3fceccefba274b1dbbd8216af8b0bc29: Deleting consensus metadata
I20240502 01:53:47.813839 20741 ts_tablet_manager.cc:1875] T bc2c55fdea5c45caaff73a0937a31779 P 4da690c0fa13407a85c0689ce8de9ade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:47.815084 20731 tablet_bootstrap.cc:492] T 975f253ff03447ff96fe84169c05097e P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:47.815289 17319 catalog_manager.cc:4830] TS 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): tablet 8f7ccf4d86204a59bcced6ec414fe23b (table rebalance_test_table_0 [id=c58427a7a33942e1a0e24f2c6c4f0ea1]) successfully deleted
I20240502 01:53:47.815347 20731 ts_tablet_manager.cc:1367] T 975f253ff03447ff96fe84169c05097e P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.075s user 0.022s sys 0.018s
I20240502 01:53:47.816193 20732 tablet_bootstrap.cc:492] T ebb9bca9f1bd41b0857939909b1110e4 P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:47.816327 20732 ts_tablet_manager.cc:1367] T ebb9bca9f1bd41b0857939909b1110e4 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20240502 01:53:47.816181 20731 raft_consensus.cc:348] T 975f253ff03447ff96fe84169c05097e P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:47.816457 20731 raft_consensus.cc:374] T 975f253ff03447ff96fe84169c05097e P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:47.816561 20731 raft_consensus.cc:729] T 975f253ff03447ff96fe84169c05097e P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:47.816820 20731 consensus_queue.cc:260] T 975f253ff03447ff96fe84169c05097e P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:47.817612 20731 ts_tablet_manager.cc:1398] T 975f253ff03447ff96fe84169c05097e P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:47.820032 20212 tablet_service.cc:1467] Processing CreateTablet for tablet 31aa6a26f64f49368ad43cc97c66f2f1 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=2da12dd1e8bc4159bc3e1040e3db33fe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:47.821280 20732 raft_consensus.cc:348] T ebb9bca9f1bd41b0857939909b1110e4 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:47.821478 20732 raft_consensus.cc:374] T ebb9bca9f1bd41b0857939909b1110e4 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:47.821453 20211 tablet_service.cc:1467] Processing CreateTablet for tablet ebb9bca9f1bd41b0857939909b1110e4 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=2da12dd1e8bc4159bc3e1040e3db33fe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:47.821565 20732 raft_consensus.cc:729] T ebb9bca9f1bd41b0857939909b1110e4 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:47.821792 20732 consensus_queue.cc:260] T ebb9bca9f1bd41b0857939909b1110e4 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:47.822382 20732 ts_tablet_manager.cc:1398] T ebb9bca9f1bd41b0857939909b1110e4 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 01:53:47.825085 20212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31aa6a26f64f49368ad43cc97c66f2f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:47.827865 20741 ts_tablet_manager.cc:1888] T bc2c55fdea5c45caaff73a0937a31779 P 4da690c0fa13407a85c0689ce8de9ade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:47.828044 20741 log.cc:1196] T bc2c55fdea5c45caaff73a0937a31779 P 4da690c0fa13407a85c0689ce8de9ade: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/bc2c55fdea5c45caaff73a0937a31779
I20240502 01:53:47.828590 20741 ts_tablet_manager.cc:1909] T bc2c55fdea5c45caaff73a0937a31779 P 4da690c0fa13407a85c0689ce8de9ade: Deleting consensus metadata
I20240502 01:53:47.830333 20741 tablet_replica.cc:330] T 8f7ccf4d86204a59bcced6ec414fe23b P 4da690c0fa13407a85c0689ce8de9ade: stopping tablet replica
I20240502 01:53:47.830626 20741 raft_consensus.cc:2229] T 8f7ccf4d86204a59bcced6ec414fe23b P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:47.830601 17321 catalog_manager.cc:4830] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): tablet bc2c55fdea5c45caaff73a0937a31779 (table rebalance_test_table_0 [id=c58427a7a33942e1a0e24f2c6c4f0ea1]) successfully deleted
I20240502 01:53:47.830914 20741 raft_consensus.cc:2258] T 8f7ccf4d86204a59bcced6ec414fe23b P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:47.831310 20211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebb9bca9f1bd41b0857939909b1110e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:47.831655 20741 ts_tablet_manager.cc:1875] T 8f7ccf4d86204a59bcced6ec414fe23b P 4da690c0fa13407a85c0689ce8de9ade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:47.819038 20213 tablet_service.cc:1467] Processing CreateTablet for tablet 975f253ff03447ff96fe84169c05097e (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=2da12dd1e8bc4159bc3e1040e3db33fe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:47.833500 20213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 975f253ff03447ff96fe84169c05097e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:47.839879 20741 ts_tablet_manager.cc:1888] T 8f7ccf4d86204a59bcced6ec414fe23b P 4da690c0fa13407a85c0689ce8de9ade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:47.840097 20741 log.cc:1196] T 8f7ccf4d86204a59bcced6ec414fe23b P 4da690c0fa13407a85c0689ce8de9ade: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/8f7ccf4d86204a59bcced6ec414fe23b
I20240502 01:53:47.840528 20741 ts_tablet_manager.cc:1909] T 8f7ccf4d86204a59bcced6ec414fe23b P 4da690c0fa13407a85c0689ce8de9ade: Deleting consensus metadata
I20240502 01:53:47.841686 17321 catalog_manager.cc:4830] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): tablet 8f7ccf4d86204a59bcced6ec414fe23b (table rebalance_test_table_0 [id=c58427a7a33942e1a0e24f2c6c4f0ea1]) successfully deleted
I20240502 01:53:47.842098 20741 tablet_replica.cc:330] T 89213ec8d6f5415abde90e807a1d4684 P 4da690c0fa13407a85c0689ce8de9ade: stopping tablet replica
I20240502 01:53:47.842344 20741 raft_consensus.cc:2229] T 89213ec8d6f5415abde90e807a1d4684 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:47.842885 20741 raft_consensus.cc:2258] T 89213ec8d6f5415abde90e807a1d4684 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:47.843638 20741 ts_tablet_manager.cc:1875] T 89213ec8d6f5415abde90e807a1d4684 P 4da690c0fa13407a85c0689ce8de9ade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:47.845957 20728 ts_tablet_manager.cc:1888] T bc2c55fdea5c45caaff73a0937a31779 P dfa1152987644bd781b46c7e42a7a685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:47.847386 20347 tablet_service.cc:1467] Processing CreateTablet for tablet 31aa6a26f64f49368ad43cc97c66f2f1 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=2da12dd1e8bc4159bc3e1040e3db33fe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:47.848520 20728 log.cc:1196] T bc2c55fdea5c45caaff73a0937a31779 P dfa1152987644bd781b46c7e42a7a685: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/wals/bc2c55fdea5c45caaff73a0937a31779
I20240502 01:53:47.849082 20347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31aa6a26f64f49368ad43cc97c66f2f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:47.849107 20728 ts_tablet_manager.cc:1909] T bc2c55fdea5c45caaff73a0937a31779 P dfa1152987644bd781b46c7e42a7a685: Deleting consensus metadata
I20240502 01:53:47.850694 17321 catalog_manager.cc:4830] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): tablet bc2c55fdea5c45caaff73a0937a31779 (table rebalance_test_table_0 [id=c58427a7a33942e1a0e24f2c6c4f0ea1]) successfully deleted
I20240502 01:53:47.847455 20346 tablet_service.cc:1467] Processing CreateTablet for tablet ebb9bca9f1bd41b0857939909b1110e4 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=2da12dd1e8bc4159bc3e1040e3db33fe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:47.853231 20728 tablet_replica.cc:330] T 89213ec8d6f5415abde90e807a1d4684 P dfa1152987644bd781b46c7e42a7a685: stopping tablet replica
I20240502 01:53:47.853519 20346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebb9bca9f1bd41b0857939909b1110e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:47.853492 20728 raft_consensus.cc:2229] T 89213ec8d6f5415abde90e807a1d4684 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:47.853739 20728 raft_consensus.cc:2258] T 89213ec8d6f5415abde90e807a1d4684 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:47.855572 20741 ts_tablet_manager.cc:1888] T 89213ec8d6f5415abde90e807a1d4684 P 4da690c0fa13407a85c0689ce8de9ade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:47.855762 20741 log.cc:1196] T 89213ec8d6f5415abde90e807a1d4684 P 4da690c0fa13407a85c0689ce8de9ade: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/89213ec8d6f5415abde90e807a1d4684
I20240502 01:53:47.856381 20741 ts_tablet_manager.cc:1909] T 89213ec8d6f5415abde90e807a1d4684 P 4da690c0fa13407a85c0689ce8de9ade: Deleting consensus metadata
I20240502 01:53:47.857494 17321 catalog_manager.cc:4830] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): tablet 89213ec8d6f5415abde90e807a1d4684 (table rebalance_test_table_0 [id=c58427a7a33942e1a0e24f2c6c4f0ea1]) successfully deleted
I20240502 01:53:47.858108 20728 ts_tablet_manager.cc:1875] T 89213ec8d6f5415abde90e807a1d4684 P dfa1152987644bd781b46c7e42a7a685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:47.859167 17326 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:53:47.864647 20715 raft_consensus.cc:482] T ebb9bca9f1bd41b0857939909b1110e4 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:47.864802 20715 raft_consensus.cc:504] T ebb9bca9f1bd41b0857939909b1110e4 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:47.865634 20715 leader_election.cc:288] T ebb9bca9f1bd41b0857939909b1110e4 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:53:47.869822 20709 tablet.cc:1722] T 9f47b05574944b2e9d35b0d90f95a8d1 P 3fceccefba274b1dbbd8216af8b0bc29: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:53:47.870509 20710 tablet.cc:1722] T 9f47b05574944b2e9d35b0d90f95a8d1 P f19cc6cb1e76460aa6711e49a5775091: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:53:47.874315 20738 ts_tablet_manager.cc:1888] T 8f7ccf4d86204a59bcced6ec414fe23b P bb31bceba76d466db24b9341c04900c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:47.875761 20711 tablet.cc:1722] T 9f47b05574944b2e9d35b0d90f95a8d1 P 2f306549407b4bb281ccf2994035e4f1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
W20240502 01:53:47.886847 17673 leader_election.cc:341] T ebb9bca9f1bd41b0857939909b1110e4 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Not found: Tablet not found: ebb9bca9f1bd41b0857939909b1110e4
I20240502 01:53:47.887343 17326 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table_new_name [id=250c3fd291e940b5a1f68621cb9e98dc] alter complete (version 3)
I20240502 01:53:47.893594 20728 ts_tablet_manager.cc:1888] T 89213ec8d6f5415abde90e807a1d4684 P dfa1152987644bd781b46c7e42a7a685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:47.895679 20728 log.cc:1196] T 89213ec8d6f5415abde90e807a1d4684 P dfa1152987644bd781b46c7e42a7a685: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/wals/89213ec8d6f5415abde90e807a1d4684
I20240502 01:53:47.896201 20728 ts_tablet_manager.cc:1909] T 89213ec8d6f5415abde90e807a1d4684 P dfa1152987644bd781b46c7e42a7a685: Deleting consensus metadata
I20240502 01:53:47.897275 17321 catalog_manager.cc:4830] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): tablet 89213ec8d6f5415abde90e807a1d4684 (table rebalance_test_table_0 [id=c58427a7a33942e1a0e24f2c6c4f0ea1]) successfully deleted
I20240502 01:53:47.899879 20698 raft_consensus.cc:482] T 975f253ff03447ff96fe84169c05097e P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:47.900069 20698 raft_consensus.cc:504] T 975f253ff03447ff96fe84169c05097e P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:47.901911 20698 leader_election.cc:288] T 975f253ff03447ff96fe84169c05097e P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
W20240502 01:53:47.904079 17522 tablet.cc:2369] T 975f253ff03447ff96fe84169c05097e P 4da690c0fa13407a85c0689ce8de9ade: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:53:47.907155 20738 log.cc:1196] T 8f7ccf4d86204a59bcced6ec414fe23b P bb31bceba76d466db24b9341c04900c3: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/wals/8f7ccf4d86204a59bcced6ec414fe23b
I20240502 01:53:47.916699 20698 raft_consensus.cc:482] T 31aa6a26f64f49368ad43cc97c66f2f1 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:47.916895 20698 raft_consensus.cc:504] T 31aa6a26f64f49368ad43cc97c66f2f1 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:47.917632 20698 leader_election.cc:288] T 31aa6a26f64f49368ad43cc97c66f2f1 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:53:47.922044 20738 ts_tablet_manager.cc:1909] T 8f7ccf4d86204a59bcced6ec414fe23b P bb31bceba76d466db24b9341c04900c3: Deleting consensus metadata
I20240502 01:53:47.925186 20738 tablet_replica.cc:330] T 89213ec8d6f5415abde90e807a1d4684 P bb31bceba76d466db24b9341c04900c3: stopping tablet replica
I20240502 01:53:47.927747 20738 raft_consensus.cc:2229] T 89213ec8d6f5415abde90e807a1d4684 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:47.928072 20738 raft_consensus.cc:2258] T 89213ec8d6f5415abde90e807a1d4684 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:47.929464 17322 catalog_manager.cc:4830] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): tablet 8f7ccf4d86204a59bcced6ec414fe23b (table rebalance_test_table_0 [id=c58427a7a33942e1a0e24f2c6c4f0ea1]) successfully deleted
W20240502 01:53:47.932615 17672 leader_election.cc:341] T ebb9bca9f1bd41b0857939909b1110e4 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): Not found: Tablet not found: ebb9bca9f1bd41b0857939909b1110e4
I20240502 01:53:47.932756 17672 leader_election.cc:302] T ebb9bca9f1bd41b0857939909b1110e4 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dfa1152987644bd781b46c7e42a7a685; no voters: 6a36c952c51542c0bf7ea3ec2857a9f9, fdc2ac372492454c85387589a91f63af
I20240502 01:53:47.933051 20299 raft_consensus.cc:2726] T ebb9bca9f1bd41b0857939909b1110e4 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:53:47.928997 20738 ts_tablet_manager.cc:1875] T 89213ec8d6f5415abde90e807a1d4684 P bb31bceba76d466db24b9341c04900c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:47.939401 20749 tablet_bootstrap.cc:492] T ebb9bca9f1bd41b0857939909b1110e4 P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:53:47.943166 20749 tablet_bootstrap.cc:654] T ebb9bca9f1bd41b0857939909b1110e4 P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:47.944536 20749 log.cc:824] T ebb9bca9f1bd41b0857939909b1110e4 P fdc2ac372492454c85387589a91f63af: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:47.951539 20738 ts_tablet_manager.cc:1888] T 89213ec8d6f5415abde90e807a1d4684 P bb31bceba76d466db24b9341c04900c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:47.948938 20233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975f253ff03447ff96fe84169c05097e" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:53:47.953624 20233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31aa6a26f64f49368ad43cc97c66f2f1" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:53:47.954160 20738 log.cc:1196] T 89213ec8d6f5415abde90e807a1d4684 P bb31bceba76d466db24b9341c04900c3: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/wals/89213ec8d6f5415abde90e807a1d4684
W20240502 01:53:47.954439 20565 leader_election.cc:341] T 975f253ff03447ff96fe84169c05097e P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:53:47.954854 20738 ts_tablet_manager.cc:1909] T 89213ec8d6f5415abde90e807a1d4684 P bb31bceba76d466db24b9341c04900c3: Deleting consensus metadata
W20240502 01:53:47.954833 20565 leader_election.cc:341] T 31aa6a26f64f49368ad43cc97c66f2f1 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:53:47.960765 17322 catalog_manager.cc:4830] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): tablet 89213ec8d6f5415abde90e807a1d4684 (table rebalance_test_table_0 [id=c58427a7a33942e1a0e24f2c6c4f0ea1]) successfully deleted
I20240502 01:53:47.961742 20754 tablet_bootstrap.cc:492] T ebb9bca9f1bd41b0857939909b1110e4 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:53:47.964978 17474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975f253ff03447ff96fe84169c05097e" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:47.965236 17474 raft_consensus.cc:2454] T 975f253ff03447ff96fe84169c05097e P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 0.
W20240502 01:53:47.965443 17783 tablet.cc:2369] T ebb9bca9f1bd41b0857939909b1110e4 P dfa1152987644bd781b46c7e42a7a685: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:53:47.965862 20564 leader_election.cc:302] T 975f253ff03447ff96fe84169c05097e P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, f19cc6cb1e76460aa6711e49a5775091; no voters: fdc2ac372492454c85387589a91f63af
I20240502 01:53:47.966321 20698 raft_consensus.cc:2781] T 975f253ff03447ff96fe84169c05097e P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:47.966419 20698 raft_consensus.cc:482] T 975f253ff03447ff96fe84169c05097e P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:47.966513 20698 raft_consensus.cc:3037] T 975f253ff03447ff96fe84169c05097e P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:47.969383 20698 raft_consensus.cc:504] T 975f253ff03447ff96fe84169c05097e P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:47.969798 20749 tablet_bootstrap.cc:492] T ebb9bca9f1bd41b0857939909b1110e4 P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:53:47.970072 20749 ts_tablet_manager.cc:1367] T ebb9bca9f1bd41b0857939909b1110e4 P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.031s user 0.007s sys 0.015s
I20240502 01:53:47.971086 20698 leader_election.cc:288] T 975f253ff03447ff96fe84169c05097e P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 election: Requested vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:47.964428 20367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31aa6a26f64f49368ad43cc97c66f2f1" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:53:47.972033 17474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975f253ff03447ff96fe84169c05097e" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:53:47.972206 17474 raft_consensus.cc:3037] T 975f253ff03447ff96fe84169c05097e P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:47.970460 20233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975f253ff03447ff96fe84169c05097e" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af"
W20240502 01:53:47.975116 20565 leader_election.cc:341] T 975f253ff03447ff96fe84169c05097e P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:53:47.975636 17474 raft_consensus.cc:2454] T 975f253ff03447ff96fe84169c05097e P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 1.
I20240502 01:53:47.975898 20754 tablet_bootstrap.cc:654] T ebb9bca9f1bd41b0857939909b1110e4 P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:47.976153 20564 leader_election.cc:302] T 975f253ff03447ff96fe84169c05097e P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, f19cc6cb1e76460aa6711e49a5775091; no voters: fdc2ac372492454c85387589a91f63af
I20240502 01:53:47.976446 20698 raft_consensus.cc:2781] T 975f253ff03447ff96fe84169c05097e P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:47.977313 20754 log.cc:824] T ebb9bca9f1bd41b0857939909b1110e4 P 6a36c952c51542c0bf7ea3ec2857a9f9: Log is configured to *not* fsync() on all Append() calls
W20240502 01:53:47.979377 20564 leader_election.cc:341] T 31aa6a26f64f49368ad43cc97c66f2f1 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:53:47.979851 20564 leader_election.cc:302] T 31aa6a26f64f49368ad43cc97c66f2f1 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f19cc6cb1e76460aa6711e49a5775091; no voters: 6a36c952c51542c0bf7ea3ec2857a9f9, fdc2ac372492454c85387589a91f63af
I20240502 01:53:47.986351 20754 tablet_bootstrap.cc:492] T ebb9bca9f1bd41b0857939909b1110e4 P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:53:47.986682 20754 ts_tablet_manager.cc:1367] T ebb9bca9f1bd41b0857939909b1110e4 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.008s
I20240502 01:53:47.989356 20762 raft_consensus.cc:2726] T 31aa6a26f64f49368ad43cc97c66f2f1 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:53:47.990373 20749 raft_consensus.cc:348] T ebb9bca9f1bd41b0857939909b1110e4 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:47.991185 20698 raft_consensus.cc:686] T 975f253ff03447ff96fe84169c05097e P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Becoming Leader. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Running, Role: LEADER
I20240502 01:53:47.991204 17326 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:53:47.991400 17326 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:53:47.991816 20698 consensus_queue.cc:237] T 975f253ff03447ff96fe84169c05097e P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:47.993631 20754 raft_consensus.cc:348] T ebb9bca9f1bd41b0857939909b1110e4 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:47.994053 20754 raft_consensus.cc:374] T ebb9bca9f1bd41b0857939909b1110e4 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:47.994204 20754 raft_consensus.cc:729] T ebb9bca9f1bd41b0857939909b1110e4 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:53:47.995012 20754 consensus_queue.cc:260] T ebb9bca9f1bd41b0857939909b1110e4 P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:47.995150 17326 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet 9f47b05574944b2e9d35b0d90f95a8d1
I20240502 01:53:47.996687 20478 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f47b05574944b2e9d35b0d90f95a8d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:47 UTC) from {username='slave'} at 127.0.0.1:35018
I20240502 01:53:47.997023 17585 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f47b05574944b2e9d35b0d90f95a8d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:47 UTC) from {username='slave'} at 127.0.0.1:55232
I20240502 01:53:47.997323 20737 tablet_replica.cc:330] T 9f47b05574944b2e9d35b0d90f95a8d1 P 3fceccefba274b1dbbd8216af8b0bc29: stopping tablet replica
I20240502 01:53:47.997421 20609 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f47b05574944b2e9d35b0d90f95a8d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:47 UTC) from {username='slave'} at 127.0.0.1:33698
I20240502 01:53:47.997557 20737 raft_consensus.cc:2229] T 9f47b05574944b2e9d35b0d90f95a8d1 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:47.998121 20737 raft_consensus.cc:2258] T 9f47b05574944b2e9d35b0d90f95a8d1 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:47.999155 20737 ts_tablet_manager.cc:1875] T 9f47b05574944b2e9d35b0d90f95a8d1 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:48.006031 20737 ts_tablet_manager.cc:1888] T 9f47b05574944b2e9d35b0d90f95a8d1 P 3fceccefba274b1dbbd8216af8b0bc29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:53:48.006210 20737 log.cc:1196] T 9f47b05574944b2e9d35b0d90f95a8d1 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/wals/9f47b05574944b2e9d35b0d90f95a8d1
I20240502 01:53:48.006743 20737 ts_tablet_manager.cc:1909] T 9f47b05574944b2e9d35b0d90f95a8d1 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting consensus metadata
I20240502 01:53:48.009685 17319 catalog_manager.cc:4830] TS 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): tablet 9f47b05574944b2e9d35b0d90f95a8d1 (table rebalancer_dynamic_table_new_name [id=250c3fd291e940b5a1f68621cb9e98dc]) successfully deleted
I20240502 01:53:48.012111 20749 raft_consensus.cc:374] T ebb9bca9f1bd41b0857939909b1110e4 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:48.012297 20749 raft_consensus.cc:729] T ebb9bca9f1bd41b0857939909b1110e4 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:53:48.013682 20749 consensus_queue.cc:260] T ebb9bca9f1bd41b0857939909b1110e4 P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:48.021946 20749 ts_tablet_manager.cc:1398] T ebb9bca9f1bd41b0857939909b1110e4 P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.047s user 0.009s sys 0.000s
I20240502 01:53:48.022667 20767 tablet_replica.cc:330] T 9f47b05574944b2e9d35b0d90f95a8d1 P f19cc6cb1e76460aa6711e49a5775091: stopping tablet replica
W20240502 01:53:48.022986 20279 tablet.cc:2369] T ebb9bca9f1bd41b0857939909b1110e4 P fdc2ac372492454c85387589a91f63af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:53:48.023192 20767 raft_consensus.cc:2229] T 9f47b05574944b2e9d35b0d90f95a8d1 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:48.023507 20767 raft_consensus.cc:2258] T 9f47b05574944b2e9d35b0d90f95a8d1 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:48.024266 20749 tablet_bootstrap.cc:492] T 975f253ff03447ff96fe84169c05097e P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:53:48.025240 20767 ts_tablet_manager.cc:1875] T 9f47b05574944b2e9d35b0d90f95a8d1 P f19cc6cb1e76460aa6711e49a5775091: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:48.026238 20749 tablet_bootstrap.cc:654] T 975f253ff03447ff96fe84169c05097e P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:48.029567 17326 catalog_manager.cc:5484] T 975f253ff03447ff96fe84169c05097e P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: term changed from 0 to 1, leader changed from <none> to f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72). New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:48.030144 20766 tablet_replica.cc:330] T 9f47b05574944b2e9d35b0d90f95a8d1 P 2f306549407b4bb281ccf2994035e4f1: stopping tablet replica
I20240502 01:53:48.030568 20766 raft_consensus.cc:2229] T 9f47b05574944b2e9d35b0d90f95a8d1 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:48.031015 20766 raft_consensus.cc:2258] T 9f47b05574944b2e9d35b0d90f95a8d1 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:48.031244 20749 tablet_bootstrap.cc:492] T 975f253ff03447ff96fe84169c05097e P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:53:48.031379 20749 ts_tablet_manager.cc:1367] T 975f253ff03447ff96fe84169c05097e P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 01:53:48.032435 20749 raft_consensus.cc:348] T 975f253ff03447ff96fe84169c05097e P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:48.032631 20749 raft_consensus.cc:374] T 975f253ff03447ff96fe84169c05097e P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:48.032732 20749 raft_consensus.cc:729] T 975f253ff03447ff96fe84169c05097e P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:53:48.032974 20766 ts_tablet_manager.cc:1875] T 9f47b05574944b2e9d35b0d90f95a8d1 P 2f306549407b4bb281ccf2994035e4f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:48.033057 20749 consensus_queue.cc:260] T 975f253ff03447ff96fe84169c05097e P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:48.034376 20749 ts_tablet_manager.cc:1398] T 975f253ff03447ff96fe84169c05097e P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 01:53:48.034665 20749 tablet_bootstrap.cc:492] T 31aa6a26f64f49368ad43cc97c66f2f1 P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:53:48.036299 20749 tablet_bootstrap.cc:654] T 31aa6a26f64f49368ad43cc97c66f2f1 P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:48.042308 20754 ts_tablet_manager.cc:1398] T ebb9bca9f1bd41b0857939909b1110e4 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.055s user 0.011s sys 0.000s
I20240502 01:53:48.044339 20754 tablet_bootstrap.cc:492] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:53:48.046212 20754 tablet_bootstrap.cc:654] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:48.048362 20749 tablet_bootstrap.cc:492] T 31aa6a26f64f49368ad43cc97c66f2f1 P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:53:48.048534 20749 ts_tablet_manager.cc:1367] T 31aa6a26f64f49368ad43cc97c66f2f1 P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.002s
I20240502 01:53:48.049306 20749 raft_consensus.cc:348] T 31aa6a26f64f49368ad43cc97c66f2f1 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:48.049535 20749 raft_consensus.cc:374] T 31aa6a26f64f49368ad43cc97c66f2f1 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:48.049609 20749 raft_consensus.cc:729] T 31aa6a26f64f49368ad43cc97c66f2f1 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:53:48.049850 20749 consensus_queue.cc:260] T 31aa6a26f64f49368ad43cc97c66f2f1 P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:48.050307 20749 ts_tablet_manager.cc:1398] T 31aa6a26f64f49368ad43cc97c66f2f1 P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:48.051321 20754 tablet_bootstrap.cc:492] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:53:48.051513 20754 ts_tablet_manager.cc:1367] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 01:53:48.052811 20766 ts_tablet_manager.cc:1888] T 9f47b05574944b2e9d35b0d90f95a8d1 P 2f306549407b4bb281ccf2994035e4f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:53:48.055233 20767 ts_tablet_manager.cc:1888] T 9f47b05574944b2e9d35b0d90f95a8d1 P f19cc6cb1e76460aa6711e49a5775091: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:53:48.055461 20766 log.cc:1196] T 9f47b05574944b2e9d35b0d90f95a8d1 P 2f306549407b4bb281ccf2994035e4f1: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal/wals/9f47b05574944b2e9d35b0d90f95a8d1
I20240502 01:53:48.056089 20766 ts_tablet_manager.cc:1909] T 9f47b05574944b2e9d35b0d90f95a8d1 P 2f306549407b4bb281ccf2994035e4f1: Deleting consensus metadata
I20240502 01:53:48.056290 20767 log.cc:1196] T 9f47b05574944b2e9d35b0d90f95a8d1 P f19cc6cb1e76460aa6711e49a5775091: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal/wals/9f47b05574944b2e9d35b0d90f95a8d1
I20240502 01:53:48.056733 20754 raft_consensus.cc:348] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:48.056937 20767 ts_tablet_manager.cc:1909] T 9f47b05574944b2e9d35b0d90f95a8d1 P f19cc6cb1e76460aa6711e49a5775091: Deleting consensus metadata
I20240502 01:53:48.057046 20754 raft_consensus.cc:374] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:48.057195 20754 raft_consensus.cc:729] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:53:48.057641 20754 consensus_queue.cc:260] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:48.059154 17321 catalog_manager.cc:4830] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): tablet 9f47b05574944b2e9d35b0d90f95a8d1 (table rebalancer_dynamic_table_new_name [id=250c3fd291e940b5a1f68621cb9e98dc]) successfully deleted
I20240502 01:53:48.061229 20754 ts_tablet_manager.cc:1398] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240502 01:53:48.065690 17322 catalog_manager.cc:4830] TS 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): tablet 9f47b05574944b2e9d35b0d90f95a8d1 (table rebalancer_dynamic_table_new_name [id=250c3fd291e940b5a1f68621cb9e98dc]) successfully deleted
I20240502 01:53:48.098281 17331 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:53:48.107846 20478 tablet_service.cc:1467] Processing CreateTablet for tablet 438e8c379c7046109ea36496ce3dda64 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=32e9f4e95fdf465f948510432b71e6af]), partition=
I20240502 01:53:48.108273 20478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 438e8c379c7046109ea36496ce3dda64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:48.108299 20211 tablet_service.cc:1467] Processing CreateTablet for tablet 438e8c379c7046109ea36496ce3dda64 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=32e9f4e95fdf465f948510432b71e6af]), partition=
I20240502 01:53:48.108621 20211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 438e8c379c7046109ea36496ce3dda64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:48.113425 17845 tablet_service.cc:1467] Processing CreateTablet for tablet 438e8c379c7046109ea36496ce3dda64 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=32e9f4e95fdf465f948510432b71e6af]), partition=
I20240502 01:53:48.113786 17845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 438e8c379c7046109ea36496ce3dda64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:48.119546 20793 tablet_bootstrap.cc:492] T 438e8c379c7046109ea36496ce3dda64 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:53:48.121300 20793 tablet_bootstrap.cc:654] T 438e8c379c7046109ea36496ce3dda64 P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:48.126292 20749 tablet_bootstrap.cc:492] T 438e8c379c7046109ea36496ce3dda64 P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:53:48.126339 20793 tablet_bootstrap.cc:492] T 438e8c379c7046109ea36496ce3dda64 P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:53:48.126642 20793 ts_tablet_manager.cc:1367] T 438e8c379c7046109ea36496ce3dda64 P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240502 01:53:48.127539 20793 raft_consensus.cc:348] T 438e8c379c7046109ea36496ce3dda64 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:48.127816 20793 raft_consensus.cc:374] T 438e8c379c7046109ea36496ce3dda64 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:48.127918 20749 tablet_bootstrap.cc:654] T 438e8c379c7046109ea36496ce3dda64 P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:48.128010 20793 raft_consensus.cc:729] T 438e8c379c7046109ea36496ce3dda64 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:53:48.128330 20793 consensus_queue.cc:260] T 438e8c379c7046109ea36496ce3dda64 P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:48.138419 20749 tablet_bootstrap.cc:492] T 438e8c379c7046109ea36496ce3dda64 P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:53:48.138609 20749 ts_tablet_manager.cc:1367] T 438e8c379c7046109ea36496ce3dda64 P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20240502 01:53:48.139513 20749 raft_consensus.cc:348] T 438e8c379c7046109ea36496ce3dda64 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:48.139746 20749 raft_consensus.cc:374] T 438e8c379c7046109ea36496ce3dda64 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:48.139832 20749 raft_consensus.cc:729] T 438e8c379c7046109ea36496ce3dda64 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:53:48.140184 20749 consensus_queue.cc:260] T 438e8c379c7046109ea36496ce3dda64 P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:48.140969 20765 raft_consensus.cc:482] T ebb9bca9f1bd41b0857939909b1110e4 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:48.141286 20765 raft_consensus.cc:504] T ebb9bca9f1bd41b0857939909b1110e4 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:48.143384 20765 leader_election.cc:288] T ebb9bca9f1bd41b0857939909b1110e4 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:53:48.143885 20749 ts_tablet_manager.cc:1398] T 438e8c379c7046109ea36496ce3dda64 P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20240502 01:53:48.144598 20798 tablet_bootstrap.cc:492] T 438e8c379c7046109ea36496ce3dda64 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:48.146513 20798 tablet_bootstrap.cc:654] T 438e8c379c7046109ea36496ce3dda64 P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:48.149966 20797 raft_consensus.cc:482] T 438e8c379c7046109ea36496ce3dda64 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:48.150434 20793 ts_tablet_manager.cc:1398] T 438e8c379c7046109ea36496ce3dda64 P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.024s user 0.003s sys 0.005s
I20240502 01:53:48.150290 20797 raft_consensus.cc:504] T 438e8c379c7046109ea36496ce3dda64 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:48.151866 20798 tablet_bootstrap.cc:492] T 438e8c379c7046109ea36496ce3dda64 P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:48.152046 20798 ts_tablet_manager.cc:1367] T 438e8c379c7046109ea36496ce3dda64 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240502 01:53:48.152829 20798 raft_consensus.cc:348] T 438e8c379c7046109ea36496ce3dda64 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:48.153124 20798 raft_consensus.cc:374] T 438e8c379c7046109ea36496ce3dda64 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:48.153264 20798 raft_consensus.cc:729] T 438e8c379c7046109ea36496ce3dda64 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:48.153481 20798 consensus_queue.cc:260] T 438e8c379c7046109ea36496ce3dda64 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:48.154053 20798 ts_tablet_manager.cc:1398] T 438e8c379c7046109ea36496ce3dda64 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:48.156105 20797 leader_election.cc:288] T 438e8c379c7046109ea36496ce3dda64 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:53:48.161959 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebb9bca9f1bd41b0857939909b1110e4" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:53:48.162209 17736 raft_consensus.cc:2454] T ebb9bca9f1bd41b0857939909b1110e4 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 0.
I20240502 01:53:48.162815 20302 leader_election.cc:302] T ebb9bca9f1bd41b0857939909b1110e4 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:48.163360 20765 raft_consensus.cc:2781] T ebb9bca9f1bd41b0857939909b1110e4 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:48.163512 20765 raft_consensus.cc:482] T ebb9bca9f1bd41b0857939909b1110e4 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:48.163614 20765 raft_consensus.cc:3037] T ebb9bca9f1bd41b0857939909b1110e4 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:48.174727 20233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebb9bca9f1bd41b0857939909b1110e4" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:53:48.175141 20233 raft_consensus.cc:2454] T ebb9bca9f1bd41b0857939909b1110e4 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 0.
I20240502 01:53:48.175695 20233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "438e8c379c7046109ea36496ce3dda64" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:53:48.175936 20233 raft_consensus.cc:2454] T 438e8c379c7046109ea36496ce3dda64 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 0.
I20240502 01:53:48.175716 20768 raft_consensus.cc:482] T 975f253ff03447ff96fe84169c05097e P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:48.176254 20768 raft_consensus.cc:504] T 975f253ff03447ff96fe84169c05097e P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
W20240502 01:53:48.177639 17913 tablet.cc:2369] T 438e8c379c7046109ea36496ce3dda64 P bb31bceba76d466db24b9341c04900c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:53:48.178038 20768 leader_election.cc:288] T 975f253ff03447ff96fe84169c05097e P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:48.178702 20434 leader_election.cc:302] T 438e8c379c7046109ea36496ce3dda64 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:53:48.179168 20797 raft_consensus.cc:2781] T 438e8c379c7046109ea36496ce3dda64 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:48.179330 20797 raft_consensus.cc:482] T 438e8c379c7046109ea36496ce3dda64 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:48.179462 20797 raft_consensus.cc:3037] T 438e8c379c7046109ea36496ce3dda64 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:48.182893 20797 raft_consensus.cc:504] T 438e8c379c7046109ea36496ce3dda64 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:48.184051 20233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "438e8c379c7046109ea36496ce3dda64" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:53:48.184255 20233 raft_consensus.cc:3037] T 438e8c379c7046109ea36496ce3dda64 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:48.186851 20765 raft_consensus.cc:504] T ebb9bca9f1bd41b0857939909b1110e4 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:48.187186 20233 raft_consensus.cc:2454] T 438e8c379c7046109ea36496ce3dda64 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 1.
I20240502 01:53:48.187680 20434 leader_election.cc:302] T 438e8c379c7046109ea36496ce3dda64 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:53:48.188037 20797 leader_election.cc:288] T 438e8c379c7046109ea36496ce3dda64 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 election: Requested vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:53:48.188040 20795 raft_consensus.cc:2781] T 438e8c379c7046109ea36496ce3dda64 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:48.188949 20795 raft_consensus.cc:686] T 438e8c379c7046109ea36496ce3dda64 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Running, Role: LEADER
I20240502 01:53:48.189615 20795 consensus_queue.cc:237] T 438e8c379c7046109ea36496ce3dda64 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:48.190521 20233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebb9bca9f1bd41b0857939909b1110e4" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:53:48.190685 20233 raft_consensus.cc:3037] T ebb9bca9f1bd41b0857939909b1110e4 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:48.190575 20765 leader_election.cc:288] T ebb9bca9f1bd41b0857939909b1110e4 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Requested vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:53:48.190905 20765 raft_consensus.cc:482] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:48.191026 20765 raft_consensus.cc:504] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:48.192831 20232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31aa6a26f64f49368ad43cc97c66f2f1" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:53:48.193193 20232 raft_consensus.cc:2454] T 31aa6a26f64f49368ad43cc97c66f2f1 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 0.
I20240502 01:53:48.193872 20233 raft_consensus.cc:2454] T ebb9bca9f1bd41b0857939909b1110e4 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 1.
I20240502 01:53:48.194824 20303 leader_election.cc:302] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:53:48.195149 20303 leader_election.cc:302] T ebb9bca9f1bd41b0857939909b1110e4 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:53:48.195801 20813 raft_consensus.cc:2781] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:48.195961 20813 raft_consensus.cc:482] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:48.196027 20765 leader_election.cc:288] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:53:48.196187 20765 raft_consensus.cc:2781] T ebb9bca9f1bd41b0857939909b1110e4 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:48.196192 20813 raft_consensus.cc:3037] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:48.196331 20765 raft_consensus.cc:686] T ebb9bca9f1bd41b0857939909b1110e4 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Becoming Leader. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Running, Role: LEADER
I20240502 01:53:48.199702 20813 raft_consensus.cc:504] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:48.199836 20765 consensus_queue.cc:237] T ebb9bca9f1bd41b0857939909b1110e4 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:48.200683 20233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31aa6a26f64f49368ad43cc97c66f2f1" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:53:48.200903 20233 raft_consensus.cc:3037] T 31aa6a26f64f49368ad43cc97c66f2f1 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:48.203279 20233 raft_consensus.cc:2454] T 31aa6a26f64f49368ad43cc97c66f2f1 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 1.
I20240502 01:53:48.204246 20303 leader_election.cc:302] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:53:48.204813 20813 leader_election.cc:288] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Requested vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:53:48.205457 20813 raft_consensus.cc:2781] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:48.205796 20813 raft_consensus.cc:686] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Becoming Leader. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Running, Role: LEADER
I20240502 01:53:48.206305 20813 consensus_queue.cc:237] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:48.209100 17331 catalog_manager.cc:5484] T 438e8c379c7046109ea36496ce3dda64 P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71). New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:48.210976 17864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "438e8c379c7046109ea36496ce3dda64" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:53:48.211220 17864 raft_consensus.cc:2454] T 438e8c379c7046109ea36496ce3dda64 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 0.
I20240502 01:53:48.212569 17867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "438e8c379c7046109ea36496ce3dda64" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:53:48.212769 17867 raft_consensus.cc:3037] T 438e8c379c7046109ea36496ce3dda64 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:48.215530 17867 raft_consensus.cc:2454] T 438e8c379c7046109ea36496ce3dda64 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 1.
I20240502 01:53:48.232391 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebb9bca9f1bd41b0857939909b1110e4" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:53:48.232652 17736 raft_consensus.cc:3037] T ebb9bca9f1bd41b0857939909b1110e4 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:48.235244 20629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975f253ff03447ff96fe84169c05097e" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091" is_pre_election: true
I20240502 01:53:48.237424 17736 raft_consensus.cc:2454] T ebb9bca9f1bd41b0857939909b1110e4 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 1.
W20240502 01:53:48.243022 20413 tablet.cc:2369] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:53:48.251112 17331 catalog_manager.cc:5484] T ebb9bca9f1bd41b0857939909b1110e4 P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70). New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } }
I20240502 01:53:48.257403 17326 catalog_manager.cc:5484] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70). New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:48.258005 17474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975f253ff03447ff96fe84169c05097e" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:48.258270 17474 raft_consensus.cc:2379] T 975f253ff03447ff96fe84169c05097e P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdc2ac372492454c85387589a91f63af in current term 1: Already voted for candidate f19cc6cb1e76460aa6711e49a5775091 in this term.
I20240502 01:53:48.260066 20167 leader_election.cc:302] T 975f253ff03447ff96fe84169c05097e P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdc2ac372492454c85387589a91f63af; no voters: 4da690c0fa13407a85c0689ce8de9ade, f19cc6cb1e76460aa6711e49a5775091
I20240502 01:53:48.260627 20768 raft_consensus.cc:3037] T 975f253ff03447ff96fe84169c05097e P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:48.262691 20629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31aa6a26f64f49368ad43cc97c66f2f1" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091" is_pre_election: true
I20240502 01:53:48.262794 20630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31aa6a26f64f49368ad43cc97c66f2f1" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091"
I20240502 01:53:48.262950 20629 raft_consensus.cc:2454] T 31aa6a26f64f49368ad43cc97c66f2f1 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 0.
I20240502 01:53:48.263485 20768 raft_consensus.cc:2726] T 975f253ff03447ff96fe84169c05097e P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:53:48.272827 17326 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240502 01:53:48.273092 17326 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240502 01:53:48.276623 17326 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet 5ad48f64a0f54050a5b6112ebb551228
I20240502 01:53:48.277355 17454 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ad48f64a0f54050a5b6112ebb551228 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:48 UTC) from {username='slave'} at 127.0.0.1:54520
I20240502 01:53:48.277604 20741 tablet_replica.cc:330] T 5ad48f64a0f54050a5b6112ebb551228 P 4da690c0fa13407a85c0689ce8de9ade: stopping tablet replica
I20240502 01:53:48.277849 20741 raft_consensus.cc:2229] T 5ad48f64a0f54050a5b6112ebb551228 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:48.278070 20741 raft_consensus.cc:2258] T 5ad48f64a0f54050a5b6112ebb551228 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:48.278662 20741 ts_tablet_manager.cc:1875] T 5ad48f64a0f54050a5b6112ebb551228 P 4da690c0fa13407a85c0689ce8de9ade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:48.283362 20741 ts_tablet_manager.cc:1888] T 5ad48f64a0f54050a5b6112ebb551228 P 4da690c0fa13407a85c0689ce8de9ade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:48.283488 20741 log.cc:1196] T 5ad48f64a0f54050a5b6112ebb551228 P 4da690c0fa13407a85c0689ce8de9ade: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/5ad48f64a0f54050a5b6112ebb551228
I20240502 01:53:48.283804 20741 ts_tablet_manager.cc:1909] T 5ad48f64a0f54050a5b6112ebb551228 P 4da690c0fa13407a85c0689ce8de9ade: Deleting consensus metadata
I20240502 01:53:48.284709 17321 catalog_manager.cc:4830] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): tablet 5ad48f64a0f54050a5b6112ebb551228 (table rebalance_test_table_2 [id=5af2852a0ad343e9aa110a94b3402a8a]) successfully deleted
I20240502 01:53:48.284790 17326 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet 4cbb5336819548ebb1239ee9016910ec
I20240502 01:53:48.285254 17845 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ad48f64a0f54050a5b6112ebb551228 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:48 UTC) from {username='slave'} at 127.0.0.1:47226
I20240502 01:53:48.285279 17326 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet 4a4cc6012ea54b5791d0df2ffe36f7eb
I20240502 01:53:48.285611 20738 tablet_replica.cc:330] T 5ad48f64a0f54050a5b6112ebb551228 P bb31bceba76d466db24b9341c04900c3: stopping tablet replica
I20240502 01:53:48.285607 17716 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cbb5336819548ebb1239ee9016910ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:48 UTC) from {username='slave'} at 127.0.0.1:52198
I20240502 01:53:48.285830 20738 raft_consensus.cc:2229] T 5ad48f64a0f54050a5b6112ebb551228 P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:48.285876 20728 tablet_replica.cc:330] T 4cbb5336819548ebb1239ee9016910ec P dfa1152987644bd781b46c7e42a7a685: stopping tablet replica
I20240502 01:53:48.286026 17585 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ad48f64a0f54050a5b6112ebb551228 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:48 UTC) from {username='slave'} at 127.0.0.1:55232
I20240502 01:53:48.286089 20728 raft_consensus.cc:2229] T 4cbb5336819548ebb1239ee9016910ec P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:48.286162 20738 raft_consensus.cc:2258] T 5ad48f64a0f54050a5b6112ebb551228 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:48.286296 20737 tablet_replica.cc:330] T 5ad48f64a0f54050a5b6112ebb551228 P 3fceccefba274b1dbbd8216af8b0bc29: stopping tablet replica
I20240502 01:53:48.286351 20728 raft_consensus.cc:2258] T 4cbb5336819548ebb1239ee9016910ec P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:48.286525 20737 raft_consensus.cc:2229] T 5ad48f64a0f54050a5b6112ebb551228 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:48.286744 20737 raft_consensus.cc:2258] T 5ad48f64a0f54050a5b6112ebb551228 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:48.286868 20738 ts_tablet_manager.cc:1875] T 5ad48f64a0f54050a5b6112ebb551228 P bb31bceba76d466db24b9341c04900c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:48.286998 20728 ts_tablet_manager.cc:1875] T 4cbb5336819548ebb1239ee9016910ec P dfa1152987644bd781b46c7e42a7a685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:48.287357 20737 ts_tablet_manager.cc:1875] T 5ad48f64a0f54050a5b6112ebb551228 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:48.289144 17846 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a4cc6012ea54b5791d0df2ffe36f7eb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:48 UTC) from {username='slave'} at 127.0.0.1:47226
I20240502 01:53:48.289327 17585 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a4cc6012ea54b5791d0df2ffe36f7eb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:48 UTC) from {username='slave'} at 127.0.0.1:55232
I20240502 01:53:48.289462 17845 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cbb5336819548ebb1239ee9016910ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:48 UTC) from {username='slave'} at 127.0.0.1:47226
I20240502 01:53:48.289594 17456 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a4cc6012ea54b5791d0df2ffe36f7eb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:48 UTC) from {username='slave'} at 127.0.0.1:54520
I20240502 01:53:48.289810 17454 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cbb5336819548ebb1239ee9016910ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:48 UTC) from {username='slave'} at 127.0.0.1:54520
I20240502 01:53:48.290135 20741 tablet_replica.cc:330] T 4a4cc6012ea54b5791d0df2ffe36f7eb P 4da690c0fa13407a85c0689ce8de9ade: stopping tablet replica
I20240502 01:53:48.290387 20741 raft_consensus.cc:2229] T 4a4cc6012ea54b5791d0df2ffe36f7eb P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:48.290673 20741 raft_consensus.cc:2258] T 4a4cc6012ea54b5791d0df2ffe36f7eb P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:48.291393 20741 ts_tablet_manager.cc:1875] T 4a4cc6012ea54b5791d0df2ffe36f7eb P 4da690c0fa13407a85c0689ce8de9ade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:48.293267 20737 ts_tablet_manager.cc:1888] T 5ad48f64a0f54050a5b6112ebb551228 P 3fceccefba274b1dbbd8216af8b0bc29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:48.293457 20737 log.cc:1196] T 5ad48f64a0f54050a5b6112ebb551228 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/wals/5ad48f64a0f54050a5b6112ebb551228
I20240502 01:53:48.293769 20728 ts_tablet_manager.cc:1888] T 4cbb5336819548ebb1239ee9016910ec P dfa1152987644bd781b46c7e42a7a685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:48.293923 20728 log.cc:1196] T 4cbb5336819548ebb1239ee9016910ec P dfa1152987644bd781b46c7e42a7a685: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/wals/4cbb5336819548ebb1239ee9016910ec
I20240502 01:53:48.293946 20737 ts_tablet_manager.cc:1909] T 5ad48f64a0f54050a5b6112ebb551228 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting consensus metadata
I20240502 01:53:48.294342 20728 ts_tablet_manager.cc:1909] T 4cbb5336819548ebb1239ee9016910ec P dfa1152987644bd781b46c7e42a7a685: Deleting consensus metadata
I20240502 01:53:48.295048 20738 ts_tablet_manager.cc:1888] T 5ad48f64a0f54050a5b6112ebb551228 P bb31bceba76d466db24b9341c04900c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:48.295099 17319 catalog_manager.cc:4830] TS 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): tablet 5ad48f64a0f54050a5b6112ebb551228 (table rebalance_test_table_2 [id=5af2852a0ad343e9aa110a94b3402a8a]) successfully deleted
I20240502 01:53:48.295169 20738 log.cc:1196] T 5ad48f64a0f54050a5b6112ebb551228 P bb31bceba76d466db24b9341c04900c3: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/wals/5ad48f64a0f54050a5b6112ebb551228
I20240502 01:53:48.295336 17321 catalog_manager.cc:4830] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): tablet 4cbb5336819548ebb1239ee9016910ec (table rebalance_test_table_2 [id=5af2852a0ad343e9aa110a94b3402a8a]) successfully deleted
I20240502 01:53:48.295511 20737 tablet_replica.cc:330] T 4a4cc6012ea54b5791d0df2ffe36f7eb P 3fceccefba274b1dbbd8216af8b0bc29: stopping tablet replica
I20240502 01:53:48.295641 20738 ts_tablet_manager.cc:1909] T 5ad48f64a0f54050a5b6112ebb551228 P bb31bceba76d466db24b9341c04900c3: Deleting consensus metadata
I20240502 01:53:48.295769 20737 raft_consensus.cc:2229] T 4a4cc6012ea54b5791d0df2ffe36f7eb P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:48.296061 20737 raft_consensus.cc:2258] T 4a4cc6012ea54b5791d0df2ffe36f7eb P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:48.296587 17322 catalog_manager.cc:4830] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): tablet 5ad48f64a0f54050a5b6112ebb551228 (table rebalance_test_table_2 [id=5af2852a0ad343e9aa110a94b3402a8a]) successfully deleted
I20240502 01:53:48.296784 20737 ts_tablet_manager.cc:1875] T 4a4cc6012ea54b5791d0df2ffe36f7eb P 3fceccefba274b1dbbd8216af8b0bc29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:48.296864 20738 tablet_replica.cc:330] T 4a4cc6012ea54b5791d0df2ffe36f7eb P bb31bceba76d466db24b9341c04900c3: stopping tablet replica
I20240502 01:53:48.297114 20738 raft_consensus.cc:2229] T 4a4cc6012ea54b5791d0df2ffe36f7eb P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:48.297410 20738 raft_consensus.cc:2258] T 4a4cc6012ea54b5791d0df2ffe36f7eb P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:48.297478 20741 ts_tablet_manager.cc:1888] T 4a4cc6012ea54b5791d0df2ffe36f7eb P 4da690c0fa13407a85c0689ce8de9ade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:48.297605 20741 log.cc:1196] T 4a4cc6012ea54b5791d0df2ffe36f7eb P 4da690c0fa13407a85c0689ce8de9ade: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/4a4cc6012ea54b5791d0df2ffe36f7eb
I20240502 01:53:48.298015 20741 ts_tablet_manager.cc:1909] T 4a4cc6012ea54b5791d0df2ffe36f7eb P 4da690c0fa13407a85c0689ce8de9ade: Deleting consensus metadata
I20240502 01:53:48.298069 20738 ts_tablet_manager.cc:1875] T 4a4cc6012ea54b5791d0df2ffe36f7eb P bb31bceba76d466db24b9341c04900c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:48.299098 17321 catalog_manager.cc:4830] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): tablet 4a4cc6012ea54b5791d0df2ffe36f7eb (table rebalance_test_table_2 [id=5af2852a0ad343e9aa110a94b3402a8a]) successfully deleted
I20240502 01:53:48.299439 20741 tablet_replica.cc:330] T 4cbb5336819548ebb1239ee9016910ec P 4da690c0fa13407a85c0689ce8de9ade: stopping tablet replica
I20240502 01:53:48.299974 20741 raft_consensus.cc:2229] T 4cbb5336819548ebb1239ee9016910ec P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:48.300295 20741 raft_consensus.cc:2258] T 4cbb5336819548ebb1239ee9016910ec P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:48.300964 20741 ts_tablet_manager.cc:1875] T 4cbb5336819548ebb1239ee9016910ec P 4da690c0fa13407a85c0689ce8de9ade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:48.303694 20737 ts_tablet_manager.cc:1888] T 4a4cc6012ea54b5791d0df2ffe36f7eb P 3fceccefba274b1dbbd8216af8b0bc29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:48.303894 20737 log.cc:1196] T 4a4cc6012ea54b5791d0df2ffe36f7eb P 3fceccefba274b1dbbd8216af8b0bc29: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/wals/4a4cc6012ea54b5791d0df2ffe36f7eb
I20240502 01:53:48.304340 20737 ts_tablet_manager.cc:1909] T 4a4cc6012ea54b5791d0df2ffe36f7eb P 3fceccefba274b1dbbd8216af8b0bc29: Deleting consensus metadata
I20240502 01:53:48.305259 17319 catalog_manager.cc:4830] TS 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): tablet 4a4cc6012ea54b5791d0df2ffe36f7eb (table rebalance_test_table_2 [id=5af2852a0ad343e9aa110a94b3402a8a]) successfully deleted
I20240502 01:53:48.305708 20738 ts_tablet_manager.cc:1888] T 4a4cc6012ea54b5791d0df2ffe36f7eb P bb31bceba76d466db24b9341c04900c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:48.305889 20738 log.cc:1196] T 4a4cc6012ea54b5791d0df2ffe36f7eb P bb31bceba76d466db24b9341c04900c3: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/wals/4a4cc6012ea54b5791d0df2ffe36f7eb
I20240502 01:53:48.306340 20738 ts_tablet_manager.cc:1909] T 4a4cc6012ea54b5791d0df2ffe36f7eb P bb31bceba76d466db24b9341c04900c3: Deleting consensus metadata
I20240502 01:53:48.306499 20741 ts_tablet_manager.cc:1888] T 4cbb5336819548ebb1239ee9016910ec P 4da690c0fa13407a85c0689ce8de9ade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:48.306622 20741 log.cc:1196] T 4cbb5336819548ebb1239ee9016910ec P 4da690c0fa13407a85c0689ce8de9ade: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/4cbb5336819548ebb1239ee9016910ec
I20240502 01:53:48.306912 20741 ts_tablet_manager.cc:1909] T 4cbb5336819548ebb1239ee9016910ec P 4da690c0fa13407a85c0689ce8de9ade: Deleting consensus metadata
I20240502 01:53:48.307291 17322 catalog_manager.cc:4830] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): tablet 4a4cc6012ea54b5791d0df2ffe36f7eb (table rebalance_test_table_2 [id=5af2852a0ad343e9aa110a94b3402a8a]) successfully deleted
I20240502 01:53:48.307653 20738 tablet_replica.cc:330] T 4cbb5336819548ebb1239ee9016910ec P bb31bceba76d466db24b9341c04900c3: stopping tablet replica
I20240502 01:53:48.307687 17321 catalog_manager.cc:4830] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): tablet 4cbb5336819548ebb1239ee9016910ec (table rebalance_test_table_2 [id=5af2852a0ad343e9aa110a94b3402a8a]) successfully deleted
I20240502 01:53:48.307844 20738 raft_consensus.cc:2229] T 4cbb5336819548ebb1239ee9016910ec P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:48.308054 20738 raft_consensus.cc:2258] T 4cbb5336819548ebb1239ee9016910ec P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:48.308696 20738 ts_tablet_manager.cc:1875] T 4cbb5336819548ebb1239ee9016910ec P bb31bceba76d466db24b9341c04900c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:48.314692 20738 ts_tablet_manager.cc:1888] T 4cbb5336819548ebb1239ee9016910ec P bb31bceba76d466db24b9341c04900c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:48.314801 20738 log.cc:1196] T 4cbb5336819548ebb1239ee9016910ec P bb31bceba76d466db24b9341c04900c3: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/wals/4cbb5336819548ebb1239ee9016910ec
I20240502 01:53:48.315189 20738 ts_tablet_manager.cc:1909] T 4cbb5336819548ebb1239ee9016910ec P bb31bceba76d466db24b9341c04900c3: Deleting consensus metadata
I20240502 01:53:48.316066 17322 catalog_manager.cc:4830] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): tablet 4cbb5336819548ebb1239ee9016910ec (table rebalance_test_table_2 [id=5af2852a0ad343e9aa110a94b3402a8a]) successfully deleted
I20240502 01:53:48.354683 17330 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:53:48.361544 17867 raft_consensus.cc:1261] T 438e8c379c7046109ea36496ce3dda64 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:48.361960 20233 raft_consensus.cc:1261] T 438e8c379c7046109ea36496ce3dda64 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:48.362112 20795 consensus_queue.cc:1035] T 438e8c379c7046109ea36496ce3dda64 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:48.363097 20795 consensus_queue.cc:1035] T 438e8c379c7046109ea36496ce3dda64 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:48.403012 17333 catalog_manager.cc:5576] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083) does not have the latest schema for tablet 438e8c379c7046109ea36496ce3dda64 (table rebalancer_dynamic_table [id=32e9f4e95fdf465f948510432b71e6af]). Expected version 1 got 0
I20240502 01:53:48.407449 20843 tablet.cc:1722] T 438e8c379c7046109ea36496ce3dda64 P 2f306549407b4bb281ccf2994035e4f1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:53:48.407964 20843 tablet.cc:1620] T 438e8c379c7046109ea36496ce3dda64 P 2f306549407b4bb281ccf2994035e4f1: MemRowSet was empty: no flush needed.
I20240502 01:53:48.408813 17333 catalog_manager.cc:5576] TS 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623) does not have the latest schema for tablet 438e8c379c7046109ea36496ce3dda64 (table rebalancer_dynamic_table [id=32e9f4e95fdf465f948510432b71e6af]). Expected version 1 got 0
I20240502 01:53:48.415540 17330 catalog_manager.cc:5576] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339) does not have the latest schema for tablet 438e8c379c7046109ea36496ce3dda64 (table rebalancer_dynamic_table [id=32e9f4e95fdf465f948510432b71e6af]). Expected version 1 got 0
I20240502 01:53:48.421703 17330 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=32e9f4e95fdf465f948510432b71e6af] alter complete (version 1)
I20240502 01:53:48.421697 20844 tablet.cc:1722] T 438e8c379c7046109ea36496ce3dda64 P bb31bceba76d466db24b9341c04900c3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:53:48.422664 20844 tablet.cc:1620] T 438e8c379c7046109ea36496ce3dda64 P bb31bceba76d466db24b9341c04900c3: MemRowSet was empty: no flush needed.
I20240502 01:53:48.426532 20845 tablet.cc:1722] T 438e8c379c7046109ea36496ce3dda64 P fdc2ac372492454c85387589a91f63af: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:53:48.427835 20845 tablet.cc:1620] T 438e8c379c7046109ea36496ce3dda64 P fdc2ac372492454c85387589a91f63af: MemRowSet was empty: no flush needed.
I20240502 01:53:48.428563 17327 catalog_manager.cc:5576] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339) does not have the latest schema for tablet 438e8c379c7046109ea36496ce3dda64 (table rebalancer_dynamic_table [id=32e9f4e95fdf465f948510432b71e6af]). Expected version 1 got 0
I20240502 01:53:48.431140 17333 catalog_manager.cc:5576] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083) does not have the latest schema for tablet 438e8c379c7046109ea36496ce3dda64 (table rebalancer_dynamic_table [id=32e9f4e95fdf465f948510432b71e6af]). Expected version 1 got 0
I20240502 01:53:48.470544 20698 consensus_queue.cc:1035] T 975f253ff03447ff96fe84169c05097e P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:48.485462 20762 consensus_queue.cc:1035] T 975f253ff03447ff96fe84169c05097e P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:48.505555 20478 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:48.521875 17454 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:53:48.526242 17585 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:53:48.526908 17327 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:53:48.540012 20843 tablet.cc:1722] T 438e8c379c7046109ea36496ce3dda64 P 2f306549407b4bb281ccf2994035e4f1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:53:48.540534 20843 tablet.cc:1620] T 438e8c379c7046109ea36496ce3dda64 P 2f306549407b4bb281ccf2994035e4f1: MemRowSet was empty: no flush needed.
I20240502 01:53:48.543561 20845 tablet.cc:1722] T 438e8c379c7046109ea36496ce3dda64 P fdc2ac372492454c85387589a91f63af: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:53:48.543985 20845 tablet.cc:1620] T 438e8c379c7046109ea36496ce3dda64 P fdc2ac372492454c85387589a91f63af: MemRowSet was empty: no flush needed.
I20240502 01:53:48.549523 20844 tablet.cc:1722] T 438e8c379c7046109ea36496ce3dda64 P bb31bceba76d466db24b9341c04900c3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:53:48.550000 20844 tablet.cc:1620] T 438e8c379c7046109ea36496ce3dda64 P bb31bceba76d466db24b9341c04900c3: MemRowSet was empty: no flush needed.
I20240502 01:53:48.556905 20346 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:48.559216 17327 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=32e9f4e95fdf465f948510432b71e6af] alter complete (version 2)
I20240502 01:53:48.579429 17845 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:53:48.592310 17716 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:53:48.624547 20609 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:48.640398 20211 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:48.663282 17328 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:53:48.674131 20843 tablet.cc:1722] T 438e8c379c7046109ea36496ce3dda64 P 2f306549407b4bb281ccf2994035e4f1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:53:48.675673 20844 tablet.cc:1722] T 438e8c379c7046109ea36496ce3dda64 P bb31bceba76d466db24b9341c04900c3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:53:48.680948 20845 tablet.cc:1722] T 438e8c379c7046109ea36496ce3dda64 P fdc2ac372492454c85387589a91f63af: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:53:48.689287 17327 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table_new_name [id=32e9f4e95fdf465f948510432b71e6af] alter complete (version 3)
I20240502 01:53:48.707537 20765 consensus_queue.cc:1035] T ebb9bca9f1bd41b0857939909b1110e4 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:48.721858 20629 raft_consensus.cc:3037] T 31aa6a26f64f49368ad43cc97c66f2f1 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:48.734028 20813 consensus_queue.cc:1035] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:48.737421 20813 consensus_queue.cc:1035] T ebb9bca9f1bd41b0857939909b1110e4 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:48.754072 20813 consensus_queue.cc:1035] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:48.789997 17327 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240502 01:53:48.790242 17327 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240502 01:53:48.795300 17327 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet 2f3528ac3c114a19892bba2304474302
I20240502 01:53:48.795424 17335 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:53:48.795616 17335 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:53:48.795994 17327 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet ae24680ad75545d892c9b79833319214
I20240502 01:53:48.796478 17327 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet 2f29bba45e1749eb96b08da5e15fb834
I20240502 01:53:48.797988 17454 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f3528ac3c114a19892bba2304474302 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:48 UTC) from {username='slave'} at 127.0.0.1:54520
I20240502 01:53:48.798359 17456 tablet_service.cc:1514] Processing DeleteTablet for tablet ae24680ad75545d892c9b79833319214 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:48 UTC) from {username='slave'} at 127.0.0.1:54520
I20240502 01:53:48.798576 17716 tablet_service.cc:1514] Processing DeleteTablet for tablet ae24680ad75545d892c9b79833319214 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:48 UTC) from {username='slave'} at 127.0.0.1:52198
I20240502 01:53:48.798636 17455 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f29bba45e1749eb96b08da5e15fb834 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:48 UTC) from {username='slave'} at 127.0.0.1:54520
I20240502 01:53:48.798866 20741 tablet_replica.cc:330] T 2f3528ac3c114a19892bba2304474302 P 4da690c0fa13407a85c0689ce8de9ade: stopping tablet replica
I20240502 01:53:48.798956 17715 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f29bba45e1749eb96b08da5e15fb834 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:48 UTC) from {username='slave'} at 127.0.0.1:52198
I20240502 01:53:48.799086 20741 raft_consensus.cc:2229] T 2f3528ac3c114a19892bba2304474302 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:48.799491 20741 raft_consensus.cc:2258] T 2f3528ac3c114a19892bba2304474302 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:48.799760 17845 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f3528ac3c114a19892bba2304474302 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:48 UTC) from {username='slave'} at 127.0.0.1:47226
I20240502 01:53:48.800041 20738 tablet_replica.cc:330] T 2f3528ac3c114a19892bba2304474302 P bb31bceba76d466db24b9341c04900c3: stopping tablet replica
I20240502 01:53:48.800063 17846 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f29bba45e1749eb96b08da5e15fb834 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:48 UTC) from {username='slave'} at 127.0.0.1:47226
I20240502 01:53:48.800287 20741 ts_tablet_manager.cc:1875] T 2f3528ac3c114a19892bba2304474302 P 4da690c0fa13407a85c0689ce8de9ade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:48.800294 20738 raft_consensus.cc:2229] T 2f3528ac3c114a19892bba2304474302 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:48.800441 17585 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f3528ac3c114a19892bba2304474302 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:48 UTC) from {username='slave'} at 127.0.0.1:55232
I20240502 01:53:48.800494 20738 raft_consensus.cc:2258] T 2f3528ac3c114a19892bba2304474302 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:48.800670 17587 tablet_service.cc:1514] Processing DeleteTablet for tablet ae24680ad75545d892c9b79833319214 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:48 UTC) from {username='slave'} at 127.0.0.1:55232
I20240502 01:53:48.800881 20737 tablet_replica.cc:330] T 2f3528ac3c114a19892bba2304474302 P 3fceccefba274b1dbbd8216af8b0bc29: stopping tablet replica
I20240502 01:53:48.801116 20737 raft_consensus.cc:2229] T 2f3528ac3c114a19892bba2304474302 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:48.801234 20738 ts_tablet_manager.cc:1875] T 2f3528ac3c114a19892bba2304474302 P bb31bceba76d466db24b9341c04900c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:48.801363 20737 raft_consensus.cc:2258] T 2f3528ac3c114a19892bba2304474302 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:48.802047 20737 ts_tablet_manager.cc:1875] T 2f3528ac3c114a19892bba2304474302 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:48.802961 17335 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet 438e8c379c7046109ea36496ce3dda64
I20240502 01:53:48.806964 20478 tablet_service.cc:1514] Processing DeleteTablet for tablet 438e8c379c7046109ea36496ce3dda64 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:48 UTC) from {username='slave'} at 127.0.0.1:35018
I20240502 01:53:48.807525 20211 tablet_service.cc:1514] Processing DeleteTablet for tablet 438e8c379c7046109ea36496ce3dda64 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:48 UTC) from {username='slave'} at 127.0.0.1:44624
I20240502 01:53:48.807718 20738 ts_tablet_manager.cc:1888] T 2f3528ac3c114a19892bba2304474302 P bb31bceba76d466db24b9341c04900c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:48.807871 20738 log.cc:1196] T 2f3528ac3c114a19892bba2304474302 P bb31bceba76d466db24b9341c04900c3: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/wals/2f3528ac3c114a19892bba2304474302
I20240502 01:53:48.808312 20738 ts_tablet_manager.cc:1909] T 2f3528ac3c114a19892bba2304474302 P bb31bceba76d466db24b9341c04900c3: Deleting consensus metadata
I20240502 01:53:48.808606 17846 tablet_service.cc:1514] Processing DeleteTablet for tablet 438e8c379c7046109ea36496ce3dda64 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:48 UTC) from {username='slave'} at 127.0.0.1:47226
I20240502 01:53:48.808475 20894 tablet_replica.cc:330] T 438e8c379c7046109ea36496ce3dda64 P fdc2ac372492454c85387589a91f63af: stopping tablet replica
I20240502 01:53:48.809358 20894 raft_consensus.cc:2229] T 438e8c379c7046109ea36496ce3dda64 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:48.809495 20741 ts_tablet_manager.cc:1888] T 2f3528ac3c114a19892bba2304474302 P 4da690c0fa13407a85c0689ce8de9ade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:48.809682 20741 log.cc:1196] T 2f3528ac3c114a19892bba2304474302 P 4da690c0fa13407a85c0689ce8de9ade: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/2f3528ac3c114a19892bba2304474302
I20240502 01:53:48.809749 20894 raft_consensus.cc:2258] T 438e8c379c7046109ea36496ce3dda64 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:48.810256 20741 ts_tablet_manager.cc:1909] T 2f3528ac3c114a19892bba2304474302 P 4da690c0fa13407a85c0689ce8de9ade: Deleting consensus metadata
I20240502 01:53:48.811105 20894 ts_tablet_manager.cc:1875] T 438e8c379c7046109ea36496ce3dda64 P fdc2ac372492454c85387589a91f63af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:48.811376 20741 tablet_replica.cc:330] T ae24680ad75545d892c9b79833319214 P 4da690c0fa13407a85c0689ce8de9ade: stopping tablet replica
I20240502 01:53:48.811506 17321 catalog_manager.cc:4830] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): tablet 2f3528ac3c114a19892bba2304474302 (table rebalance_test_table_4 [id=0a0284719f394d1dab3221ab9abcf498]) successfully deleted
I20240502 01:53:48.811760 20741 raft_consensus.cc:2229] T ae24680ad75545d892c9b79833319214 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:48.812108 20741 raft_consensus.cc:2258] T ae24680ad75545d892c9b79833319214 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:48.812695 20738 tablet_replica.cc:330] T 2f29bba45e1749eb96b08da5e15fb834 P bb31bceba76d466db24b9341c04900c3: stopping tablet replica
I20240502 01:53:48.812870 20741 ts_tablet_manager.cc:1875] T ae24680ad75545d892c9b79833319214 P 4da690c0fa13407a85c0689ce8de9ade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:48.812880 17322 catalog_manager.cc:4830] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): tablet 2f3528ac3c114a19892bba2304474302 (table rebalance_test_table_4 [id=0a0284719f394d1dab3221ab9abcf498]) successfully deleted
I20240502 01:53:48.813056 20738 raft_consensus.cc:2229] T 2f29bba45e1749eb96b08da5e15fb834 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:48.813364 20738 raft_consensus.cc:2258] T 2f29bba45e1749eb96b08da5e15fb834 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:48.814100 20738 ts_tablet_manager.cc:1875] T 2f29bba45e1749eb96b08da5e15fb834 P bb31bceba76d466db24b9341c04900c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:48.816567 20892 tablet_replica.cc:330] T ae24680ad75545d892c9b79833319214 P dfa1152987644bd781b46c7e42a7a685: stopping tablet replica
I20240502 01:53:48.818522 20894 ts_tablet_manager.cc:1888] T 438e8c379c7046109ea36496ce3dda64 P fdc2ac372492454c85387589a91f63af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:53:48.818672 20894 log.cc:1196] T 438e8c379c7046109ea36496ce3dda64 P fdc2ac372492454c85387589a91f63af: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal/wals/438e8c379c7046109ea36496ce3dda64
I20240502 01:53:48.819267 20894 ts_tablet_manager.cc:1909] T 438e8c379c7046109ea36496ce3dda64 P fdc2ac372492454c85387589a91f63af: Deleting consensus metadata
I20240502 01:53:48.818835 20737 ts_tablet_manager.cc:1888] T 2f3528ac3c114a19892bba2304474302 P 3fceccefba274b1dbbd8216af8b0bc29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:48.820183 20737 log.cc:1196] T 2f3528ac3c114a19892bba2304474302 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/wals/2f3528ac3c114a19892bba2304474302
I20240502 01:53:48.820205 20738 ts_tablet_manager.cc:1888] T 2f29bba45e1749eb96b08da5e15fb834 P bb31bceba76d466db24b9341c04900c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:48.820322 20738 log.cc:1196] T 2f29bba45e1749eb96b08da5e15fb834 P bb31bceba76d466db24b9341c04900c3: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/wals/2f29bba45e1749eb96b08da5e15fb834
I20240502 01:53:48.820807 20738 ts_tablet_manager.cc:1909] T 2f29bba45e1749eb96b08da5e15fb834 P bb31bceba76d466db24b9341c04900c3: Deleting consensus metadata
I20240502 01:53:48.821151 20737 ts_tablet_manager.cc:1909] T 2f3528ac3c114a19892bba2304474302 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting consensus metadata
I20240502 01:53:48.821602 17322 catalog_manager.cc:4830] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): tablet 438e8c379c7046109ea36496ce3dda64 (table rebalancer_dynamic_table_new_name [id=32e9f4e95fdf465f948510432b71e6af]) successfully deleted
I20240502 01:53:48.821699 20738 tablet_replica.cc:330] T 438e8c379c7046109ea36496ce3dda64 P bb31bceba76d466db24b9341c04900c3: stopping tablet replica
I20240502 01:53:48.821885 17322 catalog_manager.cc:4830] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): tablet 2f29bba45e1749eb96b08da5e15fb834 (table rebalance_test_table_4 [id=0a0284719f394d1dab3221ab9abcf498]) successfully deleted
I20240502 01:53:48.821905 20738 raft_consensus.cc:2229] T 438e8c379c7046109ea36496ce3dda64 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:48.822091 20738 raft_consensus.cc:2258] T 438e8c379c7046109ea36496ce3dda64 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:48.822875 17319 catalog_manager.cc:4830] TS 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): tablet 2f3528ac3c114a19892bba2304474302 (table rebalance_test_table_4 [id=0a0284719f394d1dab3221ab9abcf498]) successfully deleted
I20240502 01:53:48.823299 20738 ts_tablet_manager.cc:1875] T 438e8c379c7046109ea36496ce3dda64 P bb31bceba76d466db24b9341c04900c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:48.823652 20737 tablet_replica.cc:330] T ae24680ad75545d892c9b79833319214 P 3fceccefba274b1dbbd8216af8b0bc29: stopping tablet replica
I20240502 01:53:48.823846 20737 raft_consensus.cc:2229] T ae24680ad75545d892c9b79833319214 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:48.824782 20737 raft_consensus.cc:2258] T ae24680ad75545d892c9b79833319214 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:48.816882 20892 raft_consensus.cc:2229] T ae24680ad75545d892c9b79833319214 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:48.825404 20892 raft_consensus.cc:2258] T ae24680ad75545d892c9b79833319214 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:48.826476 20737 ts_tablet_manager.cc:1875] T ae24680ad75545d892c9b79833319214 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:48.827952 20892 ts_tablet_manager.cc:1875] T ae24680ad75545d892c9b79833319214 P dfa1152987644bd781b46c7e42a7a685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:48.828907 20738 ts_tablet_manager.cc:1888] T 438e8c379c7046109ea36496ce3dda64 P bb31bceba76d466db24b9341c04900c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:53:48.829095 20738 log.cc:1196] T 438e8c379c7046109ea36496ce3dda64 P bb31bceba76d466db24b9341c04900c3: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/wals/438e8c379c7046109ea36496ce3dda64
I20240502 01:53:48.829555 20738 ts_tablet_manager.cc:1909] T 438e8c379c7046109ea36496ce3dda64 P bb31bceba76d466db24b9341c04900c3: Deleting consensus metadata
I20240502 01:53:48.830472 17322 catalog_manager.cc:4830] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): tablet 438e8c379c7046109ea36496ce3dda64 (table rebalancer_dynamic_table_new_name [id=32e9f4e95fdf465f948510432b71e6af]) successfully deleted
I20240502 01:53:48.831748 20893 tablet_replica.cc:330] T 438e8c379c7046109ea36496ce3dda64 P 2f306549407b4bb281ccf2994035e4f1: stopping tablet replica
I20240502 01:53:48.832002 20893 raft_consensus.cc:2229] T 438e8c379c7046109ea36496ce3dda64 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:48.832561 20741 ts_tablet_manager.cc:1888] T ae24680ad75545d892c9b79833319214 P 4da690c0fa13407a85c0689ce8de9ade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:48.832720 20741 log.cc:1196] T ae24680ad75545d892c9b79833319214 P 4da690c0fa13407a85c0689ce8de9ade: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/ae24680ad75545d892c9b79833319214
I20240502 01:53:48.832742 20893 raft_consensus.cc:2258] T 438e8c379c7046109ea36496ce3dda64 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:48.833729 20893 ts_tablet_manager.cc:1875] T 438e8c379c7046109ea36496ce3dda64 P 2f306549407b4bb281ccf2994035e4f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:48.837437 20741 ts_tablet_manager.cc:1909] T ae24680ad75545d892c9b79833319214 P 4da690c0fa13407a85c0689ce8de9ade: Deleting consensus metadata
I20240502 01:53:48.838501 17321 catalog_manager.cc:4830] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): tablet ae24680ad75545d892c9b79833319214 (table rebalance_test_table_4 [id=0a0284719f394d1dab3221ab9abcf498]) successfully deleted
I20240502 01:53:48.838822 20741 tablet_replica.cc:330] T 2f29bba45e1749eb96b08da5e15fb834 P 4da690c0fa13407a85c0689ce8de9ade: stopping tablet replica
I20240502 01:53:48.839056 20741 raft_consensus.cc:2229] T 2f29bba45e1749eb96b08da5e15fb834 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:48.839455 20741 raft_consensus.cc:2258] T 2f29bba45e1749eb96b08da5e15fb834 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:48.839551 20893 ts_tablet_manager.cc:1888] T 438e8c379c7046109ea36496ce3dda64 P 2f306549407b4bb281ccf2994035e4f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:53:48.839720 20893 log.cc:1196] T 438e8c379c7046109ea36496ce3dda64 P 2f306549407b4bb281ccf2994035e4f1: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal/wals/438e8c379c7046109ea36496ce3dda64
I20240502 01:53:48.840098 20893 ts_tablet_manager.cc:1909] T 438e8c379c7046109ea36496ce3dda64 P 2f306549407b4bb281ccf2994035e4f1: Deleting consensus metadata
I20240502 01:53:48.840341 20741 ts_tablet_manager.cc:1875] T 2f29bba45e1749eb96b08da5e15fb834 P 4da690c0fa13407a85c0689ce8de9ade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:48.841063 17322 catalog_manager.cc:4830] TS 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): tablet 438e8c379c7046109ea36496ce3dda64 (table rebalancer_dynamic_table_new_name [id=32e9f4e95fdf465f948510432b71e6af]) successfully deleted
I20240502 01:53:48.841835 20892 ts_tablet_manager.cc:1888] T ae24680ad75545d892c9b79833319214 P dfa1152987644bd781b46c7e42a7a685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:48.842308 20892 log.cc:1196] T ae24680ad75545d892c9b79833319214 P dfa1152987644bd781b46c7e42a7a685: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/wals/ae24680ad75545d892c9b79833319214
I20240502 01:53:48.842731 20892 ts_tablet_manager.cc:1909] T ae24680ad75545d892c9b79833319214 P dfa1152987644bd781b46c7e42a7a685: Deleting consensus metadata
I20240502 01:53:48.843854 20737 ts_tablet_manager.cc:1888] T ae24680ad75545d892c9b79833319214 P 3fceccefba274b1dbbd8216af8b0bc29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:48.843981 20737 log.cc:1196] T ae24680ad75545d892c9b79833319214 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/wals/ae24680ad75545d892c9b79833319214
I20240502 01:53:48.844396 20737 ts_tablet_manager.cc:1909] T ae24680ad75545d892c9b79833319214 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting consensus metadata
I20240502 01:53:48.845098 17321 catalog_manager.cc:4830] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): tablet ae24680ad75545d892c9b79833319214 (table rebalance_test_table_4 [id=0a0284719f394d1dab3221ab9abcf498]) successfully deleted
I20240502 01:53:48.845474 17319 catalog_manager.cc:4830] TS 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): tablet ae24680ad75545d892c9b79833319214 (table rebalance_test_table_4 [id=0a0284719f394d1dab3221ab9abcf498]) successfully deleted
I20240502 01:53:48.848544 20741 ts_tablet_manager.cc:1888] T 2f29bba45e1749eb96b08da5e15fb834 P 4da690c0fa13407a85c0689ce8de9ade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:48.848649 20741 log.cc:1196] T 2f29bba45e1749eb96b08da5e15fb834 P 4da690c0fa13407a85c0689ce8de9ade: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/2f29bba45e1749eb96b08da5e15fb834
I20240502 01:53:48.849169 20892 tablet_replica.cc:330] T 2f29bba45e1749eb96b08da5e15fb834 P dfa1152987644bd781b46c7e42a7a685: stopping tablet replica
I20240502 01:53:48.849404 20892 raft_consensus.cc:2229] T 2f29bba45e1749eb96b08da5e15fb834 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:48.849630 20892 raft_consensus.cc:2258] T 2f29bba45e1749eb96b08da5e15fb834 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:48.850314 20892 ts_tablet_manager.cc:1875] T 2f29bba45e1749eb96b08da5e15fb834 P dfa1152987644bd781b46c7e42a7a685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:48.851670 17335 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_extra_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:48.857153 20741 ts_tablet_manager.cc:1909] T 2f29bba45e1749eb96b08da5e15fb834 P 4da690c0fa13407a85c0689ce8de9ade: Deleting consensus metadata
I20240502 01:53:48.857249 20892 ts_tablet_manager.cc:1888] T 2f29bba45e1749eb96b08da5e15fb834 P dfa1152987644bd781b46c7e42a7a685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:48.857379 20892 log.cc:1196] T 2f29bba45e1749eb96b08da5e15fb834 P dfa1152987644bd781b46c7e42a7a685: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/wals/2f29bba45e1749eb96b08da5e15fb834
I20240502 01:53:48.857758 20892 ts_tablet_manager.cc:1909] T 2f29bba45e1749eb96b08da5e15fb834 P dfa1152987644bd781b46c7e42a7a685: Deleting consensus metadata
I20240502 01:53:48.858448 17321 catalog_manager.cc:4830] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): tablet 2f29bba45e1749eb96b08da5e15fb834 (table rebalance_test_table_4 [id=0a0284719f394d1dab3221ab9abcf498]) successfully deleted
I20240502 01:53:48.858922 17321 catalog_manager.cc:4830] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): tablet 2f29bba45e1749eb96b08da5e15fb834 (table rebalance_test_table_4 [id=0a0284719f394d1dab3221ab9abcf498]) successfully deleted
I20240502 01:53:48.860947 20346 tablet_service.cc:1467] Processing CreateTablet for tablet ba1e82a39c2e439889da60007372a99b (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=2444c341261d4eb29c484622291d0517]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:48.861053 20609 tablet_service.cc:1467] Processing CreateTablet for tablet ba1e82a39c2e439889da60007372a99b (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=2444c341261d4eb29c484622291d0517]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:48.861439 20346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba1e82a39c2e439889da60007372a99b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:48.861347 20211 tablet_service.cc:1467] Processing CreateTablet for tablet ba1e82a39c2e439889da60007372a99b (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=2444c341261d4eb29c484622291d0517]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:48.861470 20609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba1e82a39c2e439889da60007372a99b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:48.861727 20211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba1e82a39c2e439889da60007372a99b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:48.868300 20478 tablet_service.cc:1467] Processing CreateTablet for tablet cf0237a4c65b495ba892871dd850bed9 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=2444c341261d4eb29c484622291d0517]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:48.869197 20478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf0237a4c65b495ba892871dd850bed9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:48.874727 20347 tablet_service.cc:1467] Processing CreateTablet for tablet cf0237a4c65b495ba892871dd850bed9 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=2444c341261d4eb29c484622291d0517]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:48.874871 17455 tablet_service.cc:1467] Processing CreateTablet for tablet e070da07e2214744b3dacd1b839310b9 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=2444c341261d4eb29c484622291d0517]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:48.875160 20347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf0237a4c65b495ba892871dd850bed9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:48.875293 17455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e070da07e2214744b3dacd1b839310b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:48.879745 20900 tablet_bootstrap.cc:492] T ba1e82a39c2e439889da60007372a99b P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:53:48.881493 20900 tablet_bootstrap.cc:654] T ba1e82a39c2e439889da60007372a99b P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:48.881440 20610 tablet_service.cc:1467] Processing CreateTablet for tablet cf0237a4c65b495ba892871dd850bed9 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=2444c341261d4eb29c484622291d0517]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:48.881805 20610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf0237a4c65b495ba892871dd850bed9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:48.883555 20212 tablet_service.cc:1467] Processing CreateTablet for tablet e070da07e2214744b3dacd1b839310b9 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=2444c341261d4eb29c484622291d0517]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:48.883946 20212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e070da07e2214744b3dacd1b839310b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:48.885708 20608 tablet_service.cc:1467] Processing CreateTablet for tablet e070da07e2214744b3dacd1b839310b9 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=2444c341261d4eb29c484622291d0517]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:48.886154 20608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e070da07e2214744b3dacd1b839310b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:48.893358 20901 tablet_bootstrap.cc:492] T ba1e82a39c2e439889da60007372a99b P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:53:48.895380 20901 tablet_bootstrap.cc:654] T ba1e82a39c2e439889da60007372a99b P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:48.895979 20903 tablet_bootstrap.cc:492] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:53:48.901681 20903 tablet_bootstrap.cc:654] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:48.903301 20900 tablet_bootstrap.cc:492] T ba1e82a39c2e439889da60007372a99b P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:53:48.903465 20900 ts_tablet_manager.cc:1367] T ba1e82a39c2e439889da60007372a99b P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.008s
I20240502 01:53:48.904296 20900 raft_consensus.cc:348] T ba1e82a39c2e439889da60007372a99b P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:48.904525 20900 raft_consensus.cc:374] T ba1e82a39c2e439889da60007372a99b P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:48.904604 20900 raft_consensus.cc:729] T ba1e82a39c2e439889da60007372a99b P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:53:48.904839 20900 consensus_queue.cc:260] T ba1e82a39c2e439889da60007372a99b P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:48.905483 20900 ts_tablet_manager.cc:1398] T ba1e82a39c2e439889da60007372a99b P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:48.905799 20900 tablet_bootstrap.cc:492] T e070da07e2214744b3dacd1b839310b9 P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:53:48.907475 20900 tablet_bootstrap.cc:654] T e070da07e2214744b3dacd1b839310b9 P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:48.912438 17328 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:53:48.925479 17587 tablet_service.cc:1467] Processing CreateTablet for tablet 6d119e18319943148f879494ee7b07ed (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2c26265a42364c878d9c982a6cbb7f48]), partition=
I20240502 01:53:48.925827 17587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d119e18319943148f879494ee7b07ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:48.928422 20347 tablet_service.cc:1467] Processing CreateTablet for tablet 6d119e18319943148f879494ee7b07ed (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2c26265a42364c878d9c982a6cbb7f48]), partition=
I20240502 01:53:48.928748 20347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d119e18319943148f879494ee7b07ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:48.932096 20609 tablet_service.cc:1467] Processing CreateTablet for tablet 6d119e18319943148f879494ee7b07ed (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2c26265a42364c878d9c982a6cbb7f48]), partition=
I20240502 01:53:48.932435 20609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d119e18319943148f879494ee7b07ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:48.944103 20910 tablet_bootstrap.cc:492] T 6d119e18319943148f879494ee7b07ed P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:48.944936 20906 tablet_bootstrap.cc:492] T e070da07e2214744b3dacd1b839310b9 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:48.948690 20904 tablet_bootstrap.cc:492] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:53:48.949672 20900 tablet_bootstrap.cc:492] T e070da07e2214744b3dacd1b839310b9 P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:53:48.949865 20900 ts_tablet_manager.cc:1367] T e070da07e2214744b3dacd1b839310b9 P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.044s user 0.014s sys 0.025s
I20240502 01:53:48.950476 20906 tablet_bootstrap.cc:654] T e070da07e2214744b3dacd1b839310b9 P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:48.950699 20900 raft_consensus.cc:348] T e070da07e2214744b3dacd1b839310b9 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:48.950933 20900 raft_consensus.cc:374] T e070da07e2214744b3dacd1b839310b9 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:48.951009 20900 raft_consensus.cc:729] T e070da07e2214744b3dacd1b839310b9 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:53:48.951267 20900 consensus_queue.cc:260] T e070da07e2214744b3dacd1b839310b9 P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:48.952517 20910 tablet_bootstrap.cc:654] T 6d119e18319943148f879494ee7b07ed P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:48.952744 20906 tablet_bootstrap.cc:492] T e070da07e2214744b3dacd1b839310b9 P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:48.952891 20906 ts_tablet_manager.cc:1367] T e070da07e2214744b3dacd1b839310b9 P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240502 01:53:48.953697 20906 raft_consensus.cc:348] T e070da07e2214744b3dacd1b839310b9 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:48.953902 20906 raft_consensus.cc:374] T e070da07e2214744b3dacd1b839310b9 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:48.953928 20900 ts_tablet_manager.cc:1398] T e070da07e2214744b3dacd1b839310b9 P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 01:53:48.953990 20906 raft_consensus.cc:729] T e070da07e2214744b3dacd1b839310b9 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:48.954295 20900 tablet_bootstrap.cc:492] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:53:48.954226 20906 consensus_queue.cc:260] T e070da07e2214744b3dacd1b839310b9 P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:48.954807 20906 ts_tablet_manager.cc:1398] T e070da07e2214744b3dacd1b839310b9 P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:53:48.954432 20901 tablet_bootstrap.cc:492] T ba1e82a39c2e439889da60007372a99b P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:53:48.956020 20901 ts_tablet_manager.cc:1367] T ba1e82a39c2e439889da60007372a99b P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.063s user 0.010s sys 0.000s
I20240502 01:53:48.956947 20901 raft_consensus.cc:348] T ba1e82a39c2e439889da60007372a99b P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:48.957252 20901 raft_consensus.cc:374] T ba1e82a39c2e439889da60007372a99b P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:48.957396 20901 raft_consensus.cc:729] T ba1e82a39c2e439889da60007372a99b P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:53:48.957712 20901 consensus_queue.cc:260] T ba1e82a39c2e439889da60007372a99b P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:48.960928 20900 tablet_bootstrap.cc:654] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:48.960742 20904 tablet_bootstrap.cc:654] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:48.961233 20901 ts_tablet_manager.cc:1398] T ba1e82a39c2e439889da60007372a99b P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:53:48.963745 20910 tablet_bootstrap.cc:492] T 6d119e18319943148f879494ee7b07ed P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:48.963935 20910 ts_tablet_manager.cc:1367] T 6d119e18319943148f879494ee7b07ed P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20240502 01:53:48.964746 20901 tablet_bootstrap.cc:492] T e070da07e2214744b3dacd1b839310b9 P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:53:48.964771 20910 raft_consensus.cc:348] T 6d119e18319943148f879494ee7b07ed P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:48.964977 20910 raft_consensus.cc:374] T 6d119e18319943148f879494ee7b07ed P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:48.965117 20910 raft_consensus.cc:729] T 6d119e18319943148f879494ee7b07ed P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:48.965375 20910 consensus_queue.cc:260] T 6d119e18319943148f879494ee7b07ed P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:48.966077 20910 ts_tablet_manager.cc:1398] T 6d119e18319943148f879494ee7b07ed P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:48.968148 20901 tablet_bootstrap.cc:654] T e070da07e2214744b3dacd1b839310b9 P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:48.972545 20901 tablet_bootstrap.cc:492] T e070da07e2214744b3dacd1b839310b9 P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:53:48.972769 20901 ts_tablet_manager.cc:1367] T e070da07e2214744b3dacd1b839310b9 P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240502 01:53:48.973584 20901 raft_consensus.cc:348] T e070da07e2214744b3dacd1b839310b9 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:48.974915 20901 raft_consensus.cc:374] T e070da07e2214744b3dacd1b839310b9 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:48.975049 20901 raft_consensus.cc:729] T e070da07e2214744b3dacd1b839310b9 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:53:48.975840 20901 consensus_queue.cc:260] T e070da07e2214744b3dacd1b839310b9 P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:48.976281 20900 tablet_bootstrap.cc:492] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:53:48.976437 20900 ts_tablet_manager.cc:1367] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.007s
I20240502 01:53:48.977393 20901 ts_tablet_manager.cc:1398] T e070da07e2214744b3dacd1b839310b9 P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 01:53:48.977350 20900 raft_consensus.cc:348] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:48.977623 20900 raft_consensus.cc:374] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:48.977707 20900 raft_consensus.cc:729] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:53:48.977921 20900 consensus_queue.cc:260] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:48.978412 20900 ts_tablet_manager.cc:1398] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:48.978720 20900 tablet_bootstrap.cc:492] T 6d119e18319943148f879494ee7b07ed P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:53:48.979596 20903 tablet_bootstrap.cc:492] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:53:48.979773 20903 ts_tablet_manager.cc:1367] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.084s user 0.038s sys 0.025s
I20240502 01:53:48.980245 20900 tablet_bootstrap.cc:654] T 6d119e18319943148f879494ee7b07ed P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:48.980635 20903 raft_consensus.cc:348] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:48.980821 20903 raft_consensus.cc:374] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:48.980902 20903 raft_consensus.cc:729] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:53:48.981145 20903 consensus_queue.cc:260] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:48.981860 20903 ts_tablet_manager.cc:1398] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:53:48.982107 20903 tablet_bootstrap.cc:492] T cf0237a4c65b495ba892871dd850bed9 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:53:48.982851 20762 raft_consensus.cc:482] T e070da07e2214744b3dacd1b839310b9 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:48.982990 20762 raft_consensus.cc:504] T e070da07e2214744b3dacd1b839310b9 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:48.983685 20903 tablet_bootstrap.cc:654] T cf0237a4c65b495ba892871dd850bed9 P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:48.983758 20762 leader_election.cc:288] T e070da07e2214744b3dacd1b839310b9 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:53:48.983729 20698 raft_consensus.cc:482] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:48.984367 20698 raft_consensus.cc:504] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:48.985157 20698 leader_election.cc:288] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:53:48.985100 20233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e070da07e2214744b3dacd1b839310b9" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:53:48.985349 20233 raft_consensus.cc:2454] T e070da07e2214744b3dacd1b839310b9 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 0.
I20240502 01:53:48.986848 20367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf0237a4c65b495ba892871dd850bed9" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:53:48.988119 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e070da07e2214744b3dacd1b839310b9" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:48.988350 17475 raft_consensus.cc:2454] T e070da07e2214744b3dacd1b839310b9 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 0.
W20240502 01:53:48.988984 20564 leader_election.cc:341] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:53:48.990183 20564 leader_election.cc:302] T e070da07e2214744b3dacd1b839310b9 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:53:48.990700 20762 raft_consensus.cc:2781] T e070da07e2214744b3dacd1b839310b9 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:48.990832 20762 raft_consensus.cc:482] T e070da07e2214744b3dacd1b839310b9 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:48.990916 20762 raft_consensus.cc:3037] T e070da07e2214744b3dacd1b839310b9 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:48.994545 20877 raft_consensus.cc:482] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:48.994702 20877 raft_consensus.cc:504] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:48.995579 20877 leader_election.cc:288] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:53:48.996649 20630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba1e82a39c2e439889da60007372a99b" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091" is_pre_election: true
I20240502 01:53:48.997443 20233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba1e82a39c2e439889da60007372a99b" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:53:48.997691 20233 raft_consensus.cc:2454] T ba1e82a39c2e439889da60007372a99b P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 0.
I20240502 01:53:48.996940 20630 raft_consensus.cc:2454] T ba1e82a39c2e439889da60007372a99b P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 0.
I20240502 01:53:48.997996 20762 raft_consensus.cc:504] T e070da07e2214744b3dacd1b839310b9 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:48.998456 20302 leader_election.cc:302] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:53:48.998793 20877 raft_consensus.cc:2781] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:48.998906 20877 raft_consensus.cc:482] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:48.998987 20877 raft_consensus.cc:3037] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:48.999850 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e070da07e2214744b3dacd1b839310b9" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:53:49.000102 17475 raft_consensus.cc:3037] T e070da07e2214744b3dacd1b839310b9 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:49.003227 17475 raft_consensus.cc:2454] T e070da07e2214744b3dacd1b839310b9 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 1.
I20240502 01:53:49.003373 20877 raft_consensus.cc:504] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:49.003677 20564 leader_election.cc:302] T e070da07e2214744b3dacd1b839310b9 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:53:49.004004 20698 raft_consensus.cc:2781] T e070da07e2214744b3dacd1b839310b9 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:49.006395 20630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba1e82a39c2e439889da60007372a99b" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091"
I20240502 01:53:49.006614 20630 raft_consensus.cc:3037] T ba1e82a39c2e439889da60007372a99b P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:49.009416 20877 leader_election.cc:288] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Requested vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:53:49.009495 20762 leader_election.cc:288] T e070da07e2214744b3dacd1b839310b9 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 election: Requested vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:53:49.010416 20233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba1e82a39c2e439889da60007372a99b" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:53:49.010650 20233 raft_consensus.cc:3037] T ba1e82a39c2e439889da60007372a99b P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:49.015183 20904 tablet_bootstrap.cc:492] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:53:49.015605 20630 raft_consensus.cc:2454] T ba1e82a39c2e439889da60007372a99b P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 1.
I20240502 01:53:49.016093 20302 leader_election.cc:302] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:53:49.017509 20904 ts_tablet_manager.cc:1367] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.069s user 0.005s sys 0.000s
I20240502 01:53:49.017577 20233 raft_consensus.cc:2454] T ba1e82a39c2e439889da60007372a99b P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 1.
I20240502 01:53:49.018342 20877 raft_consensus.cc:2781] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:49.018963 20877 raft_consensus.cc:686] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Becoming Leader. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Running, Role: LEADER
I20240502 01:53:49.021216 20877 consensus_queue.cc:237] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:49.021651 20904 raft_consensus.cc:348] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:49.024219 20904 raft_consensus.cc:374] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:49.024551 20904 raft_consensus.cc:729] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:53:49.026631 20904 consensus_queue.cc:260] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:49.031029 20904 ts_tablet_manager.cc:1398] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240502 01:53:49.058396 20698 raft_consensus.cc:686] T e070da07e2214744b3dacd1b839310b9 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Becoming Leader. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Running, Role: LEADER
I20240502 01:53:49.059396 20698 consensus_queue.cc:237] T e070da07e2214744b3dacd1b839310b9 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:49.059409 17328 catalog_manager.cc:5484] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70). New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:49.067842 20233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e070da07e2214744b3dacd1b839310b9" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:53:49.068740 20233 raft_consensus.cc:3037] T e070da07e2214744b3dacd1b839310b9 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:49.073719 20233 raft_consensus.cc:2454] T e070da07e2214744b3dacd1b839310b9 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 1.
I20240502 01:53:49.077762 20903 tablet_bootstrap.cc:492] T cf0237a4c65b495ba892871dd850bed9 P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:53:49.078015 20903 ts_tablet_manager.cc:1367] T cf0237a4c65b495ba892871dd850bed9 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.096s user 0.039s sys 0.053s
I20240502 01:53:49.079208 20903 raft_consensus.cc:348] T cf0237a4c65b495ba892871dd850bed9 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:49.082119 20903 raft_consensus.cc:374] T cf0237a4c65b495ba892871dd850bed9 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:49.082963 20903 raft_consensus.cc:729] T cf0237a4c65b495ba892871dd850bed9 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:53:49.084249 20903 consensus_queue.cc:260] T cf0237a4c65b495ba892871dd850bed9 P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:49.084388 20498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf0237a4c65b495ba892871dd850bed9" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
I20240502 01:53:49.084617 20498 raft_consensus.cc:2454] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 0.
I20240502 01:53:49.086398 17335 catalog_manager.cc:5484] T e070da07e2214744b3dacd1b839310b9 P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: term changed from 0 to 1, leader changed from <none> to f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72). New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:49.087412 20903 ts_tablet_manager.cc:1398] T cf0237a4c65b495ba892871dd850bed9 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240502 01:53:49.090013 20903 tablet_bootstrap.cc:492] T 6d119e18319943148f879494ee7b07ed P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:53:49.090413 20565 leader_election.cc:302] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, f19cc6cb1e76460aa6711e49a5775091; no voters: 6a36c952c51542c0bf7ea3ec2857a9f9
I20240502 01:53:49.090950 20923 raft_consensus.cc:2781] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:49.091106 20923 raft_consensus.cc:482] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:49.091295 20923 raft_consensus.cc:3037] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:49.092214 20903 tablet_bootstrap.cc:654] T 6d119e18319943148f879494ee7b07ed P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:49.093600 20900 tablet_bootstrap.cc:492] T 6d119e18319943148f879494ee7b07ed P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:53:49.094051 20900 ts_tablet_manager.cc:1367] T 6d119e18319943148f879494ee7b07ed P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.115s user 0.044s sys 0.053s
I20240502 01:53:49.094422 20923 raft_consensus.cc:504] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:49.094884 20900 raft_consensus.cc:348] T 6d119e18319943148f879494ee7b07ed P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:49.095209 20900 raft_consensus.cc:374] T 6d119e18319943148f879494ee7b07ed P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:49.095297 20900 raft_consensus.cc:729] T 6d119e18319943148f879494ee7b07ed P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:53:49.095526 20900 consensus_queue.cc:260] T 6d119e18319943148f879494ee7b07ed P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:49.095814 20498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf0237a4c65b495ba892871dd850bed9" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
I20240502 01:53:49.096011 20498 raft_consensus.cc:3037] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:49.096087 20900 ts_tablet_manager.cc:1398] T 6d119e18319943148f879494ee7b07ed P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:49.098718 20498 raft_consensus.cc:2454] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 1.
I20240502 01:53:49.101478 20565 leader_election.cc:302] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:53:49.101811 20762 raft_consensus.cc:2781] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:49.107679 20923 leader_election.cc:288] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 election: Requested vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:53:49.107705 20367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf0237a4c65b495ba892871dd850bed9" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
I20240502 01:53:49.107935 20367 raft_consensus.cc:3037] T cf0237a4c65b495ba892871dd850bed9 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:49.111191 20367 raft_consensus.cc:2454] T cf0237a4c65b495ba892871dd850bed9 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 1.
I20240502 01:53:49.128113 20903 tablet_bootstrap.cc:492] T 6d119e18319943148f879494ee7b07ed P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:53:49.128345 20903 ts_tablet_manager.cc:1367] T 6d119e18319943148f879494ee7b07ed P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.038s user 0.026s sys 0.009s
I20240502 01:53:49.129125 20903 raft_consensus.cc:348] T 6d119e18319943148f879494ee7b07ed P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:49.129331 20903 raft_consensus.cc:374] T 6d119e18319943148f879494ee7b07ed P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:49.129433 20903 raft_consensus.cc:729] T 6d119e18319943148f879494ee7b07ed P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:53:49.129679 20903 consensus_queue.cc:260] T 6d119e18319943148f879494ee7b07ed P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:49.132509 20903 ts_tablet_manager.cc:1398] T 6d119e18319943148f879494ee7b07ed P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:53:49.144829 20762 raft_consensus.cc:686] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Becoming Leader. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Running, Role: LEADER
I20240502 01:53:49.145242 20762 consensus_queue.cc:237] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:49.155985 17335 catalog_manager.cc:5484] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: term changed from 0 to 1, leader changed from <none> to f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72). New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: HEALTHY } } }
I20240502 01:53:49.302467 17328 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240502 01:53:49.302739 17328 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240502 01:53:49.305876 17328 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet 4280160c9b3a4b34b3c7d89f0fbec666
I20240502 01:53:49.306336 17328 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet 34038775b5fa4b3193b8e57f09102ea3
I20240502 01:53:49.306771 17716 tablet_service.cc:1514] Processing DeleteTablet for tablet 4280160c9b3a4b34b3c7d89f0fbec666 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:49 UTC) from {username='slave'} at 127.0.0.1:52198
I20240502 01:53:49.306960 17587 tablet_service.cc:1514] Processing DeleteTablet for tablet 4280160c9b3a4b34b3c7d89f0fbec666 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:49 UTC) from {username='slave'} at 127.0.0.1:55232
I20240502 01:53:49.306964 17455 tablet_service.cc:1514] Processing DeleteTablet for tablet 4280160c9b3a4b34b3c7d89f0fbec666 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:49 UTC) from {username='slave'} at 127.0.0.1:54520
I20240502 01:53:49.307108 20892 tablet_replica.cc:330] T 4280160c9b3a4b34b3c7d89f0fbec666 P dfa1152987644bd781b46c7e42a7a685: stopping tablet replica
I20240502 01:53:49.307293 20741 tablet_replica.cc:330] T 4280160c9b3a4b34b3c7d89f0fbec666 P 4da690c0fa13407a85c0689ce8de9ade: stopping tablet replica
I20240502 01:53:49.307271 17846 tablet_service.cc:1514] Processing DeleteTablet for tablet 34038775b5fa4b3193b8e57f09102ea3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:49 UTC) from {username='slave'} at 127.0.0.1:47226
I20240502 01:53:49.307349 20892 raft_consensus.cc:2229] T 4280160c9b3a4b34b3c7d89f0fbec666 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:49.307552 20738 tablet_replica.cc:330] T 34038775b5fa4b3193b8e57f09102ea3 P bb31bceba76d466db24b9341c04900c3: stopping tablet replica
I20240502 01:53:49.307551 20741 raft_consensus.cc:2229] T 4280160c9b3a4b34b3c7d89f0fbec666 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:49.307638 20892 raft_consensus.cc:2258] T 4280160c9b3a4b34b3c7d89f0fbec666 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:49.307761 20738 raft_consensus.cc:2229] T 34038775b5fa4b3193b8e57f09102ea3 P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:49.307562 17455 tablet_service.cc:1514] Processing DeleteTablet for tablet 34038775b5fa4b3193b8e57f09102ea3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:49 UTC) from {username='slave'} at 127.0.0.1:54520
I20240502 01:53:49.307816 20741 raft_consensus.cc:2258] T 4280160c9b3a4b34b3c7d89f0fbec666 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:49.308120 20738 raft_consensus.cc:2258] T 34038775b5fa4b3193b8e57f09102ea3 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:49.308341 20892 ts_tablet_manager.cc:1875] T 4280160c9b3a4b34b3c7d89f0fbec666 P dfa1152987644bd781b46c7e42a7a685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:49.308604 20741 ts_tablet_manager.cc:1875] T 4280160c9b3a4b34b3c7d89f0fbec666 P 4da690c0fa13407a85c0689ce8de9ade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:49.308756 20738 ts_tablet_manager.cc:1875] T 34038775b5fa4b3193b8e57f09102ea3 P bb31bceba76d466db24b9341c04900c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:49.312911 20892 ts_tablet_manager.cc:1888] T 4280160c9b3a4b34b3c7d89f0fbec666 P dfa1152987644bd781b46c7e42a7a685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:49.313174 20737 tablet_replica.cc:330] T 4280160c9b3a4b34b3c7d89f0fbec666 P 3fceccefba274b1dbbd8216af8b0bc29: stopping tablet replica
I20240502 01:53:49.313362 20737 raft_consensus.cc:2229] T 4280160c9b3a4b34b3c7d89f0fbec666 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:49.313611 20737 raft_consensus.cc:2258] T 4280160c9b3a4b34b3c7d89f0fbec666 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:49.314142 20737 ts_tablet_manager.cc:1875] T 4280160c9b3a4b34b3c7d89f0fbec666 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:49.314851 20741 ts_tablet_manager.cc:1888] T 4280160c9b3a4b34b3c7d89f0fbec666 P 4da690c0fa13407a85c0689ce8de9ade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:49.314867 20738 ts_tablet_manager.cc:1888] T 34038775b5fa4b3193b8e57f09102ea3 P bb31bceba76d466db24b9341c04900c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:49.314962 20738 log.cc:1196] T 34038775b5fa4b3193b8e57f09102ea3 P bb31bceba76d466db24b9341c04900c3: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/wals/34038775b5fa4b3193b8e57f09102ea3
I20240502 01:53:49.314980 20741 log.cc:1196] T 4280160c9b3a4b34b3c7d89f0fbec666 P 4da690c0fa13407a85c0689ce8de9ade: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/4280160c9b3a4b34b3c7d89f0fbec666
I20240502 01:53:49.315366 20738 ts_tablet_manager.cc:1909] T 34038775b5fa4b3193b8e57f09102ea3 P bb31bceba76d466db24b9341c04900c3: Deleting consensus metadata
I20240502 01:53:49.315389 20741 ts_tablet_manager.cc:1909] T 4280160c9b3a4b34b3c7d89f0fbec666 P 4da690c0fa13407a85c0689ce8de9ade: Deleting consensus metadata
I20240502 01:53:49.316344 17322 catalog_manager.cc:4830] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): tablet 34038775b5fa4b3193b8e57f09102ea3 (table rebalance_test_table_6 [id=9631b34b6ef449edb5476c988aed9d7a]) successfully deleted
I20240502 01:53:49.316567 17321 catalog_manager.cc:4830] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): tablet 4280160c9b3a4b34b3c7d89f0fbec666 (table rebalance_test_table_6 [id=9631b34b6ef449edb5476c988aed9d7a]) successfully deleted
I20240502 01:53:49.316809 17328 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet e00020f7f47f4246907a8c90fca4a26b
I20240502 01:53:49.316942 17716 tablet_service.cc:1514] Processing DeleteTablet for tablet 34038775b5fa4b3193b8e57f09102ea3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:49 UTC) from {username='slave'} at 127.0.0.1:52198
I20240502 01:53:49.312996 20892 log.cc:1196] T 4280160c9b3a4b34b3c7d89f0fbec666 P dfa1152987644bd781b46c7e42a7a685: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/wals/4280160c9b3a4b34b3c7d89f0fbec666
I20240502 01:53:49.317472 17716 tablet_service.cc:1514] Processing DeleteTablet for tablet e00020f7f47f4246907a8c90fca4a26b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:49 UTC) from {username='slave'} at 127.0.0.1:52198
I20240502 01:53:49.318068 17455 tablet_service.cc:1514] Processing DeleteTablet for tablet e00020f7f47f4246907a8c90fca4a26b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:49 UTC) from {username='slave'} at 127.0.0.1:54520
I20240502 01:53:49.318554 20741 tablet_replica.cc:330] T 34038775b5fa4b3193b8e57f09102ea3 P 4da690c0fa13407a85c0689ce8de9ade: stopping tablet replica
I20240502 01:53:49.317948 20892 ts_tablet_manager.cc:1909] T 4280160c9b3a4b34b3c7d89f0fbec666 P dfa1152987644bd781b46c7e42a7a685: Deleting consensus metadata
I20240502 01:53:49.318820 20741 raft_consensus.cc:2229] T 34038775b5fa4b3193b8e57f09102ea3 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:49.318971 17587 tablet_service.cc:1514] Processing DeleteTablet for tablet e00020f7f47f4246907a8c90fca4a26b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:49 UTC) from {username='slave'} at 127.0.0.1:55232
I20240502 01:53:49.319095 20741 raft_consensus.cc:2258] T 34038775b5fa4b3193b8e57f09102ea3 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:49.319746 20741 ts_tablet_manager.cc:1875] T 34038775b5fa4b3193b8e57f09102ea3 P 4da690c0fa13407a85c0689ce8de9ade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:49.319893 17321 catalog_manager.cc:4830] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): tablet 4280160c9b3a4b34b3c7d89f0fbec666 (table rebalance_test_table_6 [id=9631b34b6ef449edb5476c988aed9d7a]) successfully deleted
I20240502 01:53:49.320406 20892 tablet_replica.cc:330] T 34038775b5fa4b3193b8e57f09102ea3 P dfa1152987644bd781b46c7e42a7a685: stopping tablet replica
I20240502 01:53:49.320583 20737 ts_tablet_manager.cc:1888] T 4280160c9b3a4b34b3c7d89f0fbec666 P 3fceccefba274b1dbbd8216af8b0bc29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:49.320672 20892 raft_consensus.cc:2229] T 34038775b5fa4b3193b8e57f09102ea3 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:49.320724 20737 log.cc:1196] T 4280160c9b3a4b34b3c7d89f0fbec666 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/wals/4280160c9b3a4b34b3c7d89f0fbec666
I20240502 01:53:49.320916 20892 raft_consensus.cc:2258] T 34038775b5fa4b3193b8e57f09102ea3 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:49.321208 20737 ts_tablet_manager.cc:1909] T 4280160c9b3a4b34b3c7d89f0fbec666 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting consensus metadata
I20240502 01:53:49.321607 20892 ts_tablet_manager.cc:1875] T 34038775b5fa4b3193b8e57f09102ea3 P dfa1152987644bd781b46c7e42a7a685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:49.322083 20737 tablet_replica.cc:330] T e00020f7f47f4246907a8c90fca4a26b P 3fceccefba274b1dbbd8216af8b0bc29: stopping tablet replica
I20240502 01:53:49.322330 20737 raft_consensus.cc:2229] T e00020f7f47f4246907a8c90fca4a26b P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:49.322433 17319 catalog_manager.cc:4830] TS 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): tablet 4280160c9b3a4b34b3c7d89f0fbec666 (table rebalance_test_table_6 [id=9631b34b6ef449edb5476c988aed9d7a]) successfully deleted
I20240502 01:53:49.322676 20737 raft_consensus.cc:2258] T e00020f7f47f4246907a8c90fca4a26b P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:49.323351 20737 ts_tablet_manager.cc:1875] T e00020f7f47f4246907a8c90fca4a26b P 3fceccefba274b1dbbd8216af8b0bc29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:49.324259 20741 ts_tablet_manager.cc:1888] T 34038775b5fa4b3193b8e57f09102ea3 P 4da690c0fa13407a85c0689ce8de9ade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:49.324393 20741 log.cc:1196] T 34038775b5fa4b3193b8e57f09102ea3 P 4da690c0fa13407a85c0689ce8de9ade: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/34038775b5fa4b3193b8e57f09102ea3
I20240502 01:53:49.324746 20741 ts_tablet_manager.cc:1909] T 34038775b5fa4b3193b8e57f09102ea3 P 4da690c0fa13407a85c0689ce8de9ade: Deleting consensus metadata
I20240502 01:53:49.325619 20741 tablet_replica.cc:330] T e00020f7f47f4246907a8c90fca4a26b P 4da690c0fa13407a85c0689ce8de9ade: stopping tablet replica
I20240502 01:53:49.325659 17321 catalog_manager.cc:4830] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): tablet 34038775b5fa4b3193b8e57f09102ea3 (table rebalance_test_table_6 [id=9631b34b6ef449edb5476c988aed9d7a]) successfully deleted
I20240502 01:53:49.325821 20741 raft_consensus.cc:2229] T e00020f7f47f4246907a8c90fca4a26b P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:49.326063 20741 raft_consensus.cc:2258] T e00020f7f47f4246907a8c90fca4a26b P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:49.326743 20741 ts_tablet_manager.cc:1875] T e00020f7f47f4246907a8c90fca4a26b P 4da690c0fa13407a85c0689ce8de9ade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:49.328545 20892 ts_tablet_manager.cc:1888] T 34038775b5fa4b3193b8e57f09102ea3 P dfa1152987644bd781b46c7e42a7a685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:49.328685 20892 log.cc:1196] T 34038775b5fa4b3193b8e57f09102ea3 P dfa1152987644bd781b46c7e42a7a685: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/wals/34038775b5fa4b3193b8e57f09102ea3
I20240502 01:53:49.329088 20892 ts_tablet_manager.cc:1909] T 34038775b5fa4b3193b8e57f09102ea3 P dfa1152987644bd781b46c7e42a7a685: Deleting consensus metadata
I20240502 01:53:49.329603 20737 ts_tablet_manager.cc:1888] T e00020f7f47f4246907a8c90fca4a26b P 3fceccefba274b1dbbd8216af8b0bc29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:49.329774 20737 log.cc:1196] T e00020f7f47f4246907a8c90fca4a26b P 3fceccefba274b1dbbd8216af8b0bc29: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/wals/e00020f7f47f4246907a8c90fca4a26b
I20240502 01:53:49.330185 17321 catalog_manager.cc:4830] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): tablet 34038775b5fa4b3193b8e57f09102ea3 (table rebalance_test_table_6 [id=9631b34b6ef449edb5476c988aed9d7a]) successfully deleted
I20240502 01:53:49.330260 20737 ts_tablet_manager.cc:1909] T e00020f7f47f4246907a8c90fca4a26b P 3fceccefba274b1dbbd8216af8b0bc29: Deleting consensus metadata
I20240502 01:53:49.330677 20892 tablet_replica.cc:330] T e00020f7f47f4246907a8c90fca4a26b P dfa1152987644bd781b46c7e42a7a685: stopping tablet replica
I20240502 01:53:49.330880 20892 raft_consensus.cc:2229] T e00020f7f47f4246907a8c90fca4a26b P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:49.331264 20892 raft_consensus.cc:2258] T e00020f7f47f4246907a8c90fca4a26b P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:49.331427 20741 ts_tablet_manager.cc:1888] T e00020f7f47f4246907a8c90fca4a26b P 4da690c0fa13407a85c0689ce8de9ade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:49.331526 17319 catalog_manager.cc:4830] TS 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): tablet e00020f7f47f4246907a8c90fca4a26b (table rebalance_test_table_6 [id=9631b34b6ef449edb5476c988aed9d7a]) successfully deleted
I20240502 01:53:49.331568 20741 log.cc:1196] T e00020f7f47f4246907a8c90fca4a26b P 4da690c0fa13407a85c0689ce8de9ade: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/e00020f7f47f4246907a8c90fca4a26b
I20240502 01:53:49.331910 20892 ts_tablet_manager.cc:1875] T e00020f7f47f4246907a8c90fca4a26b P dfa1152987644bd781b46c7e42a7a685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:49.332046 20741 ts_tablet_manager.cc:1909] T e00020f7f47f4246907a8c90fca4a26b P 4da690c0fa13407a85c0689ce8de9ade: Deleting consensus metadata
I20240502 01:53:49.333065 17321 catalog_manager.cc:4830] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): tablet e00020f7f47f4246907a8c90fca4a26b (table rebalance_test_table_6 [id=9631b34b6ef449edb5476c988aed9d7a]) successfully deleted
I20240502 01:53:49.336783 20892 ts_tablet_manager.cc:1888] T e00020f7f47f4246907a8c90fca4a26b P dfa1152987644bd781b46c7e42a7a685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:49.336886 20892 log.cc:1196] T e00020f7f47f4246907a8c90fca4a26b P dfa1152987644bd781b46c7e42a7a685: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/wals/e00020f7f47f4246907a8c90fca4a26b
I20240502 01:53:49.337225 20892 ts_tablet_manager.cc:1909] T e00020f7f47f4246907a8c90fca4a26b P dfa1152987644bd781b46c7e42a7a685: Deleting consensus metadata
I20240502 01:53:49.338074 17321 catalog_manager.cc:4830] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): tablet e00020f7f47f4246907a8c90fca4a26b (table rebalance_test_table_6 [id=9631b34b6ef449edb5476c988aed9d7a]) successfully deleted
I20240502 01:53:49.377673 20868 raft_consensus.cc:482] T 6d119e18319943148f879494ee7b07ed P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:49.377833 20868 raft_consensus.cc:504] T 6d119e18319943148f879494ee7b07ed P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:49.378468 20868 leader_election.cc:288] T 6d119e18319943148f879494ee7b07ed P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:53:49.378971 20630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d119e18319943148f879494ee7b07ed" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091" is_pre_election: true
I20240502 01:53:49.379235 20630 raft_consensus.cc:2454] T 6d119e18319943148f879494ee7b07ed P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:53:49.380091 17542 leader_election.cc:302] T 6d119e18319943148f879494ee7b07ed P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:53:49.380390 20868 raft_consensus.cc:2781] T 6d119e18319943148f879494ee7b07ed P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:49.380517 20868 raft_consensus.cc:482] T 6d119e18319943148f879494ee7b07ed P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:49.380609 20868 raft_consensus.cc:3037] T 6d119e18319943148f879494ee7b07ed P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:49.383143 20868 raft_consensus.cc:504] T 6d119e18319943148f879494ee7b07ed P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:49.383745 20868 leader_election.cc:288] T 6d119e18319943148f879494ee7b07ed P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Requested vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:53:49.384168 20630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d119e18319943148f879494ee7b07ed" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091"
I20240502 01:53:49.384419 20630 raft_consensus.cc:3037] T 6d119e18319943148f879494ee7b07ed P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:49.386859 20367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d119e18319943148f879494ee7b07ed" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:53:49.387038 20630 raft_consensus.cc:2454] T 6d119e18319943148f879494ee7b07ed P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:53:49.386963 20366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d119e18319943148f879494ee7b07ed" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
I20240502 01:53:49.387138 20367 raft_consensus.cc:2454] T 6d119e18319943148f879494ee7b07ed P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:53:49.387941 17542 leader_election.cc:302] T 6d119e18319943148f879494ee7b07ed P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:53:49.388520 20868 raft_consensus.cc:2781] T 6d119e18319943148f879494ee7b07ed P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:49.388665 20868 raft_consensus.cc:686] T 6d119e18319943148f879494ee7b07ed P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Becoming Leader. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Running, Role: LEADER
I20240502 01:53:49.388943 20868 consensus_queue.cc:237] T 6d119e18319943148f879494ee7b07ed P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:49.392109 17328 catalog_manager.cc:5484] T 6d119e18319943148f879494ee7b07ed P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66). New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } }
I20240502 01:53:49.420840 20765 consensus_queue.cc:1035] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:49.434588 20347 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:49.442131 20478 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:49.454607 17455 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:53:49.459542 17587 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:53:49.478248 17846 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:53:49.483639 20813 consensus_queue.cc:1035] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:49.527925 17716 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:53:49.540755 17334 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:53:49.541100 20609 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:49.572278 20212 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:49.588253 20366 raft_consensus.cc:3037] T 6d119e18319943148f879494ee7b07ed P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:49.591671 20366 raft_consensus.cc:1261] T 6d119e18319943148f879494ee7b07ed P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:49.592432 20868 consensus_queue.cc:1035] T 6d119e18319943148f879494ee7b07ed P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:49.601897 20630 raft_consensus.cc:1261] T 6d119e18319943148f879494ee7b07ed P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:49.602650 20868 consensus_queue.cc:1035] T 6d119e18319943148f879494ee7b07ed P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:49.626119 17328 catalog_manager.cc:5576] TS 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155) does not have the latest schema for tablet 6d119e18319943148f879494ee7b07ed (table rebalancer_dynamic_table [id=2c26265a42364c878d9c982a6cbb7f48]). Expected version 1 got 0
I20240502 01:53:49.635373 17328 catalog_manager.cc:5576] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411) does not have the latest schema for tablet 6d119e18319943148f879494ee7b07ed (table rebalancer_dynamic_table [id=2c26265a42364c878d9c982a6cbb7f48]). Expected version 1 got 0
I20240502 01:53:49.635610 20972 tablet.cc:1722] T 6d119e18319943148f879494ee7b07ed P 3fceccefba274b1dbbd8216af8b0bc29: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:53:49.636166 20972 tablet.cc:1620] T 6d119e18319943148f879494ee7b07ed P 3fceccefba274b1dbbd8216af8b0bc29: MemRowSet was empty: no flush needed.
I20240502 01:53:49.636952 20973 tablet.cc:1722] T 6d119e18319943148f879494ee7b07ed P 6a36c952c51542c0bf7ea3ec2857a9f9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:53:49.640635 17328 catalog_manager.cc:5576] TS 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155) does not have the latest schema for tablet 6d119e18319943148f879494ee7b07ed (table rebalancer_dynamic_table [id=2c26265a42364c878d9c982a6cbb7f48]). Expected version 1 got 0
I20240502 01:53:49.642014 20973 tablet.cc:1620] T 6d119e18319943148f879494ee7b07ed P 6a36c952c51542c0bf7ea3ec2857a9f9: MemRowSet was empty: no flush needed.
I20240502 01:53:49.643828 20934 consensus_queue.cc:1035] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:49.645460 20762 consensus_queue.cc:1035] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:49.649457 17328 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=2c26265a42364c878d9c982a6cbb7f48] alter complete (version 1)
I20240502 01:53:49.652745 20762 consensus_queue.cc:1035] T e070da07e2214744b3dacd1b839310b9 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:49.662568 20974 tablet.cc:1722] T 6d119e18319943148f879494ee7b07ed P f19cc6cb1e76460aa6711e49a5775091: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:53:49.663251 20974 tablet.cc:1620] T 6d119e18319943148f879494ee7b07ed P f19cc6cb1e76460aa6711e49a5775091: MemRowSet was empty: no flush needed.
I20240502 01:53:49.667512 17328 catalog_manager.cc:5576] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411) does not have the latest schema for tablet 6d119e18319943148f879494ee7b07ed (table rebalancer_dynamic_table [id=2c26265a42364c878d9c982a6cbb7f48]). Expected version 1 got 0
I20240502 01:53:49.675472 20934 consensus_queue.cc:1035] T e070da07e2214744b3dacd1b839310b9 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:49.684631 17328 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_extra_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:49.697762 17455 tablet_service.cc:1467] Processing CreateTablet for tablet 5e8e78bd417d46cdb166087dbfa86446 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=dca50b50b02e4a8c8b90e0de6f305bd8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:49.698254 17455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e8e78bd417d46cdb166087dbfa86446. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:49.699352 20346 tablet_service.cc:1467] Processing CreateTablet for tablet e42126e16cda4d908b00e2e4a6807d6b (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=dca50b50b02e4a8c8b90e0de6f305bd8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:49.699465 20345 tablet_service.cc:1467] Processing CreateTablet for tablet a212c17ba64842c99b3d6a78a3b7cc8a (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=dca50b50b02e4a8c8b90e0de6f305bd8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:49.699843 20346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e42126e16cda4d908b00e2e4a6807d6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:49.699896 20212 tablet_service.cc:1467] Processing CreateTablet for tablet e42126e16cda4d908b00e2e4a6807d6b (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=dca50b50b02e4a8c8b90e0de6f305bd8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:49.700098 20345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a212c17ba64842c99b3d6a78a3b7cc8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:49.700342 20212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e42126e16cda4d908b00e2e4a6807d6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:49.701684 20347 tablet_service.cc:1467] Processing CreateTablet for tablet 5e8e78bd417d46cdb166087dbfa86446 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=dca50b50b02e4a8c8b90e0de6f305bd8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:49.702164 20347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e8e78bd417d46cdb166087dbfa86446. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:49.705307 17456 tablet_service.cc:1467] Processing CreateTablet for tablet e42126e16cda4d908b00e2e4a6807d6b (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=dca50b50b02e4a8c8b90e0de6f305bd8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:49.705699 17456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e42126e16cda4d908b00e2e4a6807d6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:49.709460 20478 tablet_service.cc:1467] Processing CreateTablet for tablet 5e8e78bd417d46cdb166087dbfa86446 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=dca50b50b02e4a8c8b90e0de6f305bd8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:49.709859 20478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e8e78bd417d46cdb166087dbfa86446. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:49.710254 17454 tablet_service.cc:1467] Processing CreateTablet for tablet a212c17ba64842c99b3d6a78a3b7cc8a (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=dca50b50b02e4a8c8b90e0de6f305bd8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:49.713624 17454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a212c17ba64842c99b3d6a78a3b7cc8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:49.719468 20982 tablet_bootstrap.cc:492] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:53:49.719878 20211 tablet_service.cc:1467] Processing CreateTablet for tablet a212c17ba64842c99b3d6a78a3b7cc8a (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=dca50b50b02e4a8c8b90e0de6f305bd8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:49.720304 20211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a212c17ba64842c99b3d6a78a3b7cc8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:49.722038 20982 tablet_bootstrap.cc:654] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:49.730197 20982 tablet_bootstrap.cc:492] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:53:49.730373 20982 ts_tablet_manager.cc:1367] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20240502 01:53:49.731451 20983 tablet_bootstrap.cc:492] T 5e8e78bd417d46cdb166087dbfa86446 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:53:49.732574 20982 raft_consensus.cc:348] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:49.732825 20982 raft_consensus.cc:374] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:49.732942 20982 raft_consensus.cc:729] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:53:49.733371 20982 consensus_queue.cc:260] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:49.734583 20983 tablet_bootstrap.cc:654] T 5e8e78bd417d46cdb166087dbfa86446 P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:49.735417 20982 ts_tablet_manager.cc:1398] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 01:53:49.735937 20982 tablet_bootstrap.cc:492] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:53:49.737425 20982 tablet_bootstrap.cc:654] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:49.748384 20984 tablet_bootstrap.cc:492] T a212c17ba64842c99b3d6a78a3b7cc8a P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:49.748863 20986 tablet_bootstrap.cc:492] T a212c17ba64842c99b3d6a78a3b7cc8a P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:53:49.749172 20982 tablet_bootstrap.cc:492] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:53:49.749378 20982 ts_tablet_manager.cc:1367] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.004s
I20240502 01:53:49.749570 20983 tablet_bootstrap.cc:492] T 5e8e78bd417d46cdb166087dbfa86446 P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:53:49.749787 20983 ts_tablet_manager.cc:1367] T 5e8e78bd417d46cdb166087dbfa86446 P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s
I20240502 01:53:49.750175 20982 raft_consensus.cc:348] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:49.750447 20982 raft_consensus.cc:374] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:49.750555 20982 raft_consensus.cc:729] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:53:49.750809 20983 raft_consensus.cc:348] T 5e8e78bd417d46cdb166087dbfa86446 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:49.750867 20982 consensus_queue.cc:260] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:49.751060 20983 raft_consensus.cc:374] T 5e8e78bd417d46cdb166087dbfa86446 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:49.751194 20983 raft_consensus.cc:729] T 5e8e78bd417d46cdb166087dbfa86446 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:53:49.751475 20983 consensus_queue.cc:260] T 5e8e78bd417d46cdb166087dbfa86446 P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:49.752647 20983 ts_tablet_manager.cc:1398] T 5e8e78bd417d46cdb166087dbfa86446 P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:49.754699 20982 ts_tablet_manager.cc:1398] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 01:53:49.756210 20982 tablet_bootstrap.cc:492] T 5e8e78bd417d46cdb166087dbfa86446 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:53:49.757911 20982 tablet_bootstrap.cc:654] T 5e8e78bd417d46cdb166087dbfa86446 P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:49.760578 20984 tablet_bootstrap.cc:654] T a212c17ba64842c99b3d6a78a3b7cc8a P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:49.763168 17328 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:53:49.764823 20986 tablet_bootstrap.cc:654] T a212c17ba64842c99b3d6a78a3b7cc8a P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:49.779376 20984 tablet_bootstrap.cc:492] T a212c17ba64842c99b3d6a78a3b7cc8a P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:49.779633 20984 ts_tablet_manager.cc:1367] T a212c17ba64842c99b3d6a78a3b7cc8a P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.000s
I20240502 01:53:49.780788 20984 raft_consensus.cc:348] T a212c17ba64842c99b3d6a78a3b7cc8a P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:49.781132 20984 raft_consensus.cc:374] T a212c17ba64842c99b3d6a78a3b7cc8a P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:49.781282 20984 raft_consensus.cc:729] T a212c17ba64842c99b3d6a78a3b7cc8a P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:49.781599 20984 consensus_queue.cc:260] T a212c17ba64842c99b3d6a78a3b7cc8a P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:49.782334 20984 ts_tablet_manager.cc:1398] T a212c17ba64842c99b3d6a78a3b7cc8a P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:49.782526 20972 tablet.cc:1722] T 6d119e18319943148f879494ee7b07ed P 3fceccefba274b1dbbd8216af8b0bc29: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:53:49.783013 20984 tablet_bootstrap.cc:492] T e42126e16cda4d908b00e2e4a6807d6b P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:49.783071 20972 tablet.cc:1620] T 6d119e18319943148f879494ee7b07ed P 3fceccefba274b1dbbd8216af8b0bc29: MemRowSet was empty: no flush needed.
I20240502 01:53:49.784297 20986 tablet_bootstrap.cc:492] T a212c17ba64842c99b3d6a78a3b7cc8a P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:53:49.784511 20986 ts_tablet_manager.cc:1367] T a212c17ba64842c99b3d6a78a3b7cc8a P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.002s
I20240502 01:53:49.784945 20984 tablet_bootstrap.cc:654] T e42126e16cda4d908b00e2e4a6807d6b P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:49.785409 20986 raft_consensus.cc:348] T a212c17ba64842c99b3d6a78a3b7cc8a P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:49.785701 20986 raft_consensus.cc:374] T a212c17ba64842c99b3d6a78a3b7cc8a P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:49.785830 20986 raft_consensus.cc:729] T a212c17ba64842c99b3d6a78a3b7cc8a P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:53:49.786146 20973 tablet.cc:1722] T 6d119e18319943148f879494ee7b07ed P 6a36c952c51542c0bf7ea3ec2857a9f9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:53:49.786638 20973 tablet.cc:1620] T 6d119e18319943148f879494ee7b07ed P 6a36c952c51542c0bf7ea3ec2857a9f9: MemRowSet was empty: no flush needed.
I20240502 01:53:49.786587 20986 consensus_queue.cc:260] T a212c17ba64842c99b3d6a78a3b7cc8a P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:49.791060 20986 ts_tablet_manager.cc:1398] T a212c17ba64842c99b3d6a78a3b7cc8a P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:53:49.790984 20974 tablet.cc:1722] T 6d119e18319943148f879494ee7b07ed P f19cc6cb1e76460aa6711e49a5775091: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:53:49.791527 20974 tablet.cc:1620] T 6d119e18319943148f879494ee7b07ed P f19cc6cb1e76460aa6711e49a5775091: MemRowSet was empty: no flush needed.
I20240502 01:53:49.794438 20986 tablet_bootstrap.cc:492] T e42126e16cda4d908b00e2e4a6807d6b P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:53:49.796023 20986 tablet_bootstrap.cc:654] T e42126e16cda4d908b00e2e4a6807d6b P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:49.803006 20984 tablet_bootstrap.cc:492] T e42126e16cda4d908b00e2e4a6807d6b P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:49.803260 20984 ts_tablet_manager.cc:1367] T e42126e16cda4d908b00e2e4a6807d6b P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.005s
I20240502 01:53:49.804163 20984 raft_consensus.cc:348] T e42126e16cda4d908b00e2e4a6807d6b P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:49.804455 20984 raft_consensus.cc:374] T e42126e16cda4d908b00e2e4a6807d6b P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:49.804699 20984 raft_consensus.cc:729] T e42126e16cda4d908b00e2e4a6807d6b P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:49.805178 20984 consensus_queue.cc:260] T e42126e16cda4d908b00e2e4a6807d6b P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:49.806003 20984 ts_tablet_manager.cc:1398] T e42126e16cda4d908b00e2e4a6807d6b P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:53:49.806321 17327 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=2c26265a42364c878d9c982a6cbb7f48] alter complete (version 2)
I20240502 01:53:49.806378 20984 tablet_bootstrap.cc:492] T 5e8e78bd417d46cdb166087dbfa86446 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:49.808152 20986 tablet_bootstrap.cc:492] T e42126e16cda4d908b00e2e4a6807d6b P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:53:49.808380 20986 ts_tablet_manager.cc:1367] T e42126e16cda4d908b00e2e4a6807d6b P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.004s
I20240502 01:53:49.810034 20984 tablet_bootstrap.cc:654] T 5e8e78bd417d46cdb166087dbfa86446 P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:49.814229 20986 raft_consensus.cc:348] T e42126e16cda4d908b00e2e4a6807d6b P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:49.814540 20986 raft_consensus.cc:374] T e42126e16cda4d908b00e2e4a6807d6b P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:49.814657 20986 raft_consensus.cc:729] T e42126e16cda4d908b00e2e4a6807d6b P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:53:49.814973 20986 consensus_queue.cc:260] T e42126e16cda4d908b00e2e4a6807d6b P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:49.816444 20986 ts_tablet_manager.cc:1398] T e42126e16cda4d908b00e2e4a6807d6b P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:53:49.817647 20982 tablet_bootstrap.cc:492] T 5e8e78bd417d46cdb166087dbfa86446 P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:53:49.817687 20984 tablet_bootstrap.cc:492] T 5e8e78bd417d46cdb166087dbfa86446 P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:49.817823 20984 ts_tablet_manager.cc:1367] T 5e8e78bd417d46cdb166087dbfa86446 P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.003s
I20240502 01:53:49.818483 20984 raft_consensus.cc:348] T 5e8e78bd417d46cdb166087dbfa86446 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:49.818696 20984 raft_consensus.cc:374] T 5e8e78bd417d46cdb166087dbfa86446 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:49.819037 20984 raft_consensus.cc:729] T 5e8e78bd417d46cdb166087dbfa86446 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:49.819355 20984 consensus_queue.cc:260] T 5e8e78bd417d46cdb166087dbfa86446 P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:49.820104 20984 ts_tablet_manager.cc:1398] T 5e8e78bd417d46cdb166087dbfa86446 P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:53:49.821033 17334 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240502 01:53:49.821254 17334 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240502 01:53:49.821506 20982 ts_tablet_manager.cc:1367] T 5e8e78bd417d46cdb166087dbfa86446 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.065s user 0.010s sys 0.000s
I20240502 01:53:49.823210 20982 raft_consensus.cc:348] T 5e8e78bd417d46cdb166087dbfa86446 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:49.823907 20982 raft_consensus.cc:374] T 5e8e78bd417d46cdb166087dbfa86446 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:49.824163 20982 raft_consensus.cc:729] T 5e8e78bd417d46cdb166087dbfa86446 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:53:49.824570 20982 consensus_queue.cc:260] T 5e8e78bd417d46cdb166087dbfa86446 P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:49.825572 17334 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet a3ccdd1dffe846508262911fe0c22261
I20240502 01:53:49.826531 17454 tablet_service.cc:1514] Processing DeleteTablet for tablet a3ccdd1dffe846508262911fe0c22261 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:49 UTC) from {username='slave'} at 127.0.0.1:54520
I20240502 01:53:49.826745 17334 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet 4cd692f7f2ca4dce901b7b2d2dc3a726
I20240502 01:53:49.826783 20982 ts_tablet_manager.cc:1398] T 5e8e78bd417d46cdb166087dbfa86446 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:53:49.826884 20741 tablet_replica.cc:330] T a3ccdd1dffe846508262911fe0c22261 P 4da690c0fa13407a85c0689ce8de9ade: stopping tablet replica
I20240502 01:53:49.827093 20741 raft_consensus.cc:2229] T a3ccdd1dffe846508262911fe0c22261 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:49.827430 17334 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet 1d2be05ef3b045a6bbd2de1798ef0460
I20240502 01:53:49.827448 20741 raft_consensus.cc:2258] T a3ccdd1dffe846508262911fe0c22261 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:49.828130 20741 ts_tablet_manager.cc:1875] T a3ccdd1dffe846508262911fe0c22261 P 4da690c0fa13407a85c0689ce8de9ade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:49.828611 17585 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cd692f7f2ca4dce901b7b2d2dc3a726 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:49 UTC) from {username='slave'} at 127.0.0.1:55232
I20240502 01:53:49.828668 17587 tablet_service.cc:1514] Processing DeleteTablet for tablet a3ccdd1dffe846508262911fe0c22261 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:49 UTC) from {username='slave'} at 127.0.0.1:55232
I20240502 01:53:49.828912 20737 tablet_replica.cc:330] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 3fceccefba274b1dbbd8216af8b0bc29: stopping tablet replica
I20240502 01:53:49.829139 20737 raft_consensus.cc:2229] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:49.829210 17716 tablet_service.cc:1514] Processing DeleteTablet for tablet a3ccdd1dffe846508262911fe0c22261 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:49 UTC) from {username='slave'} at 127.0.0.1:52198
I20240502 01:53:49.829497 20892 tablet_replica.cc:330] T a3ccdd1dffe846508262911fe0c22261 P dfa1152987644bd781b46c7e42a7a685: stopping tablet replica
I20240502 01:53:49.829485 20737 raft_consensus.cc:2258] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:49.829725 20892 raft_consensus.cc:2229] T a3ccdd1dffe846508262911fe0c22261 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:49.829995 20892 raft_consensus.cc:2258] T a3ccdd1dffe846508262911fe0c22261 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:49.830219 20737 ts_tablet_manager.cc:1875] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:49.830706 20892 ts_tablet_manager.cc:1875] T a3ccdd1dffe846508262911fe0c22261 P dfa1152987644bd781b46c7e42a7a685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:49.832643 20813 raft_consensus.cc:482] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:49.832866 17716 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d2be05ef3b045a6bbd2de1798ef0460 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:49 UTC) from {username='slave'} at 127.0.0.1:52198
I20240502 01:53:49.832778 20813 raft_consensus.cc:504] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:49.833180 17455 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d2be05ef3b045a6bbd2de1798ef0460 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:49 UTC) from {username='slave'} at 127.0.0.1:54520
I20240502 01:53:49.833477 17454 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cd692f7f2ca4dce901b7b2d2dc3a726 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:49 UTC) from {username='slave'} at 127.0.0.1:54520
I20240502 01:53:49.833624 20813 leader_election.cc:288] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:49.834534 17846 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cd692f7f2ca4dce901b7b2d2dc3a726 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:49 UTC) from {username='slave'} at 127.0.0.1:47226
I20240502 01:53:49.834879 17845 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d2be05ef3b045a6bbd2de1798ef0460 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:49 UTC) from {username='slave'} at 127.0.0.1:47226
I20240502 01:53:49.836030 20741 ts_tablet_manager.cc:1888] T a3ccdd1dffe846508262911fe0c22261 P 4da690c0fa13407a85c0689ce8de9ade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:49.836151 20741 log.cc:1196] T a3ccdd1dffe846508262911fe0c22261 P 4da690c0fa13407a85c0689ce8de9ade: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/a3ccdd1dffe846508262911fe0c22261
I20240502 01:53:49.836576 20892 ts_tablet_manager.cc:1888] T a3ccdd1dffe846508262911fe0c22261 P dfa1152987644bd781b46c7e42a7a685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:49.836648 20741 ts_tablet_manager.cc:1909] T a3ccdd1dffe846508262911fe0c22261 P 4da690c0fa13407a85c0689ce8de9ade: Deleting consensus metadata
I20240502 01:53:49.836820 20892 log.cc:1196] T a3ccdd1dffe846508262911fe0c22261 P dfa1152987644bd781b46c7e42a7a685: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/wals/a3ccdd1dffe846508262911fe0c22261
I20240502 01:53:49.837335 20892 ts_tablet_manager.cc:1909] T a3ccdd1dffe846508262911fe0c22261 P dfa1152987644bd781b46c7e42a7a685: Deleting consensus metadata
I20240502 01:53:49.837687 20741 tablet_replica.cc:330] T 1d2be05ef3b045a6bbd2de1798ef0460 P 4da690c0fa13407a85c0689ce8de9ade: stopping tablet replica
I20240502 01:53:49.838464 17321 catalog_manager.cc:4830] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): tablet a3ccdd1dffe846508262911fe0c22261 (table rebalance_test_table_8 [id=3f381b46ce4c4598900dada557836f91]) successfully deleted
I20240502 01:53:49.838781 20741 raft_consensus.cc:2229] T 1d2be05ef3b045a6bbd2de1798ef0460 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:49.838766 20232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e42126e16cda4d908b00e2e4a6807d6b" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:53:49.839004 20232 raft_consensus.cc:2454] T e42126e16cda4d908b00e2e4a6807d6b P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 0.
I20240502 01:53:49.839197 20741 raft_consensus.cc:2258] T 1d2be05ef3b045a6bbd2de1798ef0460 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:49.839769 17321 catalog_manager.cc:4830] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): tablet a3ccdd1dffe846508262911fe0c22261 (table rebalance_test_table_8 [id=3f381b46ce4c4598900dada557836f91]) successfully deleted
I20240502 01:53:49.840008 20741 ts_tablet_manager.cc:1875] T 1d2be05ef3b045a6bbd2de1798ef0460 P 4da690c0fa13407a85c0689ce8de9ade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:49.840142 20303 leader_election.cc:302] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:53:49.840567 20813 raft_consensus.cc:2781] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:49.840675 20813 raft_consensus.cc:482] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:49.840750 20813 raft_consensus.cc:3037] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:49.843416 20813 raft_consensus.cc:504] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:49.844659 20813 leader_election.cc:288] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Requested vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:49.844980 20892 tablet_replica.cc:330] T 1d2be05ef3b045a6bbd2de1798ef0460 P dfa1152987644bd781b46c7e42a7a685: stopping tablet replica
I20240502 01:53:49.845225 20892 raft_consensus.cc:2229] T 1d2be05ef3b045a6bbd2de1798ef0460 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:49.845638 20892 raft_consensus.cc:2258] T 1d2be05ef3b045a6bbd2de1798ef0460 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:49.846351 20892 ts_tablet_manager.cc:1875] T 1d2be05ef3b045a6bbd2de1798ef0460 P dfa1152987644bd781b46c7e42a7a685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:49.847741 20232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e42126e16cda4d908b00e2e4a6807d6b" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:53:49.847934 20232 raft_consensus.cc:3037] T e42126e16cda4d908b00e2e4a6807d6b P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:49.850656 20232 raft_consensus.cc:2454] T e42126e16cda4d908b00e2e4a6807d6b P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 1.
I20240502 01:53:49.851208 20303 leader_election.cc:302] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:53:49.851523 20813 raft_consensus.cc:2781] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:49.851661 20813 raft_consensus.cc:686] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Becoming Leader. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Running, Role: LEADER
I20240502 01:53:49.851933 20813 consensus_queue.cc:237] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:49.852311 20741 ts_tablet_manager.cc:1888] T 1d2be05ef3b045a6bbd2de1798ef0460 P 4da690c0fa13407a85c0689ce8de9ade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:49.852449 20741 log.cc:1196] T 1d2be05ef3b045a6bbd2de1798ef0460 P 4da690c0fa13407a85c0689ce8de9ade: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/1d2be05ef3b045a6bbd2de1798ef0460
I20240502 01:53:49.852877 20741 ts_tablet_manager.cc:1909] T 1d2be05ef3b045a6bbd2de1798ef0460 P 4da690c0fa13407a85c0689ce8de9ade: Deleting consensus metadata
I20240502 01:53:49.853960 17321 catalog_manager.cc:4830] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): tablet 1d2be05ef3b045a6bbd2de1798ef0460 (table rebalance_test_table_8 [id=3f381b46ce4c4598900dada557836f91]) successfully deleted
I20240502 01:53:49.854357 20741 tablet_replica.cc:330] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 4da690c0fa13407a85c0689ce8de9ade: stopping tablet replica
I20240502 01:53:49.854588 20741 raft_consensus.cc:2229] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:49.854859 20741 raft_consensus.cc:2258] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:49.855103 20892 ts_tablet_manager.cc:1888] T 1d2be05ef3b045a6bbd2de1798ef0460 P dfa1152987644bd781b46c7e42a7a685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:49.855232 20892 log.cc:1196] T 1d2be05ef3b045a6bbd2de1798ef0460 P dfa1152987644bd781b46c7e42a7a685: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/wals/1d2be05ef3b045a6bbd2de1798ef0460
I20240502 01:53:49.855664 20892 ts_tablet_manager.cc:1909] T 1d2be05ef3b045a6bbd2de1798ef0460 P dfa1152987644bd781b46c7e42a7a685: Deleting consensus metadata
I20240502 01:53:49.855703 20741 ts_tablet_manager.cc:1875] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 4da690c0fa13407a85c0689ce8de9ade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:49.856742 17321 catalog_manager.cc:4830] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): tablet 1d2be05ef3b045a6bbd2de1798ef0460 (table rebalance_test_table_8 [id=3f381b46ce4c4598900dada557836f91]) successfully deleted
I20240502 01:53:49.863752 20741 ts_tablet_manager.cc:1888] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 4da690c0fa13407a85c0689ce8de9ade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:49.863839 20871 raft_consensus.cc:482] T 5e8e78bd417d46cdb166087dbfa86446 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:49.864020 20741 log.cc:1196] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 4da690c0fa13407a85c0689ce8de9ade: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/4cd692f7f2ca4dce901b7b2d2dc3a726
I20240502 01:53:49.864012 20871 raft_consensus.cc:504] T 5e8e78bd417d46cdb166087dbfa86446 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:49.864634 20741 ts_tablet_manager.cc:1909] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 4da690c0fa13407a85c0689ce8de9ade: Deleting consensus metadata
I20240502 01:53:49.865062 20871 leader_election.cc:288] T 5e8e78bd417d46cdb166087dbfa86446 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:53:49.872038 17321 catalog_manager.cc:4830] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): tablet 4cd692f7f2ca4dce901b7b2d2dc3a726 (table rebalance_test_table_8 [id=3f381b46ce4c4598900dada557836f91]) successfully deleted
I20240502 01:53:49.872829 20737 ts_tablet_manager.cc:1888] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 3fceccefba274b1dbbd8216af8b0bc29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:49.873075 20737 log.cc:1196] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/wals/4cd692f7f2ca4dce901b7b2d2dc3a726
I20240502 01:53:49.873584 20737 ts_tablet_manager.cc:1909] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting consensus metadata
I20240502 01:53:49.874768 17319 catalog_manager.cc:4830] TS 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): tablet 4cd692f7f2ca4dce901b7b2d2dc3a726 (table rebalance_test_table_8 [id=3f381b46ce4c4598900dada557836f91]) successfully deleted
I20240502 01:53:49.874929 20737 tablet_replica.cc:330] T a3ccdd1dffe846508262911fe0c22261 P 3fceccefba274b1dbbd8216af8b0bc29: stopping tablet replica
I20240502 01:53:49.875303 20737 raft_consensus.cc:2229] T a3ccdd1dffe846508262911fe0c22261 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:49.888000 20737 raft_consensus.cc:2258] T a3ccdd1dffe846508262911fe0c22261 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:49.889003 20737 ts_tablet_manager.cc:1875] T a3ccdd1dffe846508262911fe0c22261 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:49.894315 20996 tablet_replica.cc:330] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P bb31bceba76d466db24b9341c04900c3: stopping tablet replica
I20240502 01:53:49.894649 20996 raft_consensus.cc:2229] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:49.894977 20996 raft_consensus.cc:2258] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:49.899746 20996 ts_tablet_manager.cc:1875] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P bb31bceba76d466db24b9341c04900c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:49.908759 17334 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:53:49.916754 20768 raft_consensus.cc:482] T a212c17ba64842c99b3d6a78a3b7cc8a P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:49.916944 20768 raft_consensus.cc:504] T a212c17ba64842c99b3d6a78a3b7cc8a P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:49.917784 20768 leader_election.cc:288] T a212c17ba64842c99b3d6a78a3b7cc8a P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:49.922437 20498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e8e78bd417d46cdb166087dbfa86446" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
I20240502 01:53:49.922677 20498 raft_consensus.cc:2454] T 5e8e78bd417d46cdb166087dbfa86446 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 0.
I20240502 01:53:49.923278 17412 leader_election.cc:302] T 5e8e78bd417d46cdb166087dbfa86446 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 4da690c0fa13407a85c0689ce8de9ade; no voters:
I20240502 01:53:49.923667 20871 raft_consensus.cc:2781] T 5e8e78bd417d46cdb166087dbfa86446 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:49.923807 20871 raft_consensus.cc:482] T 5e8e78bd417d46cdb166087dbfa86446 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:49.923902 20871 raft_consensus.cc:3037] T 5e8e78bd417d46cdb166087dbfa86446 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:49.925765 20972 tablet.cc:1722] T 6d119e18319943148f879494ee7b07ed P 3fceccefba274b1dbbd8216af8b0bc29: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:53:49.926762 20871 raft_consensus.cc:504] T 5e8e78bd417d46cdb166087dbfa86446 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:49.927879 20871 leader_election.cc:288] T 5e8e78bd417d46cdb166087dbfa86446 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Requested vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:53:49.928687 20498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e8e78bd417d46cdb166087dbfa86446" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
I20240502 01:53:49.928893 20498 raft_consensus.cc:3037] T 5e8e78bd417d46cdb166087dbfa86446 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:49.929189 20877 raft_consensus.cc:482] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:49.929360 20877 raft_consensus.cc:504] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:49.929973 20877 leader_election.cc:288] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:53:49.930946 20974 tablet.cc:1722] T 6d119e18319943148f879494ee7b07ed P f19cc6cb1e76460aa6711e49a5775091: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:53:49.931416 20498 raft_consensus.cc:2454] T 5e8e78bd417d46cdb166087dbfa86446 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
I20240502 01:53:49.932343 17412 leader_election.cc:302] T 5e8e78bd417d46cdb166087dbfa86446 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 4da690c0fa13407a85c0689ce8de9ade; no voters:
I20240502 01:53:49.932695 20871 raft_consensus.cc:2781] T 5e8e78bd417d46cdb166087dbfa86446 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:49.937311 20232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a212c17ba64842c99b3d6a78a3b7cc8a" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:53:49.937557 20232 raft_consensus.cc:2454] T a212c17ba64842c99b3d6a78a3b7cc8a P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 0.
I20240502 01:53:49.938210 20303 leader_election.cc:302] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:53:49.939139 20765 raft_consensus.cc:2781] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:49.939251 20765 raft_consensus.cc:482] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:49.939353 20765 raft_consensus.cc:3037] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:49.942020 20765 raft_consensus.cc:504] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:49.942663 20765 leader_election.cc:288] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Requested vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:53:49.944540 17334 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table_new_name [id=2c26265a42364c878d9c982a6cbb7f48] alter complete (version 3)
I20240502 01:53:49.949573 20232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a212c17ba64842c99b3d6a78a3b7cc8a" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:53:49.949821 20232 raft_consensus.cc:3037] T a212c17ba64842c99b3d6a78a3b7cc8a P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:49.952414 20232 raft_consensus.cc:2454] T a212c17ba64842c99b3d6a78a3b7cc8a P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 1.
I20240502 01:53:49.952867 20303 leader_election.cc:302] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:53:50.005208 20973 tablet.cc:1722] T 6d119e18319943148f879494ee7b07ed P 6a36c952c51542c0bf7ea3ec2857a9f9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:53:50.015700 17474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e42126e16cda4d908b00e2e4a6807d6b" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:50.016027 17474 raft_consensus.cc:2454] T e42126e16cda4d908b00e2e4a6807d6b P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 0.
I20240502 01:53:50.016840 21009 raft_consensus.cc:2781] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:50.019038 17474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e42126e16cda4d908b00e2e4a6807d6b" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:53:50.019383 17474 raft_consensus.cc:3037] T e42126e16cda4d908b00e2e4a6807d6b P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:50.022774 17474 raft_consensus.cc:2454] T e42126e16cda4d908b00e2e4a6807d6b P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 1.
I20240502 01:53:50.023798 17474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a212c17ba64842c99b3d6a78a3b7cc8a" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:50.024066 17474 raft_consensus.cc:2454] T a212c17ba64842c99b3d6a78a3b7cc8a P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 0.
I20240502 01:53:50.024469 21010 raft_consensus.cc:482] T 5e8e78bd417d46cdb166087dbfa86446 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:50.025180 21010 raft_consensus.cc:504] T 5e8e78bd417d46cdb166087dbfa86446 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:50.024650 21009 raft_consensus.cc:686] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Becoming Leader. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Running, Role: LEADER
I20240502 01:53:50.026805 21009 consensus_queue.cc:237] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:50.029513 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e8e78bd417d46cdb166087dbfa86446" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:50.030499 17474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a212c17ba64842c99b3d6a78a3b7cc8a" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:53:50.030740 17474 raft_consensus.cc:3037] T a212c17ba64842c99b3d6a78a3b7cc8a P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:50.035477 17474 raft_consensus.cc:2454] T a212c17ba64842c99b3d6a78a3b7cc8a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 1.
I20240502 01:53:50.041646 21010 leader_election.cc:288] T 5e8e78bd417d46cdb166087dbfa86446 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:53:50.053185 17334 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:53:50.053457 17334 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:53:50.057693 17334 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet 6d119e18319943148f879494ee7b07ed
I20240502 01:53:50.059263 20609 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d119e18319943148f879494ee7b07ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:50 UTC) from {username='slave'} at 127.0.0.1:33698
I20240502 01:53:50.059695 17587 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d119e18319943148f879494ee7b07ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:50 UTC) from {username='slave'} at 127.0.0.1:55232
W20240502 01:53:50.069489 17409 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.42.66:38531, user_credentials={real_user=slave}} blocked reactor thread for 133253us
I20240502 01:53:50.073660 17476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a212c17ba64842c99b3d6a78a3b7cc8a" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:50.073932 17476 raft_consensus.cc:2379] T a212c17ba64842c99b3d6a78a3b7cc8a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdc2ac372492454c85387589a91f63af in current term 1: Already voted for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in this term.
I20240502 01:53:50.074476 20347 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d119e18319943148f879494ee7b07ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:50 UTC) from {username='slave'} at 127.0.0.1:40196
I20240502 01:53:50.074528 20366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e8e78bd417d46cdb166087dbfa86446" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:53:50.074755 20366 raft_consensus.cc:2454] T 5e8e78bd417d46cdb166087dbfa86446 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 0.
I20240502 01:53:50.075735 20367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e8e78bd417d46cdb166087dbfa86446" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
I20240502 01:53:50.075922 21014 tablet_replica.cc:330] T 6d119e18319943148f879494ee7b07ed P 6a36c952c51542c0bf7ea3ec2857a9f9: stopping tablet replica
I20240502 01:53:50.075985 20367 raft_consensus.cc:3037] T 5e8e78bd417d46cdb166087dbfa86446 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:50.076445 21014 raft_consensus.cc:2229] T 6d119e18319943148f879494ee7b07ed P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:50.076795 21014 raft_consensus.cc:2258] T 6d119e18319943148f879494ee7b07ed P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:50.078395 21014 ts_tablet_manager.cc:1875] T 6d119e18319943148f879494ee7b07ed P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:50.079074 20367 raft_consensus.cc:2454] T 5e8e78bd417d46cdb166087dbfa86446 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
I20240502 01:53:50.085690 21014 ts_tablet_manager.cc:1888] T 6d119e18319943148f879494ee7b07ed P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:53:50.085894 21014 log.cc:1196] T 6d119e18319943148f879494ee7b07ed P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal/wals/6d119e18319943148f879494ee7b07ed
I20240502 01:53:50.086478 21014 ts_tablet_manager.cc:1909] T 6d119e18319943148f879494ee7b07ed P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting consensus metadata
I20240502 01:53:50.088761 17321 catalog_manager.cc:4830] TS 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): tablet 6d119e18319943148f879494ee7b07ed (table rebalancer_dynamic_table_new_name [id=2c26265a42364c878d9c982a6cbb7f48]) successfully deleted
I20240502 01:53:50.094146 21013 tablet_replica.cc:330] T 6d119e18319943148f879494ee7b07ed P f19cc6cb1e76460aa6711e49a5775091: stopping tablet replica
I20240502 01:53:50.094496 21013 raft_consensus.cc:2229] T 6d119e18319943148f879494ee7b07ed P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:50.096549 21013 raft_consensus.cc:2258] T 6d119e18319943148f879494ee7b07ed P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:50.097762 21013 ts_tablet_manager.cc:1875] T 6d119e18319943148f879494ee7b07ed P f19cc6cb1e76460aa6711e49a5775091: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:50.107967 20367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a212c17ba64842c99b3d6a78a3b7cc8a" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:53:50.108660 20167 leader_election.cc:302] T a212c17ba64842c99b3d6a78a3b7cc8a P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdc2ac372492454c85387589a91f63af; no voters: 4da690c0fa13407a85c0689ce8de9ade, 6a36c952c51542c0bf7ea3ec2857a9f9
I20240502 01:53:50.109130 20768 raft_consensus.cc:2726] T a212c17ba64842c99b3d6a78a3b7cc8a P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:53:50.109822 20871 raft_consensus.cc:686] T 5e8e78bd417d46cdb166087dbfa86446 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Becoming Leader. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Running, Role: LEADER
I20240502 01:53:50.109988 17334 catalog_manager.cc:5484] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70). New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } }
I20240502 01:53:50.110162 20871 consensus_queue.cc:237] T 5e8e78bd417d46cdb166087dbfa86446 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:50.087978 20737 ts_tablet_manager.cc:1888] T a3ccdd1dffe846508262911fe0c22261 P 3fceccefba274b1dbbd8216af8b0bc29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:50.112461 20498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e8e78bd417d46cdb166087dbfa86446" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
I20240502 01:53:50.112653 20737 log.cc:1196] T a3ccdd1dffe846508262911fe0c22261 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/wals/a3ccdd1dffe846508262911fe0c22261
I20240502 01:53:50.112759 20498 raft_consensus.cc:2379] T 5e8e78bd417d46cdb166087dbfa86446 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in current term 1: Already voted for candidate 4da690c0fa13407a85c0689ce8de9ade in this term.
I20240502 01:53:50.113876 20737 ts_tablet_manager.cc:1909] T a3ccdd1dffe846508262911fe0c22261 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting consensus metadata
I20240502 01:53:50.115058 17319 catalog_manager.cc:4830] TS 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): tablet a3ccdd1dffe846508262911fe0c22261 (table rebalance_test_table_8 [id=3f381b46ce4c4598900dada557836f91]) successfully deleted
I20240502 01:53:50.116791 17334 catalog_manager.cc:5484] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70). New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:50.117867 20302 leader_election.cc:302] T 5e8e78bd417d46cdb166087dbfa86446 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9; no voters: 2f306549407b4bb281ccf2994035e4f1, 4da690c0fa13407a85c0689ce8de9ade
I20240502 01:53:50.118158 20813 raft_consensus.cc:2726] T 5e8e78bd417d46cdb166087dbfa86446 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:53:50.119612 17327 catalog_manager.cc:5484] T 5e8e78bd417d46cdb166087dbfa86446 P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: term changed from 0 to 1, leader changed from <none> to 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65). New cstate: current_term: 1 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:50.122680 17864 consensus_queue.cc:237] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
I20240502 01:53:50.123243 20996 ts_tablet_manager.cc:1888] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P bb31bceba76d466db24b9341c04900c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:50.123363 20996 log.cc:1196] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P bb31bceba76d466db24b9341c04900c3: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/wals/4cd692f7f2ca4dce901b7b2d2dc3a726
I20240502 01:53:50.123909 20996 ts_tablet_manager.cc:1909] T 4cd692f7f2ca4dce901b7b2d2dc3a726 P bb31bceba76d466db24b9341c04900c3: Deleting consensus metadata
I20240502 01:53:50.124903 17322 catalog_manager.cc:4830] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): tablet 4cd692f7f2ca4dce901b7b2d2dc3a726 (table rebalance_test_table_8 [id=3f381b46ce4c4598900dada557836f91]) successfully deleted
I20240502 01:53:50.127696 17735 raft_consensus.cc:1261] T 377e804dbeed4913a7d435cfdd1e2d8e P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:50.129380 20747 consensus_queue.cc:1035] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:50.133703 17475 raft_consensus.cc:1261] T 377e804dbeed4913a7d435cfdd1e2d8e P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:50.134341 20747 consensus_queue.cc:1035] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:50.141250 20737 tablet_replica.cc:330] T 6d119e18319943148f879494ee7b07ed P 3fceccefba274b1dbbd8216af8b0bc29: stopping tablet replica
I20240502 01:53:50.141471 20737 raft_consensus.cc:2229] T 6d119e18319943148f879494ee7b07ed P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:50.141813 20737 raft_consensus.cc:2258] T 6d119e18319943148f879494ee7b07ed P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 01:53:50.148198 17803 consensus_peers.cc:485] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3 -> Peer 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): Couldn't send request to peer 2f306549407b4bb281ccf2994035e4f1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 377e804dbeed4913a7d435cfdd1e2d8e. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:50.148414 21013 ts_tablet_manager.cc:1888] T 6d119e18319943148f879494ee7b07ed P f19cc6cb1e76460aa6711e49a5775091: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:53:50.148561 21013 log.cc:1196] T 6d119e18319943148f879494ee7b07ed P f19cc6cb1e76460aa6711e49a5775091: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal/wals/6d119e18319943148f879494ee7b07ed
I20240502 01:53:50.148927 21013 ts_tablet_manager.cc:1909] T 6d119e18319943148f879494ee7b07ed P f19cc6cb1e76460aa6711e49a5775091: Deleting consensus metadata
I20240502 01:53:50.149125 20996 tablet_replica.cc:330] T 1d2be05ef3b045a6bbd2de1798ef0460 P bb31bceba76d466db24b9341c04900c3: stopping tablet replica
I20240502 01:53:50.149319 20996 raft_consensus.cc:2229] T 1d2be05ef3b045a6bbd2de1798ef0460 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:50.149502 20996 raft_consensus.cc:2258] T 1d2be05ef3b045a6bbd2de1798ef0460 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:50.150065 20996 ts_tablet_manager.cc:1875] T 1d2be05ef3b045a6bbd2de1798ef0460 P bb31bceba76d466db24b9341c04900c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:50.157063 20996 ts_tablet_manager.cc:1888] T 1d2be05ef3b045a6bbd2de1798ef0460 P bb31bceba76d466db24b9341c04900c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:50.157243 20996 log.cc:1196] T 1d2be05ef3b045a6bbd2de1798ef0460 P bb31bceba76d466db24b9341c04900c3: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/wals/1d2be05ef3b045a6bbd2de1798ef0460
I20240502 01:53:50.157702 20996 ts_tablet_manager.cc:1909] T 1d2be05ef3b045a6bbd2de1798ef0460 P bb31bceba76d466db24b9341c04900c3: Deleting consensus metadata
I20240502 01:53:50.157657 17473 raft_consensus.cc:2932] T 377e804dbeed4913a7d435cfdd1e2d8e P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:53:50.158857 17322 catalog_manager.cc:4830] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): tablet 1d2be05ef3b045a6bbd2de1798ef0460 (table rebalance_test_table_8 [id=3f381b46ce4c4598900dada557836f91]) successfully deleted
I20240502 01:53:50.160497 17327 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:53:50.162027 17321 catalog_manager.cc:4830] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): tablet 6d119e18319943148f879494ee7b07ed (table rebalancer_dynamic_table_new_name [id=2c26265a42364c878d9c982a6cbb7f48]) successfully deleted
I20240502 01:53:50.156203 20747 raft_consensus.cc:2932] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:53:50.165555 17334 catalog_manager.cc:5484] T 377e804dbeed4913a7d435cfdd1e2d8e P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: config changed from index -1 to 4, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:53:50.169307 20737 ts_tablet_manager.cc:1875] T 6d119e18319943148f879494ee7b07ed P 3fceccefba274b1dbbd8216af8b0bc29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:50.172834 17736 raft_consensus.cc:2932] T 377e804dbeed4913a7d435cfdd1e2d8e P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:53:50.177645 20211 tablet_service.cc:1467] Processing CreateTablet for tablet 06ff53fc9e1e48c69eb7484367fa5c74 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=754c57718ffc4923b82fb0c782be44c9]), partition=
I20240502 01:53:50.177632 20478 tablet_service.cc:1467] Processing CreateTablet for tablet 06ff53fc9e1e48c69eb7484367fa5c74 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=754c57718ffc4923b82fb0c782be44c9]), partition=
I20240502 01:53:50.177978 20478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06ff53fc9e1e48c69eb7484367fa5c74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:50.177978 20211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06ff53fc9e1e48c69eb7484367fa5c74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:50.182777 20609 tablet_service.cc:1467] Processing CreateTablet for tablet 06ff53fc9e1e48c69eb7484367fa5c74 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=754c57718ffc4923b82fb0c782be44c9]), partition=
I20240502 01:53:50.183135 20609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06ff53fc9e1e48c69eb7484367fa5c74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:50.191129 20983 tablet_bootstrap.cc:492] T 06ff53fc9e1e48c69eb7484367fa5c74 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:53:50.192764 20983 tablet_bootstrap.cc:654] T 06ff53fc9e1e48c69eb7484367fa5c74 P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:50.219369 20986 tablet_bootstrap.cc:492] T 06ff53fc9e1e48c69eb7484367fa5c74 P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:53:50.221066 20986 tablet_bootstrap.cc:654] T 06ff53fc9e1e48c69eb7484367fa5c74 P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:50.226469 21031 tablet_bootstrap.cc:492] T 06ff53fc9e1e48c69eb7484367fa5c74 P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:53:50.229259 21031 tablet_bootstrap.cc:654] T 06ff53fc9e1e48c69eb7484367fa5c74 P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:50.237673 21031 tablet_bootstrap.cc:492] T 06ff53fc9e1e48c69eb7484367fa5c74 P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:53:50.237869 21031 ts_tablet_manager.cc:1367] T 06ff53fc9e1e48c69eb7484367fa5c74 P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20240502 01:53:50.238673 21031 raft_consensus.cc:348] T 06ff53fc9e1e48c69eb7484367fa5c74 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:50.239207 21031 raft_consensus.cc:374] T 06ff53fc9e1e48c69eb7484367fa5c74 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:50.239352 21031 raft_consensus.cc:729] T 06ff53fc9e1e48c69eb7484367fa5c74 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:53:50.239650 21031 consensus_queue.cc:260] T 06ff53fc9e1e48c69eb7484367fa5c74 P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:50.240306 21031 ts_tablet_manager.cc:1398] T 06ff53fc9e1e48c69eb7484367fa5c74 P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:53:50.241647 20986 tablet_bootstrap.cc:492] T 06ff53fc9e1e48c69eb7484367fa5c74 P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:53:50.242157 20986 ts_tablet_manager.cc:1367] T 06ff53fc9e1e48c69eb7484367fa5c74 P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.009s
I20240502 01:53:50.243068 20986 raft_consensus.cc:348] T 06ff53fc9e1e48c69eb7484367fa5c74 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:50.243413 20986 raft_consensus.cc:374] T 06ff53fc9e1e48c69eb7484367fa5c74 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:50.243551 20986 raft_consensus.cc:729] T 06ff53fc9e1e48c69eb7484367fa5c74 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:53:50.243849 20986 consensus_queue.cc:260] T 06ff53fc9e1e48c69eb7484367fa5c74 P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:50.244922 20986 ts_tablet_manager.cc:1398] T 06ff53fc9e1e48c69eb7484367fa5c74 P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:53:50.247725 20737 ts_tablet_manager.cc:1888] T 6d119e18319943148f879494ee7b07ed P 3fceccefba274b1dbbd8216af8b0bc29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:53:50.248445 20737 log.cc:1196] T 6d119e18319943148f879494ee7b07ed P 3fceccefba274b1dbbd8216af8b0bc29: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/wals/6d119e18319943148f879494ee7b07ed
I20240502 01:53:50.248801 20737 ts_tablet_manager.cc:1909] T 6d119e18319943148f879494ee7b07ed P 3fceccefba274b1dbbd8216af8b0bc29: Deleting consensus metadata
I20240502 01:53:50.253124 20983 tablet_bootstrap.cc:492] T 06ff53fc9e1e48c69eb7484367fa5c74 P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:53:50.253327 20983 ts_tablet_manager.cc:1367] T 06ff53fc9e1e48c69eb7484367fa5c74 P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.062s user 0.006s sys 0.031s
I20240502 01:53:50.253497 17319 catalog_manager.cc:4830] TS 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): tablet 6d119e18319943148f879494ee7b07ed (table rebalancer_dynamic_table_new_name [id=2c26265a42364c878d9c982a6cbb7f48]) successfully deleted
I20240502 01:53:50.254201 20983 raft_consensus.cc:348] T 06ff53fc9e1e48c69eb7484367fa5c74 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:50.254451 20983 raft_consensus.cc:374] T 06ff53fc9e1e48c69eb7484367fa5c74 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:50.254529 20983 raft_consensus.cc:729] T 06ff53fc9e1e48c69eb7484367fa5c74 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:53:50.254773 20983 consensus_queue.cc:260] T 06ff53fc9e1e48c69eb7484367fa5c74 P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:50.257560 20983 ts_tablet_manager.cc:1398] T 06ff53fc9e1e48c69eb7484367fa5c74 P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:53:50.283280 17735 consensus_queue.cc:237] T 52430bbf347a4c5191edb23e1aa5d385 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } }
I20240502 01:53:50.285737 17475 raft_consensus.cc:1261] T 52430bbf347a4c5191edb23e1aa5d385 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:53:50.286151 17673 consensus_peers.cc:485] T 52430bbf347a4c5191edb23e1aa5d385 P dfa1152987644bd781b46c7e42a7a685 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 52430bbf347a4c5191edb23e1aa5d385. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:50.286458 17607 raft_consensus.cc:1261] T 52430bbf347a4c5191edb23e1aa5d385 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:50.286350 21040 consensus_queue.cc:1035] T 52430bbf347a4c5191edb23e1aa5d385 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:50.286998 21040 consensus_queue.cc:1035] T 52430bbf347a4c5191edb23e1aa5d385 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:50.289268 20873 raft_consensus.cc:2932] T 52430bbf347a4c5191edb23e1aa5d385 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:53:50.291311 20813 consensus_queue.cc:1035] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:50.291509 17607 raft_consensus.cc:2932] T 52430bbf347a4c5191edb23e1aa5d385 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:53:50.293429 17475 raft_consensus.cc:2932] T 52430bbf347a4c5191edb23e1aa5d385 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:53:50.294682 17334 catalog_manager.cc:5484] T 52430bbf347a4c5191edb23e1aa5d385 P dfa1152987644bd781b46c7e42a7a685 reported cstate change: config changed from index -1 to 4, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:50.299968 20765 consensus_queue.cc:1035] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:50.327448 17473 consensus_queue.cc:237] T 5b092dd1cc004c26a33e17c28af4d3ba P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } }
W20240502 01:53:50.329957 17411 consensus_peers.cc:485] T 5b092dd1cc004c26a33e17c28af4d3ba P 4da690c0fa13407a85c0689ce8de9ade -> Peer 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): Couldn't send request to peer 6a36c952c51542c0bf7ea3ec2857a9f9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5b092dd1cc004c26a33e17c28af4d3ba. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:50.330180 17735 raft_consensus.cc:1261] T 5b092dd1cc004c26a33e17c28af4d3ba P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:50.330789 21005 consensus_queue.cc:1035] T 5b092dd1cc004c26a33e17c28af4d3ba P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:50.330948 17607 raft_consensus.cc:1261] T 5b092dd1cc004c26a33e17c28af4d3ba P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:50.331421 21005 consensus_queue.cc:1035] T 5b092dd1cc004c26a33e17c28af4d3ba P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:50.334138 17328 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240502 01:53:50.334312 17328 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240502 01:53:50.334277 21006 raft_consensus.cc:2932] T 5b092dd1cc004c26a33e17c28af4d3ba P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } } }
I20240502 01:53:50.335150 17735 raft_consensus.cc:2932] T 5b092dd1cc004c26a33e17c28af4d3ba P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } } }
I20240502 01:53:50.338467 17328 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet 8652d27049f64a6eb65cdba6f3875765
I20240502 01:53:50.339021 17328 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet b572585ef00942df88881c4afbe7b7cd
I20240502 01:53:50.339416 17328 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet 9e4bc5ed97814e79b602697a9204aa5f
I20240502 01:53:50.339900 17455 tablet_service.cc:1514] Processing DeleteTablet for tablet b572585ef00942df88881c4afbe7b7cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:50 UTC) from {username='slave'} at 127.0.0.1:54520
I20240502 01:53:50.340685 17327 catalog_manager.cc:5484] T 5b092dd1cc004c26a33e17c28af4d3ba P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: config changed from index -1 to 4, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New cstate: current_term: 1 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:50.341159 17454 tablet_service.cc:1514] Processing DeleteTablet for tablet 8652d27049f64a6eb65cdba6f3875765 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:50 UTC) from {username='slave'} at 127.0.0.1:54520
I20240502 01:53:50.341727 20741 tablet_replica.cc:330] T b572585ef00942df88881c4afbe7b7cd P 4da690c0fa13407a85c0689ce8de9ade: stopping tablet replica
I20240502 01:53:50.341737 17587 tablet_service.cc:1514] Processing DeleteTablet for tablet 8652d27049f64a6eb65cdba6f3875765 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:50 UTC) from {username='slave'} at 127.0.0.1:55232
I20240502 01:53:50.341842 17585 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e4bc5ed97814e79b602697a9204aa5f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:50 UTC) from {username='slave'} at 127.0.0.1:55232
I20240502 01:53:50.341956 20741 raft_consensus.cc:2229] T b572585ef00942df88881c4afbe7b7cd P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:50.341995 20737 tablet_replica.cc:330] T 8652d27049f64a6eb65cdba6f3875765 P 3fceccefba274b1dbbd8216af8b0bc29: stopping tablet replica
I20240502 01:53:50.342173 20737 raft_consensus.cc:2229] T 8652d27049f64a6eb65cdba6f3875765 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:50.342278 20741 raft_consensus.cc:2258] T b572585ef00942df88881c4afbe7b7cd P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:50.342437 20737 raft_consensus.cc:2258] T 8652d27049f64a6eb65cdba6f3875765 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:50.342972 20741 ts_tablet_manager.cc:1875] T b572585ef00942df88881c4afbe7b7cd P 4da690c0fa13407a85c0689ce8de9ade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:50.343078 20737 ts_tablet_manager.cc:1875] T 8652d27049f64a6eb65cdba6f3875765 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:50.344137 17607 raft_consensus.cc:2932] T 5b092dd1cc004c26a33e17c28af4d3ba P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } } }
I20240502 01:53:50.345158 17716 tablet_service.cc:1514] Processing DeleteTablet for tablet b572585ef00942df88881c4afbe7b7cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:50 UTC) from {username='slave'} at 127.0.0.1:52198
I20240502 01:53:50.345156 17846 tablet_service.cc:1514] Processing DeleteTablet for tablet 8652d27049f64a6eb65cdba6f3875765 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:50 UTC) from {username='slave'} at 127.0.0.1:47226
I20240502 01:53:50.345495 20996 tablet_replica.cc:330] T 8652d27049f64a6eb65cdba6f3875765 P bb31bceba76d466db24b9341c04900c3: stopping tablet replica
I20240502 01:53:50.345561 20892 tablet_replica.cc:330] T b572585ef00942df88881c4afbe7b7cd P dfa1152987644bd781b46c7e42a7a685: stopping tablet replica
I20240502 01:53:50.345518 17845 tablet_service.cc:1514] Processing DeleteTablet for tablet b572585ef00942df88881c4afbe7b7cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:50 UTC) from {username='slave'} at 127.0.0.1:47226
I20240502 01:53:50.345711 20996 raft_consensus.cc:2229] T 8652d27049f64a6eb65cdba6f3875765 P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:50.345767 20892 raft_consensus.cc:2229] T b572585ef00942df88881c4afbe7b7cd P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:50.345999 20892 raft_consensus.cc:2258] T b572585ef00942df88881c4afbe7b7cd P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:50.346045 20996 raft_consensus.cc:2258] T 8652d27049f64a6eb65cdba6f3875765 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:50.346678 20892 ts_tablet_manager.cc:1875] T b572585ef00942df88881c4afbe7b7cd P dfa1152987644bd781b46c7e42a7a685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:50.346774 20996 ts_tablet_manager.cc:1875] T 8652d27049f64a6eb65cdba6f3875765 P bb31bceba76d466db24b9341c04900c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:50.349082 17847 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e4bc5ed97814e79b602697a9204aa5f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:50 UTC) from {username='slave'} at 127.0.0.1:47226
I20240502 01:53:50.349809 17454 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e4bc5ed97814e79b602697a9204aa5f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:50 UTC) from {username='slave'} at 127.0.0.1:54520
I20240502 01:53:50.353669 20737 ts_tablet_manager.cc:1888] T 8652d27049f64a6eb65cdba6f3875765 P 3fceccefba274b1dbbd8216af8b0bc29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:50.353853 20737 log.cc:1196] T 8652d27049f64a6eb65cdba6f3875765 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/wals/8652d27049f64a6eb65cdba6f3875765
I20240502 01:53:50.354986 20737 ts_tablet_manager.cc:1909] T 8652d27049f64a6eb65cdba6f3875765 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting consensus metadata
I20240502 01:53:50.355533 20996 ts_tablet_manager.cc:1888] T 8652d27049f64a6eb65cdba6f3875765 P bb31bceba76d466db24b9341c04900c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:50.356024 20996 log.cc:1196] T 8652d27049f64a6eb65cdba6f3875765 P bb31bceba76d466db24b9341c04900c3: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/wals/8652d27049f64a6eb65cdba6f3875765
I20240502 01:53:50.356324 20892 ts_tablet_manager.cc:1888] T b572585ef00942df88881c4afbe7b7cd P dfa1152987644bd781b46c7e42a7a685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:50.356309 17319 catalog_manager.cc:4830] TS 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): tablet 8652d27049f64a6eb65cdba6f3875765 (table rebalance_test_table_10 [id=b69ea3782c3c43dfa2760155beffc87b]) successfully deleted
I20240502 01:53:50.356624 20737 tablet_replica.cc:330] T 9e4bc5ed97814e79b602697a9204aa5f P 3fceccefba274b1dbbd8216af8b0bc29: stopping tablet replica
I20240502 01:53:50.356889 20737 raft_consensus.cc:2229] T 9e4bc5ed97814e79b602697a9204aa5f P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:50.357055 20996 ts_tablet_manager.cc:1909] T 8652d27049f64a6eb65cdba6f3875765 P bb31bceba76d466db24b9341c04900c3: Deleting consensus metadata
I20240502 01:53:50.357295 20737 raft_consensus.cc:2258] T 9e4bc5ed97814e79b602697a9204aa5f P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:50.357878 20996 tablet_replica.cc:330] T b572585ef00942df88881c4afbe7b7cd P bb31bceba76d466db24b9341c04900c3: stopping tablet replica
I20240502 01:53:50.358040 20737 ts_tablet_manager.cc:1875] T 9e4bc5ed97814e79b602697a9204aa5f P 3fceccefba274b1dbbd8216af8b0bc29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:50.358012 17322 catalog_manager.cc:4830] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): tablet 8652d27049f64a6eb65cdba6f3875765 (table rebalance_test_table_10 [id=b69ea3782c3c43dfa2760155beffc87b]) successfully deleted
I20240502 01:53:50.358244 20996 raft_consensus.cc:2229] T b572585ef00942df88881c4afbe7b7cd P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:50.358464 20996 raft_consensus.cc:2258] T b572585ef00942df88881c4afbe7b7cd P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:50.358980 20996 ts_tablet_manager.cc:1875] T b572585ef00942df88881c4afbe7b7cd P bb31bceba76d466db24b9341c04900c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:50.360634 20741 ts_tablet_manager.cc:1888] T b572585ef00942df88881c4afbe7b7cd P 4da690c0fa13407a85c0689ce8de9ade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:50.360776 20741 log.cc:1196] T b572585ef00942df88881c4afbe7b7cd P 4da690c0fa13407a85c0689ce8de9ade: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/b572585ef00942df88881c4afbe7b7cd
I20240502 01:53:50.361241 20741 ts_tablet_manager.cc:1909] T b572585ef00942df88881c4afbe7b7cd P 4da690c0fa13407a85c0689ce8de9ade: Deleting consensus metadata
I20240502 01:53:50.362267 17321 catalog_manager.cc:4830] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): tablet b572585ef00942df88881c4afbe7b7cd (table rebalance_test_table_10 [id=b69ea3782c3c43dfa2760155beffc87b]) successfully deleted
I20240502 01:53:50.362516 20892 log.cc:1196] T b572585ef00942df88881c4afbe7b7cd P dfa1152987644bd781b46c7e42a7a685: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/wals/b572585ef00942df88881c4afbe7b7cd
I20240502 01:53:50.362910 20892 ts_tablet_manager.cc:1909] T b572585ef00942df88881c4afbe7b7cd P dfa1152987644bd781b46c7e42a7a685: Deleting consensus metadata
I20240502 01:53:50.363095 20996 ts_tablet_manager.cc:1888] T b572585ef00942df88881c4afbe7b7cd P bb31bceba76d466db24b9341c04900c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:50.363216 20996 log.cc:1196] T b572585ef00942df88881c4afbe7b7cd P bb31bceba76d466db24b9341c04900c3: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/wals/b572585ef00942df88881c4afbe7b7cd
I20240502 01:53:50.363653 20737 ts_tablet_manager.cc:1888] T 9e4bc5ed97814e79b602697a9204aa5f P 3fceccefba274b1dbbd8216af8b0bc29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:50.363852 20737 log.cc:1196] T 9e4bc5ed97814e79b602697a9204aa5f P 3fceccefba274b1dbbd8216af8b0bc29: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/wals/9e4bc5ed97814e79b602697a9204aa5f
I20240502 01:53:50.363873 20741 tablet_replica.cc:330] T 8652d27049f64a6eb65cdba6f3875765 P 4da690c0fa13407a85c0689ce8de9ade: stopping tablet replica
I20240502 01:53:50.363946 17321 catalog_manager.cc:4830] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): tablet b572585ef00942df88881c4afbe7b7cd (table rebalance_test_table_10 [id=b69ea3782c3c43dfa2760155beffc87b]) successfully deleted
I20240502 01:53:50.364081 20741 raft_consensus.cc:2229] T 8652d27049f64a6eb65cdba6f3875765 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:50.364346 20741 raft_consensus.cc:2258] T 8652d27049f64a6eb65cdba6f3875765 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:50.364459 20737 ts_tablet_manager.cc:1909] T 9e4bc5ed97814e79b602697a9204aa5f P 3fceccefba274b1dbbd8216af8b0bc29: Deleting consensus metadata
I20240502 01:53:50.365000 20741 ts_tablet_manager.cc:1875] T 8652d27049f64a6eb65cdba6f3875765 P 4da690c0fa13407a85c0689ce8de9ade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:50.365490 20996 ts_tablet_manager.cc:1909] T b572585ef00942df88881c4afbe7b7cd P bb31bceba76d466db24b9341c04900c3: Deleting consensus metadata
I20240502 01:53:50.365658 17319 catalog_manager.cc:4830] TS 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): tablet 9e4bc5ed97814e79b602697a9204aa5f (table rebalance_test_table_10 [id=b69ea3782c3c43dfa2760155beffc87b]) successfully deleted
I20240502 01:53:50.366482 20996 tablet_replica.cc:330] T 9e4bc5ed97814e79b602697a9204aa5f P bb31bceba76d466db24b9341c04900c3: stopping tablet replica
I20240502 01:53:50.366586 17322 catalog_manager.cc:4830] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): tablet b572585ef00942df88881c4afbe7b7cd (table rebalance_test_table_10 [id=b69ea3782c3c43dfa2760155beffc87b]) successfully deleted
I20240502 01:53:50.366679 20996 raft_consensus.cc:2229] T 9e4bc5ed97814e79b602697a9204aa5f P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:50.366888 20996 raft_consensus.cc:2258] T 9e4bc5ed97814e79b602697a9204aa5f P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:50.367540 20996 ts_tablet_manager.cc:1875] T 9e4bc5ed97814e79b602697a9204aa5f P bb31bceba76d466db24b9341c04900c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:50.370007 20741 ts_tablet_manager.cc:1888] T 8652d27049f64a6eb65cdba6f3875765 P 4da690c0fa13407a85c0689ce8de9ade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:50.370132 20741 log.cc:1196] T 8652d27049f64a6eb65cdba6f3875765 P 4da690c0fa13407a85c0689ce8de9ade: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/8652d27049f64a6eb65cdba6f3875765
I20240502 01:53:50.370569 20741 ts_tablet_manager.cc:1909] T 8652d27049f64a6eb65cdba6f3875765 P 4da690c0fa13407a85c0689ce8de9ade: Deleting consensus metadata
I20240502 01:53:50.371335 20741 tablet_replica.cc:330] T 9e4bc5ed97814e79b602697a9204aa5f P 4da690c0fa13407a85c0689ce8de9ade: stopping tablet replica
I20240502 01:53:50.371533 20741 raft_consensus.cc:2229] T 9e4bc5ed97814e79b602697a9204aa5f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:50.371500 17321 catalog_manager.cc:4830] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): tablet 8652d27049f64a6eb65cdba6f3875765 (table rebalance_test_table_10 [id=b69ea3782c3c43dfa2760155beffc87b]) successfully deleted
I20240502 01:53:50.371757 20741 raft_consensus.cc:2258] T 9e4bc5ed97814e79b602697a9204aa5f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:50.372354 20741 ts_tablet_manager.cc:1875] T 9e4bc5ed97814e79b602697a9204aa5f P 4da690c0fa13407a85c0689ce8de9ade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:50.374069 20996 ts_tablet_manager.cc:1888] T 9e4bc5ed97814e79b602697a9204aa5f P bb31bceba76d466db24b9341c04900c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:50.374249 20996 log.cc:1196] T 9e4bc5ed97814e79b602697a9204aa5f P bb31bceba76d466db24b9341c04900c3: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/wals/9e4bc5ed97814e79b602697a9204aa5f
I20240502 01:53:50.374673 20996 ts_tablet_manager.cc:1909] T 9e4bc5ed97814e79b602697a9204aa5f P bb31bceba76d466db24b9341c04900c3: Deleting consensus metadata
I20240502 01:53:50.375687 17322 catalog_manager.cc:4830] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): tablet 9e4bc5ed97814e79b602697a9204aa5f (table rebalance_test_table_10 [id=b69ea3782c3c43dfa2760155beffc87b]) successfully deleted
I20240502 01:53:50.377089 20741 ts_tablet_manager.cc:1888] T 9e4bc5ed97814e79b602697a9204aa5f P 4da690c0fa13407a85c0689ce8de9ade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:50.377199 20741 log.cc:1196] T 9e4bc5ed97814e79b602697a9204aa5f P 4da690c0fa13407a85c0689ce8de9ade: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/9e4bc5ed97814e79b602697a9204aa5f
I20240502 01:53:50.377526 20741 ts_tablet_manager.cc:1909] T 9e4bc5ed97814e79b602697a9204aa5f P 4da690c0fa13407a85c0689ce8de9ade: Deleting consensus metadata
I20240502 01:53:50.378327 17321 catalog_manager.cc:4830] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): tablet 9e4bc5ed97814e79b602697a9204aa5f (table rebalance_test_table_10 [id=b69ea3782c3c43dfa2760155beffc87b]) successfully deleted
I20240502 01:53:50.498747 21034 raft_consensus.cc:482] T 06ff53fc9e1e48c69eb7484367fa5c74 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:50.498953 21034 raft_consensus.cc:504] T 06ff53fc9e1e48c69eb7484367fa5c74 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:50.499986 21034 leader_election.cc:288] T 06ff53fc9e1e48c69eb7484367fa5c74 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:53:50.500782 20232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06ff53fc9e1e48c69eb7484367fa5c74" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:53:50.501053 20232 raft_consensus.cc:2454] T 06ff53fc9e1e48c69eb7484367fa5c74 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 0.
I20240502 01:53:50.501703 20434 leader_election.cc:302] T 06ff53fc9e1e48c69eb7484367fa5c74 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:53:50.502000 21034 raft_consensus.cc:2781] T 06ff53fc9e1e48c69eb7484367fa5c74 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:50.502152 21034 raft_consensus.cc:482] T 06ff53fc9e1e48c69eb7484367fa5c74 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:50.502262 21034 raft_consensus.cc:3037] T 06ff53fc9e1e48c69eb7484367fa5c74 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:50.504953 21034 raft_consensus.cc:504] T 06ff53fc9e1e48c69eb7484367fa5c74 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:50.506709 20232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06ff53fc9e1e48c69eb7484367fa5c74" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:53:50.506903 20232 raft_consensus.cc:3037] T 06ff53fc9e1e48c69eb7484367fa5c74 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:50.510777 20765 consensus_queue.cc:1035] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:50.514797 20232 raft_consensus.cc:2454] T 06ff53fc9e1e48c69eb7484367fa5c74 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 1.
I20240502 01:53:50.515017 20630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06ff53fc9e1e48c69eb7484367fa5c74" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091" is_pre_election: true
I20240502 01:53:50.515281 20630 raft_consensus.cc:2454] T 06ff53fc9e1e48c69eb7484367fa5c74 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 0.
I20240502 01:53:50.515317 20434 leader_election.cc:302] T 06ff53fc9e1e48c69eb7484367fa5c74 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:53:50.517918 21034 leader_election.cc:288] T 06ff53fc9e1e48c69eb7484367fa5c74 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 election: Requested vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:53:50.518393 20630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06ff53fc9e1e48c69eb7484367fa5c74" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091"
I20240502 01:53:50.518576 20630 raft_consensus.cc:3037] T 06ff53fc9e1e48c69eb7484367fa5c74 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:50.520133 21034 raft_consensus.cc:2781] T 06ff53fc9e1e48c69eb7484367fa5c74 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:50.520505 21034 raft_consensus.cc:686] T 06ff53fc9e1e48c69eb7484367fa5c74 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Running, Role: LEADER
I20240502 01:53:50.520787 21034 consensus_queue.cc:237] T 06ff53fc9e1e48c69eb7484367fa5c74 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:50.521878 20630 raft_consensus.cc:2454] T 06ff53fc9e1e48c69eb7484367fa5c74 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 1.
I20240502 01:53:50.528682 20871 consensus_queue.cc:1035] T 5e8e78bd417d46cdb166087dbfa86446 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:50.531138 20877 consensus_queue.cc:1035] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:50.537426 20871 consensus_queue.cc:1035] T 5e8e78bd417d46cdb166087dbfa86446 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:50.560726 17335 catalog_manager.cc:5484] T 06ff53fc9e1e48c69eb7484367fa5c74 P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71). New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:50.569929 21098 ts_tablet_manager.cc:902] T 377e804dbeed4913a7d435cfdd1e2d8e P 2f306549407b4bb281ccf2994035e4f1: Initiating tablet copy from peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:53:50.589531 20478 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:50.603565 21098 tablet_copy_client.cc:308] T 377e804dbeed4913a7d435cfdd1e2d8e P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.68:44083
I20240502 01:53:50.604877 17887 tablet_copy_service.cc:140] P bb31bceba76d466db24b9341c04900c3: Received BeginTabletCopySession request for tablet 377e804dbeed4913a7d435cfdd1e2d8e from peer 2f306549407b4bb281ccf2994035e4f1 ({username='slave'} at 127.11.42.71:48285)
I20240502 01:53:50.605124 17887 tablet_copy_service.cc:161] P bb31bceba76d466db24b9341c04900c3: Beginning new tablet copy session on tablet 377e804dbeed4913a7d435cfdd1e2d8e from peer 2f306549407b4bb281ccf2994035e4f1 at {username='slave'} at 127.11.42.71:48285: session id = 2f306549407b4bb281ccf2994035e4f1-377e804dbeed4913a7d435cfdd1e2d8e
I20240502 01:53:50.608857 17887 tablet_copy_source_session.cc:205] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:50.616446 21098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 377e804dbeed4913a7d435cfdd1e2d8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:50.617329 17585 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:53:50.634265 17454 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:53:50.650930 21098 tablet_copy_client.cc:783] T 377e804dbeed4913a7d435cfdd1e2d8e P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:50.651350 21098 tablet_copy_client.cc:647] T 377e804dbeed4913a7d435cfdd1e2d8e P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:50.653905 21098 tablet_copy_client.cc:523] T 377e804dbeed4913a7d435cfdd1e2d8e P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:50.667610 20347 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:53:50.668067 21098 tablet_bootstrap.cc:492] T 377e804dbeed4913a7d435cfdd1e2d8e P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:53:50.686911 17335 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:53:50.694413 20630 raft_consensus.cc:1261] T 06ff53fc9e1e48c69eb7484367fa5c74 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:50.695295 20233 raft_consensus.cc:1261] T 06ff53fc9e1e48c69eb7484367fa5c74 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:50.695858 21034 consensus_queue.cc:1035] T 06ff53fc9e1e48c69eb7484367fa5c74 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:50.696368 21092 consensus_queue.cc:1035] T 06ff53fc9e1e48c69eb7484367fa5c74 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:50.712898 17332 catalog_manager.cc:5576] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339) does not have the latest schema for tablet 06ff53fc9e1e48c69eb7484367fa5c74 (table rebalancer_dynamic_table [id=754c57718ffc4923b82fb0c782be44c9]). Expected version 1 got 0
I20240502 01:53:50.715057 21046 mvcc.cc:205] Tried to move back new op lower bound from 7023062346517852160 to 7023062345815777280. Current Snapshot: MvccSnapshot[applied={T|T < 7023062346517852160}]
I20240502 01:53:50.715309 21123 tablet.cc:1722] T 06ff53fc9e1e48c69eb7484367fa5c74 P fdc2ac372492454c85387589a91f63af: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:53:50.715970 21123 tablet.cc:1620] T 06ff53fc9e1e48c69eb7484367fa5c74 P fdc2ac372492454c85387589a91f63af: MemRowSet was empty: no flush needed.
I20240502 01:53:50.719100 17335 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_extra_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:50.722180 17332 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=754c57718ffc4923b82fb0c782be44c9] alter complete (version 1)
I20240502 01:53:50.742803 21122 tablet.cc:1722] T 06ff53fc9e1e48c69eb7484367fa5c74 P 2f306549407b4bb281ccf2994035e4f1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:53:50.743455 21122 tablet.cc:1620] T 06ff53fc9e1e48c69eb7484367fa5c74 P 2f306549407b4bb281ccf2994035e4f1: MemRowSet was empty: no flush needed.
I20240502 01:53:50.746532 17454 tablet_service.cc:1467] Processing CreateTablet for tablet 648b1cc4d8b54c5eb0948ca8c1984a1f (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=3e245c8534e84c55922b7cd77670a4a3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:50.746996 17454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 648b1cc4d8b54c5eb0948ca8c1984a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:50.749439 20609 tablet_service.cc:1467] Processing CreateTablet for tablet 62887e332443457f8a5b41a982354dc4 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=3e245c8534e84c55922b7cd77670a4a3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:50.749878 20609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62887e332443457f8a5b41a982354dc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:50.754020 20347 tablet_service.cc:1467] Processing CreateTablet for tablet 648b1cc4d8b54c5eb0948ca8c1984a1f (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=3e245c8534e84c55922b7cd77670a4a3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:50.754474 20347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 648b1cc4d8b54c5eb0948ca8c1984a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:50.759621 17585 tablet_service.cc:1467] Processing CreateTablet for tablet cb7d51b2b9f84d608d67e771cd958231 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=3e245c8534e84c55922b7cd77670a4a3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:50.760083 17585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb7d51b2b9f84d608d67e771cd958231. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:50.770818 20211 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:50.771703 20478 tablet_service.cc:1467] Processing CreateTablet for tablet 648b1cc4d8b54c5eb0948ca8c1984a1f (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=3e245c8534e84c55922b7cd77670a4a3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:50.771798 20477 tablet_service.cc:1467] Processing CreateTablet for tablet cb7d51b2b9f84d608d67e771cd958231 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=3e245c8534e84c55922b7cd77670a4a3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:50.772055 20212 tablet_service.cc:1467] Processing CreateTablet for tablet 62887e332443457f8a5b41a982354dc4 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=3e245c8534e84c55922b7cd77670a4a3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:50.772137 20478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 648b1cc4d8b54c5eb0948ca8c1984a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:50.772452 20212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62887e332443457f8a5b41a982354dc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:50.772684 20476 tablet_service.cc:1467] Processing CreateTablet for tablet 62887e332443457f8a5b41a982354dc4 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=3e245c8534e84c55922b7cd77670a4a3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:50.777348 20476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62887e332443457f8a5b41a982354dc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:50.777442 20211 tablet_service.cc:1467] Processing CreateTablet for tablet cb7d51b2b9f84d608d67e771cd958231 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=3e245c8534e84c55922b7cd77670a4a3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:50.777827 20211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb7d51b2b9f84d608d67e771cd958231. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:50.781358 20477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb7d51b2b9f84d608d67e771cd958231. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:50.798126 21128 tablet_bootstrap.cc:492] T 62887e332443457f8a5b41a982354dc4 P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:53:50.800302 21128 tablet_bootstrap.cc:654] T 62887e332443457f8a5b41a982354dc4 P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:50.808673 17335 catalog_manager.cc:5576] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411) does not have the latest schema for tablet 06ff53fc9e1e48c69eb7484367fa5c74 (table rebalancer_dynamic_table [id=754c57718ffc4923b82fb0c782be44c9]). Expected version 1 got 0
I20240502 01:53:50.810206 21127 tablet_bootstrap.cc:492] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:53:50.812721 21129 tablet_bootstrap.cc:492] T cb7d51b2b9f84d608d67e771cd958231 P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:53:50.816303 21129 tablet_bootstrap.cc:654] T cb7d51b2b9f84d608d67e771cd958231 P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:50.822544 17716 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:53:50.822892 21127 tablet_bootstrap.cc:654] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:50.830240 17335 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:53:50.832449 21129 tablet_bootstrap.cc:492] T cb7d51b2b9f84d608d67e771cd958231 P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:53:50.832653 21129 ts_tablet_manager.cc:1367] T cb7d51b2b9f84d608d67e771cd958231 P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.002s
I20240502 01:53:50.833559 21129 raft_consensus.cc:348] T cb7d51b2b9f84d608d67e771cd958231 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:50.833864 21129 raft_consensus.cc:374] T cb7d51b2b9f84d608d67e771cd958231 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:50.834046 21129 raft_consensus.cc:729] T cb7d51b2b9f84d608d67e771cd958231 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:53:50.834434 21129 consensus_queue.cc:260] T cb7d51b2b9f84d608d67e771cd958231 P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:50.835204 21129 ts_tablet_manager.cc:1398] T cb7d51b2b9f84d608d67e771cd958231 P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:50.840293 21129 tablet_bootstrap.cc:492] T 62887e332443457f8a5b41a982354dc4 P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:53:50.843706 21129 tablet_bootstrap.cc:654] T 62887e332443457f8a5b41a982354dc4 P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:50.845295 21127 tablet_bootstrap.cc:492] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:53:50.845479 21127 ts_tablet_manager.cc:1367] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.000s
I20240502 01:53:50.846210 21127 raft_consensus.cc:348] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:50.846483 21127 raft_consensus.cc:374] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:50.846613 21127 raft_consensus.cc:729] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:53:50.846896 21127 consensus_queue.cc:260] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:50.847172 17332 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240502 01:53:50.847455 17332 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240502 01:53:50.847678 21127 ts_tablet_manager.cc:1398] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:50.847991 21127 tablet_bootstrap.cc:492] T cb7d51b2b9f84d608d67e771cd958231 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:53:50.849463 21127 tablet_bootstrap.cc:654] T cb7d51b2b9f84d608d67e771cd958231 P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:50.853272 21098 tablet_bootstrap.cc:492] T 377e804dbeed4913a7d435cfdd1e2d8e P 2f306549407b4bb281ccf2994035e4f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:50.853716 21098 tablet_bootstrap.cc:492] T 377e804dbeed4913a7d435cfdd1e2d8e P 2f306549407b4bb281ccf2994035e4f1: Bootstrap complete.
I20240502 01:53:50.854032 21098 ts_tablet_manager.cc:1367] T 377e804dbeed4913a7d435cfdd1e2d8e P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.196s user 0.050s sys 0.007s
I20240502 01:53:50.854053 17332 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet f67557c8f4a2434bb594b1a3d527b95a
I20240502 01:53:50.855082 17587 tablet_service.cc:1514] Processing DeleteTablet for tablet f67557c8f4a2434bb594b1a3d527b95a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:50 UTC) from {username='slave'} at 127.0.0.1:55232
I20240502 01:53:50.855377 20737 tablet_replica.cc:330] T f67557c8f4a2434bb594b1a3d527b95a P 3fceccefba274b1dbbd8216af8b0bc29: stopping tablet replica
I20240502 01:53:50.855571 20737 raft_consensus.cc:2229] T f67557c8f4a2434bb594b1a3d527b95a P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:50.855690 17455 tablet_service.cc:1514] Processing DeleteTablet for tablet f67557c8f4a2434bb594b1a3d527b95a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:50 UTC) from {username='slave'} at 127.0.0.1:54520
I20240502 01:53:50.855916 20737 raft_consensus.cc:2258] T f67557c8f4a2434bb594b1a3d527b95a P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:50.856370 17332 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet 7d86e5e0319c4056bee6b2110784fd7a
I20240502 01:53:50.856516 21134 tablet_bootstrap.cc:492] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:53:50.856613 20737 ts_tablet_manager.cc:1875] T f67557c8f4a2434bb594b1a3d527b95a P 3fceccefba274b1dbbd8216af8b0bc29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:50.858098 17455 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d86e5e0319c4056bee6b2110784fd7a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:50 UTC) from {username='slave'} at 127.0.0.1:54520
I20240502 01:53:50.858350 17716 tablet_service.cc:1514] Processing DeleteTablet for tablet f67557c8f4a2434bb594b1a3d527b95a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:50 UTC) from {username='slave'} at 127.0.0.1:52198
I20240502 01:53:50.858469 17332 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet 4cd17e0649214dbcb26d76addc10ddda
I20240502 01:53:50.858695 20892 tablet_replica.cc:330] T f67557c8f4a2434bb594b1a3d527b95a P dfa1152987644bd781b46c7e42a7a685: stopping tablet replica
I20240502 01:53:50.858892 20892 raft_consensus.cc:2229] T f67557c8f4a2434bb594b1a3d527b95a P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:50.859174 20892 raft_consensus.cc:2258] T f67557c8f4a2434bb594b1a3d527b95a P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:50.859849 20892 ts_tablet_manager.cc:1875] T f67557c8f4a2434bb594b1a3d527b95a P dfa1152987644bd781b46c7e42a7a685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:50.859910 17455 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cd17e0649214dbcb26d76addc10ddda with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:50 UTC) from {username='slave'} at 127.0.0.1:54520
I20240502 01:53:50.861182 17587 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d86e5e0319c4056bee6b2110784fd7a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:50 UTC) from {username='slave'} at 127.0.0.1:55232
I20240502 01:53:50.861871 17845 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cd17e0649214dbcb26d76addc10ddda with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:50 UTC) from {username='slave'} at 127.0.0.1:47226
I20240502 01:53:50.862043 17847 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d86e5e0319c4056bee6b2110784fd7a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:50 UTC) from {username='slave'} at 127.0.0.1:47226
I20240502 01:53:50.862166 20996 tablet_replica.cc:330] T 4cd17e0649214dbcb26d76addc10ddda P bb31bceba76d466db24b9341c04900c3: stopping tablet replica
I20240502 01:53:50.862409 20996 raft_consensus.cc:2229] T 4cd17e0649214dbcb26d76addc10ddda P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:50.862672 20996 raft_consensus.cc:2258] T 4cd17e0649214dbcb26d76addc10ddda P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:50.862710 17334 catalog_manager.cc:5576] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411) does not have the latest schema for tablet 06ff53fc9e1e48c69eb7484367fa5c74 (table rebalancer_dynamic_table [id=754c57718ffc4923b82fb0c782be44c9]). Expected version 2 got 0
I20240502 01:53:50.863394 20996 ts_tablet_manager.cc:1875] T 4cd17e0649214dbcb26d76addc10ddda P bb31bceba76d466db24b9341c04900c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:50.863756 20737 ts_tablet_manager.cc:1888] T f67557c8f4a2434bb594b1a3d527b95a P 3fceccefba274b1dbbd8216af8b0bc29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:50.863878 20737 log.cc:1196] T f67557c8f4a2434bb594b1a3d527b95a P 3fceccefba274b1dbbd8216af8b0bc29: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/wals/f67557c8f4a2434bb594b1a3d527b95a
I20240502 01:53:50.858155 20741 tablet_replica.cc:330] T f67557c8f4a2434bb594b1a3d527b95a P 4da690c0fa13407a85c0689ce8de9ade: stopping tablet replica
I20240502 01:53:50.864313 20737 ts_tablet_manager.cc:1909] T f67557c8f4a2434bb594b1a3d527b95a P 3fceccefba274b1dbbd8216af8b0bc29: Deleting consensus metadata
I20240502 01:53:50.864383 20741 raft_consensus.cc:2229] T f67557c8f4a2434bb594b1a3d527b95a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:50.864655 20741 raft_consensus.cc:2258] T f67557c8f4a2434bb594b1a3d527b95a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:50.865226 17716 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cd17e0649214dbcb26d76addc10ddda with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:50 UTC) from {username='slave'} at 127.0.0.1:52198
I20240502 01:53:50.865317 17319 catalog_manager.cc:4830] TS 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): tablet f67557c8f4a2434bb594b1a3d527b95a (table rebalance_test_table_12 [id=128b9c63621549098646906560fbc531]) successfully deleted
I20240502 01:53:50.865628 20737 tablet_replica.cc:330] T 7d86e5e0319c4056bee6b2110784fd7a P 3fceccefba274b1dbbd8216af8b0bc29: stopping tablet replica
I20240502 01:53:50.867359 20741 ts_tablet_manager.cc:1875] T f67557c8f4a2434bb594b1a3d527b95a P 4da690c0fa13407a85c0689ce8de9ade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:50.867543 20737 raft_consensus.cc:2229] T 7d86e5e0319c4056bee6b2110784fd7a P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:50.867789 20737 raft_consensus.cc:2258] T 7d86e5e0319c4056bee6b2110784fd7a P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:50.868541 20737 ts_tablet_manager.cc:1875] T 7d86e5e0319c4056bee6b2110784fd7a P 3fceccefba274b1dbbd8216af8b0bc29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:50.870160 20996 ts_tablet_manager.cc:1888] T 4cd17e0649214dbcb26d76addc10ddda P bb31bceba76d466db24b9341c04900c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:50.870334 20996 log.cc:1196] T 4cd17e0649214dbcb26d76addc10ddda P bb31bceba76d466db24b9341c04900c3: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/wals/4cd17e0649214dbcb26d76addc10ddda
I20240502 01:53:50.871235 20996 ts_tablet_manager.cc:1909] T 4cd17e0649214dbcb26d76addc10ddda P bb31bceba76d466db24b9341c04900c3: Deleting consensus metadata
I20240502 01:53:50.854749 21098 raft_consensus.cc:348] T 377e804dbeed4913a7d435cfdd1e2d8e P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
I20240502 01:53:50.871554 21098 raft_consensus.cc:729] T 377e804dbeed4913a7d435cfdd1e2d8e P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: LEARNER
I20240502 01:53:50.871835 21098 consensus_queue.cc:260] T 377e804dbeed4913a7d435cfdd1e2d8e P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
I20240502 01:53:50.873926 20996 tablet_replica.cc:330] T 7d86e5e0319c4056bee6b2110784fd7a P bb31bceba76d466db24b9341c04900c3: stopping tablet replica
I20240502 01:53:50.874168 20996 raft_consensus.cc:2229] T 7d86e5e0319c4056bee6b2110784fd7a P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:50.874493 20996 raft_consensus.cc:2258] T 7d86e5e0319c4056bee6b2110784fd7a P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:50.875319 17322 catalog_manager.cc:4830] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): tablet 4cd17e0649214dbcb26d76addc10ddda (table rebalance_test_table_12 [id=128b9c63621549098646906560fbc531]) successfully deleted
I20240502 01:53:50.876873 20996 ts_tablet_manager.cc:1875] T 7d86e5e0319c4056bee6b2110784fd7a P bb31bceba76d466db24b9341c04900c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:50.879051 20741 ts_tablet_manager.cc:1888] T f67557c8f4a2434bb594b1a3d527b95a P 4da690c0fa13407a85c0689ce8de9ade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:50.879513 21126 tablet.cc:1722] T 06ff53fc9e1e48c69eb7484367fa5c74 P f19cc6cb1e76460aa6711e49a5775091: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:53:50.880657 21126 tablet.cc:1620] T 06ff53fc9e1e48c69eb7484367fa5c74 P f19cc6cb1e76460aa6711e49a5775091: MemRowSet was empty: no flush needed.
I20240502 01:53:50.885113 20741 log.cc:1196] T f67557c8f4a2434bb594b1a3d527b95a P 4da690c0fa13407a85c0689ce8de9ade: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/f67557c8f4a2434bb594b1a3d527b95a
I20240502 01:53:50.885210 21098 ts_tablet_manager.cc:1398] T 377e804dbeed4913a7d435cfdd1e2d8e P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.031s user 0.002s sys 0.001s
I20240502 01:53:50.885689 20741 ts_tablet_manager.cc:1909] T f67557c8f4a2434bb594b1a3d527b95a P 4da690c0fa13407a85c0689ce8de9ade: Deleting consensus metadata
I20240502 01:53:50.885699 21122 tablet.cc:1722] T 06ff53fc9e1e48c69eb7484367fa5c74 P 2f306549407b4bb281ccf2994035e4f1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:53:50.886121 21122 tablet.cc:1620] T 06ff53fc9e1e48c69eb7484367fa5c74 P 2f306549407b4bb281ccf2994035e4f1: MemRowSet was empty: no flush needed.
I20240502 01:53:50.886780 21123 tablet.cc:1722] T 06ff53fc9e1e48c69eb7484367fa5c74 P fdc2ac372492454c85387589a91f63af: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:53:50.887342 21123 tablet.cc:1620] T 06ff53fc9e1e48c69eb7484367fa5c74 P fdc2ac372492454c85387589a91f63af: MemRowSet was empty: no flush needed.
I20240502 01:53:50.887953 20892 ts_tablet_manager.cc:1888] T f67557c8f4a2434bb594b1a3d527b95a P dfa1152987644bd781b46c7e42a7a685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:50.888128 20892 log.cc:1196] T f67557c8f4a2434bb594b1a3d527b95a P dfa1152987644bd781b46c7e42a7a685: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/wals/f67557c8f4a2434bb594b1a3d527b95a
I20240502 01:53:50.888684 21134 tablet_bootstrap.cc:654] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:50.888765 20892 ts_tablet_manager.cc:1909] T f67557c8f4a2434bb594b1a3d527b95a P dfa1152987644bd781b46c7e42a7a685: Deleting consensus metadata
I20240502 01:53:50.889899 20892 tablet_replica.cc:330] T 4cd17e0649214dbcb26d76addc10ddda P dfa1152987644bd781b46c7e42a7a685: stopping tablet replica
I20240502 01:53:50.890017 17321 catalog_manager.cc:4830] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): tablet f67557c8f4a2434bb594b1a3d527b95a (table rebalance_test_table_12 [id=128b9c63621549098646906560fbc531]) successfully deleted
I20240502 01:53:50.890319 20609 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:50.892575 17887 tablet_copy_service.cc:342] P bb31bceba76d466db24b9341c04900c3: Request end of tablet copy session 2f306549407b4bb281ccf2994035e4f1-377e804dbeed4913a7d435cfdd1e2d8e received from {username='slave'} at 127.11.42.71:48285
I20240502 01:53:50.892772 17887 tablet_copy_service.cc:433] P bb31bceba76d466db24b9341c04900c3: ending tablet copy session 2f306549407b4bb281ccf2994035e4f1-377e804dbeed4913a7d435cfdd1e2d8e on tablet 377e804dbeed4913a7d435cfdd1e2d8e with peer 2f306549407b4bb281ccf2994035e4f1
I20240502 01:53:50.894701 20737 ts_tablet_manager.cc:1888] T 7d86e5e0319c4056bee6b2110784fd7a P 3fceccefba274b1dbbd8216af8b0bc29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:50.895052 20737 log.cc:1196] T 7d86e5e0319c4056bee6b2110784fd7a P 3fceccefba274b1dbbd8216af8b0bc29: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/wals/7d86e5e0319c4056bee6b2110784fd7a
I20240502 01:53:50.895823 20737 ts_tablet_manager.cc:1909] T 7d86e5e0319c4056bee6b2110784fd7a P 3fceccefba274b1dbbd8216af8b0bc29: Deleting consensus metadata
I20240502 01:53:50.897908 17319 catalog_manager.cc:4830] TS 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): tablet 7d86e5e0319c4056bee6b2110784fd7a (table rebalance_test_table_12 [id=128b9c63621549098646906560fbc531]) successfully deleted
I20240502 01:53:50.899302 20741 tablet_replica.cc:330] T 7d86e5e0319c4056bee6b2110784fd7a P 4da690c0fa13407a85c0689ce8de9ade: stopping tablet replica
I20240502 01:53:50.899418 17321 catalog_manager.cc:4830] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): tablet f67557c8f4a2434bb594b1a3d527b95a (table rebalance_test_table_12 [id=128b9c63621549098646906560fbc531]) successfully deleted
I20240502 01:53:50.899823 20741 raft_consensus.cc:2229] T 7d86e5e0319c4056bee6b2110784fd7a P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:50.900493 20741 raft_consensus.cc:2258] T 7d86e5e0319c4056bee6b2110784fd7a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:50.901654 17327 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=754c57718ffc4923b82fb0c782be44c9] alter complete (version 2)
I20240502 01:53:50.902405 20892 raft_consensus.cc:2229] T 4cd17e0649214dbcb26d76addc10ddda P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:50.902967 20892 raft_consensus.cc:2258] T 4cd17e0649214dbcb26d76addc10ddda P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:50.904332 20892 ts_tablet_manager.cc:1875] T 4cd17e0649214dbcb26d76addc10ddda P dfa1152987644bd781b46c7e42a7a685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:50.908484 17335 catalog_manager.cc:5576] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339) does not have the latest schema for tablet 06ff53fc9e1e48c69eb7484367fa5c74 (table rebalancer_dynamic_table [id=754c57718ffc4923b82fb0c782be44c9]). Expected version 2 got 1
I20240502 01:53:50.910710 21131 tablet_bootstrap.cc:492] T cb7d51b2b9f84d608d67e771cd958231 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:50.913211 21131 tablet_bootstrap.cc:654] T cb7d51b2b9f84d608d67e771cd958231 P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:50.914454 21134 tablet_bootstrap.cc:492] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:53:50.914625 21134 ts_tablet_manager.cc:1367] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.092s user 0.002s sys 0.002s
I20240502 01:53:50.901957 20741 ts_tablet_manager.cc:1875] T 7d86e5e0319c4056bee6b2110784fd7a P 4da690c0fa13407a85c0689ce8de9ade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:50.917850 21127 tablet_bootstrap.cc:492] T cb7d51b2b9f84d608d67e771cd958231 P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:53:50.918119 21127 ts_tablet_manager.cc:1367] T cb7d51b2b9f84d608d67e771cd958231 P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.070s user 0.002s sys 0.003s
I20240502 01:53:50.919608 21134 raft_consensus.cc:348] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:50.914860 21133 tablet_bootstrap.cc:492] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:50.921725 21129 tablet_bootstrap.cc:492] T 62887e332443457f8a5b41a982354dc4 P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:53:50.922060 21129 ts_tablet_manager.cc:1367] T 62887e332443457f8a5b41a982354dc4 P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.082s user 0.002s sys 0.003s
I20240502 01:53:50.923669 21133 tablet_bootstrap.cc:654] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:50.923452 21129 raft_consensus.cc:348] T 62887e332443457f8a5b41a982354dc4 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:50.923887 21129 raft_consensus.cc:374] T 62887e332443457f8a5b41a982354dc4 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:50.924034 21129 raft_consensus.cc:729] T 62887e332443457f8a5b41a982354dc4 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:53:50.924890 20996 ts_tablet_manager.cc:1888] T 7d86e5e0319c4056bee6b2110784fd7a P bb31bceba76d466db24b9341c04900c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:50.925063 20996 log.cc:1196] T 7d86e5e0319c4056bee6b2110784fd7a P bb31bceba76d466db24b9341c04900c3: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/wals/7d86e5e0319c4056bee6b2110784fd7a
I20240502 01:53:50.925484 20996 ts_tablet_manager.cc:1909] T 7d86e5e0319c4056bee6b2110784fd7a P bb31bceba76d466db24b9341c04900c3: Deleting consensus metadata
I20240502 01:53:50.926725 21126 tablet.cc:1722] T 06ff53fc9e1e48c69eb7484367fa5c74 P f19cc6cb1e76460aa6711e49a5775091: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:53:50.927017 21134 raft_consensus.cc:374] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:50.927295 21134 raft_consensus.cc:729] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:53:50.927260 17322 catalog_manager.cc:4830] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): tablet 7d86e5e0319c4056bee6b2110784fd7a (table rebalance_test_table_12 [id=128b9c63621549098646906560fbc531]) successfully deleted
I20240502 01:53:50.927811 21134 consensus_queue.cc:260] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:50.928544 21126 tablet.cc:1620] T 06ff53fc9e1e48c69eb7484367fa5c74 P f19cc6cb1e76460aa6711e49a5775091: MemRowSet was empty: no flush needed.
I20240502 01:53:50.930712 21134 ts_tablet_manager.cc:1398] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20240502 01:53:50.930284 21128 tablet_bootstrap.cc:492] T 62887e332443457f8a5b41a982354dc4 P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:53:50.931844 21128 ts_tablet_manager.cc:1367] T 62887e332443457f8a5b41a982354dc4 P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.134s user 0.002s sys 0.009s
I20240502 01:53:50.924922 21129 consensus_queue.cc:260] T 62887e332443457f8a5b41a982354dc4 P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:50.934732 21128 raft_consensus.cc:348] T 62887e332443457f8a5b41a982354dc4 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:50.935454 17332 catalog_manager.cc:5576] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411) does not have the latest schema for tablet 06ff53fc9e1e48c69eb7484367fa5c74 (table rebalancer_dynamic_table [id=754c57718ffc4923b82fb0c782be44c9]). Expected version 2 got 1
I20240502 01:53:50.937601 21128 raft_consensus.cc:374] T 62887e332443457f8a5b41a982354dc4 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:50.938179 21139 ts_tablet_manager.cc:902] T 5b092dd1cc004c26a33e17c28af4d3ba P 6a36c952c51542c0bf7ea3ec2857a9f9: Initiating tablet copy from peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:50.939348 21129 ts_tablet_manager.cc:1398] T 62887e332443457f8a5b41a982354dc4 P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20240502 01:53:50.939649 21127 raft_consensus.cc:348] T cb7d51b2b9f84d608d67e771cd958231 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:50.939893 21127 raft_consensus.cc:374] T cb7d51b2b9f84d608d67e771cd958231 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:50.939977 21127 raft_consensus.cc:729] T cb7d51b2b9f84d608d67e771cd958231 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:53:50.940268 21127 consensus_queue.cc:260] T cb7d51b2b9f84d608d67e771cd958231 P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:50.940797 21127 ts_tablet_manager.cc:1398] T cb7d51b2b9f84d608d67e771cd958231 P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.023s user 0.000s sys 0.001s
I20240502 01:53:50.943586 21127 tablet_bootstrap.cc:492] T 62887e332443457f8a5b41a982354dc4 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:53:50.946172 21128 raft_consensus.cc:729] T 62887e332443457f8a5b41a982354dc4 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:53:50.949575 21128 consensus_queue.cc:260] T 62887e332443457f8a5b41a982354dc4 P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:50.950153 21128 ts_tablet_manager.cc:1398] T 62887e332443457f8a5b41a982354dc4 P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.018s user 0.000s sys 0.002s
I20240502 01:53:50.950516 21127 tablet_bootstrap.cc:654] T 62887e332443457f8a5b41a982354dc4 P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:50.958338 20741 ts_tablet_manager.cc:1888] T 7d86e5e0319c4056bee6b2110784fd7a P 4da690c0fa13407a85c0689ce8de9ade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:50.958555 20741 log.cc:1196] T 7d86e5e0319c4056bee6b2110784fd7a P 4da690c0fa13407a85c0689ce8de9ade: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/7d86e5e0319c4056bee6b2110784fd7a
I20240502 01:53:50.958995 20741 ts_tablet_manager.cc:1909] T 7d86e5e0319c4056bee6b2110784fd7a P 4da690c0fa13407a85c0689ce8de9ade: Deleting consensus metadata
I20240502 01:53:50.959481 21131 tablet_bootstrap.cc:492] T cb7d51b2b9f84d608d67e771cd958231 P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:50.959657 21131 ts_tablet_manager.cc:1367] T cb7d51b2b9f84d608d67e771cd958231 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.000s
I20240502 01:53:50.960225 20741 tablet_replica.cc:330] T 4cd17e0649214dbcb26d76addc10ddda P 4da690c0fa13407a85c0689ce8de9ade: stopping tablet replica
I20240502 01:53:50.960498 20741 raft_consensus.cc:2229] T 4cd17e0649214dbcb26d76addc10ddda P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:50.960497 21131 raft_consensus.cc:348] T cb7d51b2b9f84d608d67e771cd958231 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:50.960745 21131 raft_consensus.cc:374] T cb7d51b2b9f84d608d67e771cd958231 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:50.960881 20741 raft_consensus.cc:2258] T 4cd17e0649214dbcb26d76addc10ddda P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:50.960901 21131 raft_consensus.cc:729] T cb7d51b2b9f84d608d67e771cd958231 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:50.961189 17321 catalog_manager.cc:4830] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): tablet 7d86e5e0319c4056bee6b2110784fd7a (table rebalance_test_table_12 [id=128b9c63621549098646906560fbc531]) successfully deleted
I20240502 01:53:50.961169 21131 consensus_queue.cc:260] T cb7d51b2b9f84d608d67e771cd958231 P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:50.961781 20741 ts_tablet_manager.cc:1875] T 4cd17e0649214dbcb26d76addc10ddda P 4da690c0fa13407a85c0689ce8de9ade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:50.962031 21131 ts_tablet_manager.cc:1398] T cb7d51b2b9f84d608d67e771cd958231 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:53:50.968097 20741 ts_tablet_manager.cc:1888] T 4cd17e0649214dbcb26d76addc10ddda P 4da690c0fa13407a85c0689ce8de9ade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:50.968283 20741 log.cc:1196] T 4cd17e0649214dbcb26d76addc10ddda P 4da690c0fa13407a85c0689ce8de9ade: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/4cd17e0649214dbcb26d76addc10ddda
I20240502 01:53:50.969300 20741 ts_tablet_manager.cc:1909] T 4cd17e0649214dbcb26d76addc10ddda P 4da690c0fa13407a85c0689ce8de9ade: Deleting consensus metadata
I20240502 01:53:50.970342 17321 catalog_manager.cc:4830] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): tablet 4cd17e0649214dbcb26d76addc10ddda (table rebalance_test_table_12 [id=128b9c63621549098646906560fbc531]) successfully deleted
I20240502 01:53:50.972196 20892 ts_tablet_manager.cc:1888] T 4cd17e0649214dbcb26d76addc10ddda P dfa1152987644bd781b46c7e42a7a685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:50.972537 20892 log.cc:1196] T 4cd17e0649214dbcb26d76addc10ddda P dfa1152987644bd781b46c7e42a7a685: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/wals/4cd17e0649214dbcb26d76addc10ddda
I20240502 01:53:50.975095 21117 ts_tablet_manager.cc:902] T 52430bbf347a4c5191edb23e1aa5d385 P fdc2ac372492454c85387589a91f63af: Initiating tablet copy from peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:53:50.980329 21133 tablet_bootstrap.cc:492] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:50.980561 21133 ts_tablet_manager.cc:1367] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.066s user 0.003s sys 0.001s
I20240502 01:53:50.981531 21133 raft_consensus.cc:348] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:50.982430 17847 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:53:50.983063 21127 tablet_bootstrap.cc:492] T 62887e332443457f8a5b41a982354dc4 P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:53:50.983269 21127 ts_tablet_manager.cc:1367] T 62887e332443457f8a5b41a982354dc4 P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.040s user 0.001s sys 0.003s
I20240502 01:53:50.984198 21127 raft_consensus.cc:348] T 62887e332443457f8a5b41a982354dc4 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:50.984458 21127 raft_consensus.cc:374] T 62887e332443457f8a5b41a982354dc4 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:50.984591 21127 raft_consensus.cc:729] T 62887e332443457f8a5b41a982354dc4 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:53:50.984930 21127 consensus_queue.cc:260] T 62887e332443457f8a5b41a982354dc4 P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:50.985358 21133 raft_consensus.cc:374] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:50.986644 21133 raft_consensus.cc:729] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:50.986953 21133 consensus_queue.cc:260] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:50.991858 21127 ts_tablet_manager.cc:1398] T 62887e332443457f8a5b41a982354dc4 P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:50.992172 21133 ts_tablet_manager.cc:1398] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240502 01:53:50.992456 20892 ts_tablet_manager.cc:1909] T 4cd17e0649214dbcb26d76addc10ddda P dfa1152987644bd781b46c7e42a7a685: Deleting consensus metadata
I20240502 01:53:50.995261 17321 catalog_manager.cc:4830] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): tablet 4cd17e0649214dbcb26d76addc10ddda (table rebalance_test_table_12 [id=128b9c63621549098646906560fbc531]) successfully deleted
I20240502 01:53:51.005872 17332 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:53:51.012269 20768 raft_consensus.cc:482] T 62887e332443457f8a5b41a982354dc4 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:51.012472 20768 raft_consensus.cc:504] T 62887e332443457f8a5b41a982354dc4 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:51.013343 20768 leader_election.cc:288] T 62887e332443457f8a5b41a982354dc4 P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:53:51.014268 20630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62887e332443457f8a5b41a982354dc4" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091" is_pre_election: true
I20240502 01:53:51.014499 20630 raft_consensus.cc:2454] T 62887e332443457f8a5b41a982354dc4 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 0.
I20240502 01:53:51.014819 21122 tablet.cc:1722] T 06ff53fc9e1e48c69eb7484367fa5c74 P 2f306549407b4bb281ccf2994035e4f1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:53:51.016682 20167 leader_election.cc:302] T 62887e332443457f8a5b41a982354dc4 P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f19cc6cb1e76460aa6711e49a5775091, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:53:51.019400 20768 raft_consensus.cc:2781] T 62887e332443457f8a5b41a982354dc4 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:51.019569 20768 raft_consensus.cc:482] T 62887e332443457f8a5b41a982354dc4 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:51.019671 20768 raft_consensus.cc:3037] T 62887e332443457f8a5b41a982354dc4 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:51.020607 21126 tablet.cc:1722] T 06ff53fc9e1e48c69eb7484367fa5c74 P f19cc6cb1e76460aa6711e49a5775091: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:53:51.023927 20768 raft_consensus.cc:504] T 62887e332443457f8a5b41a982354dc4 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:51.028010 20630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62887e332443457f8a5b41a982354dc4" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091"
I20240502 01:53:51.028224 20630 raft_consensus.cc:3037] T 62887e332443457f8a5b41a982354dc4 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:51.031476 20630 raft_consensus.cc:2454] T 62887e332443457f8a5b41a982354dc4 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 1.
I20240502 01:53:51.033028 21094 raft_consensus.cc:482] T cb7d51b2b9f84d608d67e771cd958231 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:51.033182 21094 raft_consensus.cc:504] T cb7d51b2b9f84d608d67e771cd958231 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:51.033620 20167 leader_election.cc:302] T 62887e332443457f8a5b41a982354dc4 P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f19cc6cb1e76460aa6711e49a5775091, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:53:51.034003 21094 leader_election.cc:288] T cb7d51b2b9f84d608d67e771cd958231 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:53:51.034511 21094 raft_consensus.cc:482] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:51.034655 21094 raft_consensus.cc:504] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:51.035324 21094 leader_election.cc:288] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:51.038760 17334 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table_new_name [id=754c57718ffc4923b82fb0c782be44c9] alter complete (version 3)
I20240502 01:53:51.052964 20768 leader_election.cc:288] T 62887e332443457f8a5b41a982354dc4 P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 election: Requested vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:53:51.053330 20768 raft_consensus.cc:2781] T 62887e332443457f8a5b41a982354dc4 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:51.053568 20768 raft_consensus.cc:686] T 62887e332443457f8a5b41a982354dc4 P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Becoming Leader. State: Replica: fdc2ac372492454c85387589a91f63af, State: Running, Role: LEADER
I20240502 01:53:51.054042 20768 consensus_queue.cc:237] T 62887e332443457f8a5b41a982354dc4 P fdc2ac372492454c85387589a91f63af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:51.058455 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb7d51b2b9f84d608d67e771cd958231" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:53:51.058740 17604 raft_consensus.cc:2454] T cb7d51b2b9f84d608d67e771cd958231 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 0.
I20240502 01:53:51.059527 20431 leader_election.cc:302] T cb7d51b2b9f84d608d67e771cd958231 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 3fceccefba274b1dbbd8216af8b0bc29; no voters:
I20240502 01:53:51.059840 21094 raft_consensus.cc:2781] T cb7d51b2b9f84d608d67e771cd958231 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:51.059928 21094 raft_consensus.cc:482] T cb7d51b2b9f84d608d67e771cd958231 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:51.060022 21094 raft_consensus.cc:3037] T cb7d51b2b9f84d608d67e771cd958231 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:51.062741 21094 raft_consensus.cc:504] T cb7d51b2b9f84d608d67e771cd958231 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:51.063659 21094 leader_election.cc:288] T cb7d51b2b9f84d608d67e771cd958231 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 election: Requested vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:53:51.064313 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb7d51b2b9f84d608d67e771cd958231" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:53:51.064553 17604 raft_consensus.cc:3037] T cb7d51b2b9f84d608d67e771cd958231 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:51.067384 17604 raft_consensus.cc:2454] T cb7d51b2b9f84d608d67e771cd958231 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 1.
I20240502 01:53:51.068495 20431 leader_election.cc:302] T cb7d51b2b9f84d608d67e771cd958231 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 3fceccefba274b1dbbd8216af8b0bc29; no voters:
I20240502 01:53:51.068857 21094 raft_consensus.cc:2781] T cb7d51b2b9f84d608d67e771cd958231 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:51.069170 21094 raft_consensus.cc:686] T cb7d51b2b9f84d608d67e771cd958231 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Running, Role: LEADER
I20240502 01:53:51.069479 21094 consensus_queue.cc:237] T cb7d51b2b9f84d608d67e771cd958231 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:51.107620 20232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb7d51b2b9f84d608d67e771cd958231" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:53:51.107913 20232 raft_consensus.cc:2454] T cb7d51b2b9f84d608d67e771cd958231 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 0.
I20240502 01:53:51.108475 20232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb7d51b2b9f84d608d67e771cd958231" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:53:51.108706 20232 raft_consensus.cc:3037] T cb7d51b2b9f84d608d67e771cd958231 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:51.111608 20232 raft_consensus.cc:2454] T cb7d51b2b9f84d608d67e771cd958231 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 1.
I20240502 01:53:51.112485 21123 tablet.cc:1722] T 06ff53fc9e1e48c69eb7484367fa5c74 P fdc2ac372492454c85387589a91f63af: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:53:51.120481 20367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "648b1cc4d8b54c5eb0948ca8c1984a1f" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:53:51.120729 20367 raft_consensus.cc:2454] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 0.
I20240502 01:53:51.121412 20433 leader_election.cc:302] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 6a36c952c51542c0bf7ea3ec2857a9f9; no voters:
I20240502 01:53:51.121744 21092 raft_consensus.cc:2781] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:51.121846 21092 raft_consensus.cc:482] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:51.121918 21092 raft_consensus.cc:3037] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:51.124905 21092 raft_consensus.cc:504] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:51.125721 21092 leader_election.cc:288] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 election: Requested vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:51.126348 20367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "648b1cc4d8b54c5eb0948ca8c1984a1f" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
I20240502 01:53:51.126539 20367 raft_consensus.cc:3037] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:51.129339 20367 raft_consensus.cc:2454] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 1.
I20240502 01:53:51.130779 17335 catalog_manager.cc:5484] T 62887e332443457f8a5b41a982354dc4 P fdc2ac372492454c85387589a91f63af reported cstate change: term changed from 0 to 1, leader changed from <none> to fdc2ac372492454c85387589a91f63af (127.11.42.69). New cstate: current_term: 1 leader_uuid: "fdc2ac372492454c85387589a91f63af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: HEALTHY } } }
W20240502 01:53:51.132686 20167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.42.72:42411, user_credentials={real_user=slave}} blocked reactor thread for 99051.2us
I20240502 01:53:51.134511 20433 leader_election.cc:302] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 6a36c952c51542c0bf7ea3ec2857a9f9; no voters:
I20240502 01:53:51.134892 21094 raft_consensus.cc:2781] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:51.134851 17327 catalog_manager.cc:5484] T cb7d51b2b9f84d608d67e771cd958231 P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71). New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:51.135128 21094 raft_consensus.cc:686] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Running, Role: LEADER
I20240502 01:53:51.135416 21094 consensus_queue.cc:237] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:51.140007 20497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62887e332443457f8a5b41a982354dc4" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
I20240502 01:53:51.140231 20497 raft_consensus.cc:2454] T 62887e332443457f8a5b41a982354dc4 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 0.
I20240502 01:53:51.141192 20497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62887e332443457f8a5b41a982354dc4" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
I20240502 01:53:51.141351 20497 raft_consensus.cc:3037] T 62887e332443457f8a5b41a982354dc4 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:51.143689 20497 raft_consensus.cc:2454] T 62887e332443457f8a5b41a982354dc4 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 1.
I20240502 01:53:51.144237 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "648b1cc4d8b54c5eb0948ca8c1984a1f" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:53:51.144534 17475 raft_consensus.cc:3037] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:51.144503 17473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "648b1cc4d8b54c5eb0948ca8c1984a1f" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:51.146138 21139 tablet_copy_client.cc:308] T 5b092dd1cc004c26a33e17c28af4d3ba P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.65:38321
I20240502 01:53:51.147190 17475 raft_consensus.cc:2454] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 1.
I20240502 01:53:51.148752 17496 tablet_copy_service.cc:140] P 4da690c0fa13407a85c0689ce8de9ade: Received BeginTabletCopySession request for tablet 5b092dd1cc004c26a33e17c28af4d3ba from peer 6a36c952c51542c0bf7ea3ec2857a9f9 ({username='slave'} at 127.11.42.70:46151)
I20240502 01:53:51.148936 17496 tablet_copy_service.cc:161] P 4da690c0fa13407a85c0689ce8de9ade: Beginning new tablet copy session on tablet 5b092dd1cc004c26a33e17c28af4d3ba from peer 6a36c952c51542c0bf7ea3ec2857a9f9 at {username='slave'} at 127.11.42.70:46151: session id = 6a36c952c51542c0bf7ea3ec2857a9f9-5b092dd1cc004c26a33e17c28af4d3ba
I20240502 01:53:51.152539 17496 tablet_copy_source_session.cc:205] T 5b092dd1cc004c26a33e17c28af4d3ba P 4da690c0fa13407a85c0689ce8de9ade: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:51.155330 17327 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:53:51.155563 21139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b092dd1cc004c26a33e17c28af4d3ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:51.156083 17327 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:53:51.157390 20497 raft_consensus.cc:1203] T 377e804dbeed4913a7d435cfdd1e2d8e P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:53:51.157588 17332 catalog_manager.cc:5484] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71). New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } }
I20240502 01:53:51.163264 17327 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet 06ff53fc9e1e48c69eb7484367fa5c74
I20240502 01:53:51.164304 20609 tablet_service.cc:1514] Processing DeleteTablet for tablet 06ff53fc9e1e48c69eb7484367fa5c74 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:51 UTC) from {username='slave'} at 127.0.0.1:33698
I20240502 01:53:51.164459 20211 tablet_service.cc:1514] Processing DeleteTablet for tablet 06ff53fc9e1e48c69eb7484367fa5c74 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:51 UTC) from {username='slave'} at 127.0.0.1:44624
I20240502 01:53:51.164835 20478 tablet_service.cc:1514] Processing DeleteTablet for tablet 06ff53fc9e1e48c69eb7484367fa5c74 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:51 UTC) from {username='slave'} at 127.0.0.1:35018
I20240502 01:53:51.188688 21163 tablet_replica.cc:330] T 06ff53fc9e1e48c69eb7484367fa5c74 P f19cc6cb1e76460aa6711e49a5775091: stopping tablet replica
I20240502 01:53:51.189591 21163 raft_consensus.cc:2229] T 06ff53fc9e1e48c69eb7484367fa5c74 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:51.189895 21163 raft_consensus.cc:2258] T 06ff53fc9e1e48c69eb7484367fa5c74 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:51.190845 21163 ts_tablet_manager.cc:1875] T 06ff53fc9e1e48c69eb7484367fa5c74 P f19cc6cb1e76460aa6711e49a5775091: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:51.208552 21164 tablet_replica.cc:330] T 06ff53fc9e1e48c69eb7484367fa5c74 P fdc2ac372492454c85387589a91f63af: stopping tablet replica
I20240502 01:53:51.208819 21165 tablet_replica.cc:330] T 06ff53fc9e1e48c69eb7484367fa5c74 P 2f306549407b4bb281ccf2994035e4f1: stopping tablet replica
I20240502 01:53:51.209124 21165 raft_consensus.cc:2229] T 06ff53fc9e1e48c69eb7484367fa5c74 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:51.210536 21164 raft_consensus.cc:2229] T 06ff53fc9e1e48c69eb7484367fa5c74 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:51.211194 21165 raft_consensus.cc:2258] T 06ff53fc9e1e48c69eb7484367fa5c74 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:51.211488 21164 raft_consensus.cc:2258] T 06ff53fc9e1e48c69eb7484367fa5c74 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:51.212071 21163 ts_tablet_manager.cc:1888] T 06ff53fc9e1e48c69eb7484367fa5c74 P f19cc6cb1e76460aa6711e49a5775091: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:53:51.212610 21117 tablet_copy_client.cc:308] T 52430bbf347a4c5191edb23e1aa5d385 P fdc2ac372492454c85387589a91f63af: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.67:36115
I20240502 01:53:51.212743 21164 ts_tablet_manager.cc:1875] T 06ff53fc9e1e48c69eb7484367fa5c74 P fdc2ac372492454c85387589a91f63af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:51.213587 21165 ts_tablet_manager.cc:1875] T 06ff53fc9e1e48c69eb7484367fa5c74 P 2f306549407b4bb281ccf2994035e4f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:51.214701 21139 tablet_copy_client.cc:783] T 5b092dd1cc004c26a33e17c28af4d3ba P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:51.215078 21139 tablet_copy_client.cc:647] T 5b092dd1cc004c26a33e17c28af4d3ba P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:51.218317 21163 log.cc:1196] T 06ff53fc9e1e48c69eb7484367fa5c74 P f19cc6cb1e76460aa6711e49a5775091: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal/wals/06ff53fc9e1e48c69eb7484367fa5c74
I20240502 01:53:51.219179 21163 ts_tablet_manager.cc:1909] T 06ff53fc9e1e48c69eb7484367fa5c74 P f19cc6cb1e76460aa6711e49a5775091: Deleting consensus metadata
I20240502 01:53:51.221303 17321 catalog_manager.cc:4830] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): tablet 06ff53fc9e1e48c69eb7484367fa5c74 (table rebalancer_dynamic_table_new_name [id=754c57718ffc4923b82fb0c782be44c9]) successfully deleted
I20240502 01:53:51.225044 21139 tablet_copy_client.cc:523] T 5b092dd1cc004c26a33e17c28af4d3ba P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:51.225097 17757 tablet_copy_service.cc:140] P dfa1152987644bd781b46c7e42a7a685: Received BeginTabletCopySession request for tablet 52430bbf347a4c5191edb23e1aa5d385 from peer fdc2ac372492454c85387589a91f63af ({username='slave'} at 127.11.42.69:44021)
I20240502 01:53:51.225299 17757 tablet_copy_service.cc:161] P dfa1152987644bd781b46c7e42a7a685: Beginning new tablet copy session on tablet 52430bbf347a4c5191edb23e1aa5d385 from peer fdc2ac372492454c85387589a91f63af at {username='slave'} at 127.11.42.69:44021: session id = fdc2ac372492454c85387589a91f63af-52430bbf347a4c5191edb23e1aa5d385
I20240502 01:53:51.228920 17757 tablet_copy_source_session.cc:205] T 52430bbf347a4c5191edb23e1aa5d385 P dfa1152987644bd781b46c7e42a7a685: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:51.231578 21117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52430bbf347a4c5191edb23e1aa5d385. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:51.233606 21139 tablet_bootstrap.cc:492] T 5b092dd1cc004c26a33e17c28af4d3ba P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:53:51.239040 21117 tablet_copy_client.cc:783] T 52430bbf347a4c5191edb23e1aa5d385 P fdc2ac372492454c85387589a91f63af: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:51.239321 21117 tablet_copy_client.cc:647] T 52430bbf347a4c5191edb23e1aa5d385 P fdc2ac372492454c85387589a91f63af: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:51.240857 21164 ts_tablet_manager.cc:1888] T 06ff53fc9e1e48c69eb7484367fa5c74 P fdc2ac372492454c85387589a91f63af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:53:51.241906 21165 ts_tablet_manager.cc:1888] T 06ff53fc9e1e48c69eb7484367fa5c74 P 2f306549407b4bb281ccf2994035e4f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:53:51.242209 21165 log.cc:1196] T 06ff53fc9e1e48c69eb7484367fa5c74 P 2f306549407b4bb281ccf2994035e4f1: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal/wals/06ff53fc9e1e48c69eb7484367fa5c74
I20240502 01:53:51.242110 21117 tablet_copy_client.cc:523] T 52430bbf347a4c5191edb23e1aa5d385 P fdc2ac372492454c85387589a91f63af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:51.247419 21117 tablet_bootstrap.cc:492] T 52430bbf347a4c5191edb23e1aa5d385 P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:53:51.247910 21165 ts_tablet_manager.cc:1909] T 06ff53fc9e1e48c69eb7484367fa5c74 P 2f306549407b4bb281ccf2994035e4f1: Deleting consensus metadata
I20240502 01:53:51.249024 17322 catalog_manager.cc:4830] TS 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): tablet 06ff53fc9e1e48c69eb7484367fa5c74 (table rebalancer_dynamic_table_new_name [id=754c57718ffc4923b82fb0c782be44c9]) successfully deleted
I20240502 01:53:51.248008 21164 log.cc:1196] T 06ff53fc9e1e48c69eb7484367fa5c74 P fdc2ac372492454c85387589a91f63af: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal/wals/06ff53fc9e1e48c69eb7484367fa5c74
I20240502 01:53:51.251076 21164 ts_tablet_manager.cc:1909] T 06ff53fc9e1e48c69eb7484367fa5c74 P fdc2ac372492454c85387589a91f63af: Deleting consensus metadata
I20240502 01:53:51.252079 17322 catalog_manager.cc:4830] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): tablet 06ff53fc9e1e48c69eb7484367fa5c74 (table rebalancer_dynamic_table_new_name [id=754c57718ffc4923b82fb0c782be44c9]) successfully deleted
I20240502 01:53:51.267186 17335 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:53:51.276553 20347 tablet_service.cc:1467] Processing CreateTablet for tablet b39ec4b13e2a423f8c386f4c8bb6fe05 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e33e661e202249fca162bb1e059c31a8]), partition=
I20240502 01:53:51.276959 20347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b39ec4b13e2a423f8c386f4c8bb6fe05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:51.277734 20211 tablet_service.cc:1467] Processing CreateTablet for tablet b39ec4b13e2a423f8c386f4c8bb6fe05 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e33e661e202249fca162bb1e059c31a8]), partition=
I20240502 01:53:51.278060 20211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b39ec4b13e2a423f8c386f4c8bb6fe05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:51.278167 20478 tablet_service.cc:1467] Processing CreateTablet for tablet b39ec4b13e2a423f8c386f4c8bb6fe05 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e33e661e202249fca162bb1e059c31a8]), partition=
I20240502 01:53:51.278484 20478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b39ec4b13e2a423f8c386f4c8bb6fe05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:51.282945 21134 tablet_bootstrap.cc:492] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:53:51.284484 21134 tablet_bootstrap.cc:654] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:51.288923 21129 tablet_bootstrap.cc:492] T b39ec4b13e2a423f8c386f4c8bb6fe05 P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:53:51.290422 21129 tablet_bootstrap.cc:654] T b39ec4b13e2a423f8c386f4c8bb6fe05 P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:51.293987 21127 tablet_bootstrap.cc:492] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:53:51.295637 21127 tablet_bootstrap.cc:654] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:51.311126 21129 tablet_bootstrap.cc:492] T b39ec4b13e2a423f8c386f4c8bb6fe05 P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:53:51.311239 21127 tablet_bootstrap.cc:492] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:53:51.311386 21129 ts_tablet_manager.cc:1367] T b39ec4b13e2a423f8c386f4c8bb6fe05 P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.001s
I20240502 01:53:51.311512 21127 ts_tablet_manager.cc:1367] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.003s
I20240502 01:53:51.312175 21129 raft_consensus.cc:348] T b39ec4b13e2a423f8c386f4c8bb6fe05 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:51.312501 21129 raft_consensus.cc:374] T b39ec4b13e2a423f8c386f4c8bb6fe05 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:51.312640 21129 raft_consensus.cc:729] T b39ec4b13e2a423f8c386f4c8bb6fe05 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:53:51.312930 21127 raft_consensus.cc:348] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:51.312942 21129 consensus_queue.cc:260] T b39ec4b13e2a423f8c386f4c8bb6fe05 P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:51.313264 21127 raft_consensus.cc:374] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:51.313380 21127 raft_consensus.cc:729] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:53:51.313525 21129 ts_tablet_manager.cc:1398] T b39ec4b13e2a423f8c386f4c8bb6fe05 P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:51.313697 21127 consensus_queue.cc:260] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:51.315485 21127 ts_tablet_manager.cc:1398] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:51.316720 21139 tablet_bootstrap.cc:492] T 5b092dd1cc004c26a33e17c28af4d3ba P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:51.317273 21139 tablet_bootstrap.cc:492] T 5b092dd1cc004c26a33e17c28af4d3ba P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap complete.
I20240502 01:53:51.317627 21139 ts_tablet_manager.cc:1367] T 5b092dd1cc004c26a33e17c28af4d3ba P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.084s user 0.055s sys 0.005s
I20240502 01:53:51.318629 21139 raft_consensus.cc:348] T 5b092dd1cc004c26a33e17c28af4d3ba P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } }
I20240502 01:53:51.319038 21139 raft_consensus.cc:729] T 5b092dd1cc004c26a33e17c28af4d3ba P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: LEARNER
I20240502 01:53:51.319327 21139 consensus_queue.cc:260] T 5b092dd1cc004c26a33e17c28af4d3ba P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } }
I20240502 01:53:51.321759 21139 ts_tablet_manager.cc:1398] T 5b092dd1cc004c26a33e17c28af4d3ba P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:53:51.321723 20367 raft_consensus.cc:1203] T 5b092dd1cc004c26a33e17c28af4d3ba P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:53:51.322621 17496 tablet_copy_service.cc:342] P 4da690c0fa13407a85c0689ce8de9ade: Request end of tablet copy session 6a36c952c51542c0bf7ea3ec2857a9f9-5b092dd1cc004c26a33e17c28af4d3ba received from {username='slave'} at 127.11.42.70:46151
I20240502 01:53:51.322813 17496 tablet_copy_service.cc:433] P 4da690c0fa13407a85c0689ce8de9ade: ending tablet copy session 6a36c952c51542c0bf7ea3ec2857a9f9-5b092dd1cc004c26a33e17c28af4d3ba on tablet 5b092dd1cc004c26a33e17c28af4d3ba with peer 6a36c952c51542c0bf7ea3ec2857a9f9
I20240502 01:53:51.330483 21134 tablet_bootstrap.cc:492] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:53:51.330646 21134 ts_tablet_manager.cc:1367] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.048s user 0.011s sys 0.018s
I20240502 01:53:51.331327 21134 raft_consensus.cc:348] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:51.331605 21134 raft_consensus.cc:374] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:51.331735 21134 raft_consensus.cc:729] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:53:51.332489 21134 consensus_queue.cc:260] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:51.332978 21134 ts_tablet_manager.cc:1398] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:53:51.361653 17332 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240502 01:53:51.361863 17332 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240502 01:53:51.365862 17332 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet a12dd8d5f6164fb8a77d6a197aa0eb7e
I20240502 01:53:51.366427 17332 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet cf68f7ef20ba4064baee80ffddca53fe
I20240502 01:53:51.366823 17332 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet 26ac703604664aff911ebe8fd7d7f101
I20240502 01:53:51.366995 17585 tablet_service.cc:1514] Processing DeleteTablet for tablet a12dd8d5f6164fb8a77d6a197aa0eb7e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:51 UTC) from {username='slave'} at 127.0.0.1:55232
I20240502 01:53:51.367290 20737 tablet_replica.cc:330] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 3fceccefba274b1dbbd8216af8b0bc29: stopping tablet replica
I20240502 01:53:51.367517 20737 raft_consensus.cc:2229] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:51.367898 20737 raft_consensus.cc:2258] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:51.367975 17454 tablet_service.cc:1514] Processing DeleteTablet for tablet a12dd8d5f6164fb8a77d6a197aa0eb7e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:51 UTC) from {username='slave'} at 127.0.0.1:54520
I20240502 01:53:51.368160 17716 tablet_service.cc:1514] Processing DeleteTablet for tablet cf68f7ef20ba4064baee80ffddca53fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:51 UTC) from {username='slave'} at 127.0.0.1:52198
I20240502 01:53:51.368211 17456 tablet_service.cc:1514] Processing DeleteTablet for tablet 26ac703604664aff911ebe8fd7d7f101 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:51 UTC) from {username='slave'} at 127.0.0.1:54520
I20240502 01:53:51.368242 17455 tablet_service.cc:1514] Processing DeleteTablet for tablet cf68f7ef20ba4064baee80ffddca53fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:51 UTC) from {username='slave'} at 127.0.0.1:54520
I20240502 01:53:51.368671 20737 ts_tablet_manager.cc:1875] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 3fceccefba274b1dbbd8216af8b0bc29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:51.368712 17847 tablet_service.cc:1514] Processing DeleteTablet for tablet a12dd8d5f6164fb8a77d6a197aa0eb7e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:51 UTC) from {username='slave'} at 127.0.0.1:47226
I20240502 01:53:51.368921 17585 tablet_service.cc:1514] Processing DeleteTablet for tablet 26ac703604664aff911ebe8fd7d7f101 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:51 UTC) from {username='slave'} at 127.0.0.1:55232
I20240502 01:53:51.368988 20996 tablet_replica.cc:330] T a12dd8d5f6164fb8a77d6a197aa0eb7e P bb31bceba76d466db24b9341c04900c3: stopping tablet replica
I20240502 01:53:51.369055 20892 tablet_replica.cc:330] T cf68f7ef20ba4064baee80ffddca53fe P dfa1152987644bd781b46c7e42a7a685: stopping tablet replica
I20240502 01:53:51.369240 20996 raft_consensus.cc:2229] T a12dd8d5f6164fb8a77d6a197aa0eb7e P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:51.369246 20892 raft_consensus.cc:2229] T cf68f7ef20ba4064baee80ffddca53fe P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:51.369470 20996 raft_consensus.cc:2258] T a12dd8d5f6164fb8a77d6a197aa0eb7e P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:51.369586 20892 raft_consensus.cc:2258] T cf68f7ef20ba4064baee80ffddca53fe P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:51.370158 20996 ts_tablet_manager.cc:1875] T a12dd8d5f6164fb8a77d6a197aa0eb7e P bb31bceba76d466db24b9341c04900c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:51.370216 20892 ts_tablet_manager.cc:1875] T cf68f7ef20ba4064baee80ffddca53fe P dfa1152987644bd781b46c7e42a7a685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:51.372110 20741 tablet_replica.cc:330] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 4da690c0fa13407a85c0689ce8de9ade: stopping tablet replica
I20240502 01:53:51.372289 20741 raft_consensus.cc:2229] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:51.372469 20741 raft_consensus.cc:2258] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:51.372992 20741 ts_tablet_manager.cc:1875] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 4da690c0fa13407a85c0689ce8de9ade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:51.373212 17846 tablet_service.cc:1514] Processing DeleteTablet for tablet 26ac703604664aff911ebe8fd7d7f101 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:51 UTC) from {username='slave'} at 127.0.0.1:47226
I20240502 01:53:51.376847 20892 ts_tablet_manager.cc:1888] T cf68f7ef20ba4064baee80ffddca53fe P dfa1152987644bd781b46c7e42a7a685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:51.376956 20892 log.cc:1196] T cf68f7ef20ba4064baee80ffddca53fe P dfa1152987644bd781b46c7e42a7a685: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/wals/cf68f7ef20ba4064baee80ffddca53fe
I20240502 01:53:51.377295 17845 tablet_service.cc:1514] Processing DeleteTablet for tablet cf68f7ef20ba4064baee80ffddca53fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:51 UTC) from {username='slave'} at 127.0.0.1:47226
I20240502 01:53:51.377403 20892 ts_tablet_manager.cc:1909] T cf68f7ef20ba4064baee80ffddca53fe P dfa1152987644bd781b46c7e42a7a685: Deleting consensus metadata
I20240502 01:53:51.378068 20996 ts_tablet_manager.cc:1888] T a12dd8d5f6164fb8a77d6a197aa0eb7e P bb31bceba76d466db24b9341c04900c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:51.378266 20996 log.cc:1196] T a12dd8d5f6164fb8a77d6a197aa0eb7e P bb31bceba76d466db24b9341c04900c3: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/wals/a12dd8d5f6164fb8a77d6a197aa0eb7e
I20240502 01:53:51.378746 20996 ts_tablet_manager.cc:1909] T a12dd8d5f6164fb8a77d6a197aa0eb7e P bb31bceba76d466db24b9341c04900c3: Deleting consensus metadata
I20240502 01:53:51.378846 20741 ts_tablet_manager.cc:1888] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 4da690c0fa13407a85c0689ce8de9ade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:51.378849 17321 catalog_manager.cc:4830] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): tablet cf68f7ef20ba4064baee80ffddca53fe (table rebalance_test_table_14 [id=a07db144a5294a4dbfc503167983593e]) successfully deleted
I20240502 01:53:51.379004 20741 log.cc:1196] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 4da690c0fa13407a85c0689ce8de9ade: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/a12dd8d5f6164fb8a77d6a197aa0eb7e
I20240502 01:53:51.379446 20741 ts_tablet_manager.cc:1909] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 4da690c0fa13407a85c0689ce8de9ade: Deleting consensus metadata
I20240502 01:53:51.379993 20996 tablet_replica.cc:330] T 26ac703604664aff911ebe8fd7d7f101 P bb31bceba76d466db24b9341c04900c3: stopping tablet replica
I20240502 01:53:51.380240 17322 catalog_manager.cc:4830] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): tablet a12dd8d5f6164fb8a77d6a197aa0eb7e (table rebalance_test_table_14 [id=a07db144a5294a4dbfc503167983593e]) successfully deleted
I20240502 01:53:51.380560 20996 raft_consensus.cc:2229] T 26ac703604664aff911ebe8fd7d7f101 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:51.380860 20996 raft_consensus.cc:2258] T 26ac703604664aff911ebe8fd7d7f101 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:51.381548 20996 ts_tablet_manager.cc:1875] T 26ac703604664aff911ebe8fd7d7f101 P bb31bceba76d466db24b9341c04900c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:51.381618 17321 catalog_manager.cc:4830] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): tablet a12dd8d5f6164fb8a77d6a197aa0eb7e (table rebalance_test_table_14 [id=a07db144a5294a4dbfc503167983593e]) successfully deleted
I20240502 01:53:51.381758 20741 tablet_replica.cc:330] T cf68f7ef20ba4064baee80ffddca53fe P 4da690c0fa13407a85c0689ce8de9ade: stopping tablet replica
I20240502 01:53:51.381969 20741 raft_consensus.cc:2229] T cf68f7ef20ba4064baee80ffddca53fe P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:51.382223 20741 raft_consensus.cc:2258] T cf68f7ef20ba4064baee80ffddca53fe P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:51.382225 20737 ts_tablet_manager.cc:1888] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 3fceccefba274b1dbbd8216af8b0bc29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:51.382344 20737 log.cc:1196] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 3fceccefba274b1dbbd8216af8b0bc29: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/wals/a12dd8d5f6164fb8a77d6a197aa0eb7e
I20240502 01:53:51.382894 20737 ts_tablet_manager.cc:1909] T a12dd8d5f6164fb8a77d6a197aa0eb7e P 3fceccefba274b1dbbd8216af8b0bc29: Deleting consensus metadata
I20240502 01:53:51.383124 20741 ts_tablet_manager.cc:1875] T cf68f7ef20ba4064baee80ffddca53fe P 4da690c0fa13407a85c0689ce8de9ade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:51.383978 17319 catalog_manager.cc:4830] TS 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): tablet a12dd8d5f6164fb8a77d6a197aa0eb7e (table rebalance_test_table_14 [id=a07db144a5294a4dbfc503167983593e]) successfully deleted
I20240502 01:53:51.384363 20737 tablet_replica.cc:330] T 26ac703604664aff911ebe8fd7d7f101 P 3fceccefba274b1dbbd8216af8b0bc29: stopping tablet replica
I20240502 01:53:51.384624 20737 raft_consensus.cc:2229] T 26ac703604664aff911ebe8fd7d7f101 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:51.384994 20737 raft_consensus.cc:2258] T 26ac703604664aff911ebe8fd7d7f101 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:51.385732 20737 ts_tablet_manager.cc:1875] T 26ac703604664aff911ebe8fd7d7f101 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:51.389674 20996 ts_tablet_manager.cc:1888] T 26ac703604664aff911ebe8fd7d7f101 P bb31bceba76d466db24b9341c04900c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:51.389813 20996 log.cc:1196] T 26ac703604664aff911ebe8fd7d7f101 P bb31bceba76d466db24b9341c04900c3: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/wals/26ac703604664aff911ebe8fd7d7f101
I20240502 01:53:51.390414 20741 ts_tablet_manager.cc:1888] T cf68f7ef20ba4064baee80ffddca53fe P 4da690c0fa13407a85c0689ce8de9ade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:51.390537 20741 log.cc:1196] T cf68f7ef20ba4064baee80ffddca53fe P 4da690c0fa13407a85c0689ce8de9ade: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/cf68f7ef20ba4064baee80ffddca53fe
I20240502 01:53:51.390614 20996 ts_tablet_manager.cc:1909] T 26ac703604664aff911ebe8fd7d7f101 P bb31bceba76d466db24b9341c04900c3: Deleting consensus metadata
I20240502 01:53:51.390977 20741 ts_tablet_manager.cc:1909] T cf68f7ef20ba4064baee80ffddca53fe P 4da690c0fa13407a85c0689ce8de9ade: Deleting consensus metadata
I20240502 01:53:51.391644 17322 catalog_manager.cc:4830] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): tablet 26ac703604664aff911ebe8fd7d7f101 (table rebalance_test_table_14 [id=a07db144a5294a4dbfc503167983593e]) successfully deleted
I20240502 01:53:51.391966 20996 tablet_replica.cc:330] T cf68f7ef20ba4064baee80ffddca53fe P bb31bceba76d466db24b9341c04900c3: stopping tablet replica
I20240502 01:53:51.391932 17321 catalog_manager.cc:4830] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): tablet cf68f7ef20ba4064baee80ffddca53fe (table rebalance_test_table_14 [id=a07db144a5294a4dbfc503167983593e]) successfully deleted
I20240502 01:53:51.392040 20741 tablet_replica.cc:330] T 26ac703604664aff911ebe8fd7d7f101 P 4da690c0fa13407a85c0689ce8de9ade: stopping tablet replica
I20240502 01:53:51.392177 20996 raft_consensus.cc:2229] T cf68f7ef20ba4064baee80ffddca53fe P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:51.392249 20741 raft_consensus.cc:2229] T 26ac703604664aff911ebe8fd7d7f101 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:51.392406 20996 raft_consensus.cc:2258] T cf68f7ef20ba4064baee80ffddca53fe P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:51.392478 20741 raft_consensus.cc:2258] T 26ac703604664aff911ebe8fd7d7f101 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:51.392772 20737 ts_tablet_manager.cc:1888] T 26ac703604664aff911ebe8fd7d7f101 P 3fceccefba274b1dbbd8216af8b0bc29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:51.392980 20737 log.cc:1196] T 26ac703604664aff911ebe8fd7d7f101 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/wals/26ac703604664aff911ebe8fd7d7f101
I20240502 01:53:51.393170 20996 ts_tablet_manager.cc:1875] T cf68f7ef20ba4064baee80ffddca53fe P bb31bceba76d466db24b9341c04900c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:51.393275 20741 ts_tablet_manager.cc:1875] T 26ac703604664aff911ebe8fd7d7f101 P 4da690c0fa13407a85c0689ce8de9ade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:51.393563 20737 ts_tablet_manager.cc:1909] T 26ac703604664aff911ebe8fd7d7f101 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting consensus metadata
I20240502 01:53:51.394496 17319 catalog_manager.cc:4830] TS 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): tablet 26ac703604664aff911ebe8fd7d7f101 (table rebalance_test_table_14 [id=a07db144a5294a4dbfc503167983593e]) successfully deleted
I20240502 01:53:51.398257 20741 ts_tablet_manager.cc:1888] T 26ac703604664aff911ebe8fd7d7f101 P 4da690c0fa13407a85c0689ce8de9ade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:51.398398 20741 log.cc:1196] T 26ac703604664aff911ebe8fd7d7f101 P 4da690c0fa13407a85c0689ce8de9ade: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/26ac703604664aff911ebe8fd7d7f101
I20240502 01:53:51.398842 20741 ts_tablet_manager.cc:1909] T 26ac703604664aff911ebe8fd7d7f101 P 4da690c0fa13407a85c0689ce8de9ade: Deleting consensus metadata
I20240502 01:53:51.399097 20996 ts_tablet_manager.cc:1888] T cf68f7ef20ba4064baee80ffddca53fe P bb31bceba76d466db24b9341c04900c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:51.399322 20996 log.cc:1196] T cf68f7ef20ba4064baee80ffddca53fe P bb31bceba76d466db24b9341c04900c3: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/wals/cf68f7ef20ba4064baee80ffddca53fe
I20240502 01:53:51.399804 17321 catalog_manager.cc:4830] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): tablet 26ac703604664aff911ebe8fd7d7f101 (table rebalance_test_table_14 [id=a07db144a5294a4dbfc503167983593e]) successfully deleted
I20240502 01:53:51.399874 20996 ts_tablet_manager.cc:1909] T cf68f7ef20ba4064baee80ffddca53fe P bb31bceba76d466db24b9341c04900c3: Deleting consensus metadata
I20240502 01:53:51.400722 17322 catalog_manager.cc:4830] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): tablet cf68f7ef20ba4064baee80ffddca53fe (table rebalance_test_table_14 [id=a07db144a5294a4dbfc503167983593e]) successfully deleted
I20240502 01:53:51.404239 21117 tablet_bootstrap.cc:492] T 52430bbf347a4c5191edb23e1aa5d385 P fdc2ac372492454c85387589a91f63af: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:51.404594 21117 tablet_bootstrap.cc:492] T 52430bbf347a4c5191edb23e1aa5d385 P fdc2ac372492454c85387589a91f63af: Bootstrap complete.
I20240502 01:53:51.404806 21117 ts_tablet_manager.cc:1367] T 52430bbf347a4c5191edb23e1aa5d385 P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.157s user 0.051s sys 0.003s
I20240502 01:53:51.405491 21117 raft_consensus.cc:348] T 52430bbf347a4c5191edb23e1aa5d385 P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } }
I20240502 01:53:51.405712 21117 raft_consensus.cc:729] T 52430bbf347a4c5191edb23e1aa5d385 P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: LEARNER
I20240502 01:53:51.405928 21117 consensus_queue.cc:260] T 52430bbf347a4c5191edb23e1aa5d385 P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } }
I20240502 01:53:51.406431 21117 ts_tablet_manager.cc:1398] T 52430bbf347a4c5191edb23e1aa5d385 P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 01:53:51.407341 17757 tablet_copy_service.cc:342] P dfa1152987644bd781b46c7e42a7a685: Request end of tablet copy session fdc2ac372492454c85387589a91f63af-52430bbf347a4c5191edb23e1aa5d385 received from {username='slave'} at 127.11.42.69:44021
I20240502 01:53:51.407516 17757 tablet_copy_service.cc:433] P dfa1152987644bd781b46c7e42a7a685: ending tablet copy session fdc2ac372492454c85387589a91f63af-52430bbf347a4c5191edb23e1aa5d385 on tablet 52430bbf347a4c5191edb23e1aa5d385 with peer fdc2ac372492454c85387589a91f63af
I20240502 01:53:51.410825 20367 consensus_queue.cc:237] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } }
I20240502 01:53:51.413755 20630 raft_consensus.cc:1261] T 31aa6a26f64f49368ad43cc97c66f2f1 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:51.414759 20232 raft_consensus.cc:1261] T 31aa6a26f64f49368ad43cc97c66f2f1 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:51.415055 20877 consensus_queue.cc:1035] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:53:51.415374 20813 consensus_queue.cc:1035] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:53:51.419438 21009 raft_consensus.cc:2932] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } } }
I20240502 01:53:51.420102 20232 raft_consensus.cc:2932] T 31aa6a26f64f49368ad43cc97c66f2f1 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } } }
W20240502 01:53:51.422758 20303 consensus_peers.cc:485] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): Couldn't send request to peer bb31bceba76d466db24b9341c04900c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 31aa6a26f64f49368ad43cc97c66f2f1. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:51.425127 20630 raft_consensus.cc:2932] T 31aa6a26f64f49368ad43cc97c66f2f1 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } } }
I20240502 01:53:51.425588 17332 catalog_manager.cc:5484] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: config changed from index -1 to 2, NON_VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) added. New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:51.484210 20768 consensus_queue.cc:1035] T 62887e332443457f8a5b41a982354dc4 P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:51.484872 21154 consensus_queue.cc:1035] T 62887e332443457f8a5b41a982354dc4 P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240502 01:53:51.501876 21094 consensus_queue.cc:1035] T cb7d51b2b9f84d608d67e771cd958231 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:51.506511 21092 consensus_queue.cc:1035] T cb7d51b2b9f84d608d67e771cd958231 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:51.535480 21092 consensus_queue.cc:1035] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:51.549954 21092 consensus_queue.cc:1035] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:51.562026 21009 raft_consensus.cc:482] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:51.562196 21009 raft_consensus.cc:504] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:51.562872 21009 leader_election.cc:288] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:53:51.564165 20232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b39ec4b13e2a423f8c386f4c8bb6fe05" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:53:51.564410 20232 raft_consensus.cc:2454] T b39ec4b13e2a423f8c386f4c8bb6fe05 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 0.
I20240502 01:53:51.565128 20497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b39ec4b13e2a423f8c386f4c8bb6fe05" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
I20240502 01:53:51.565347 20497 raft_consensus.cc:2454] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 0.
I20240502 01:53:51.566437 20303 leader_election.cc:302] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:53:51.568936 20877 raft_consensus.cc:2781] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:51.569064 20877 raft_consensus.cc:482] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:51.569169 20877 raft_consensus.cc:3037] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:51.571854 20877 raft_consensus.cc:504] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:51.574347 20877 leader_election.cc:288] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Requested vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:53:51.574944 20497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b39ec4b13e2a423f8c386f4c8bb6fe05" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
I20240502 01:53:51.574945 20232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b39ec4b13e2a423f8c386f4c8bb6fe05" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:53:51.575137 20497 raft_consensus.cc:3037] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:51.575129 20232 raft_consensus.cc:3037] T b39ec4b13e2a423f8c386f4c8bb6fe05 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:51.577843 20497 raft_consensus.cc:2454] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 1.
I20240502 01:53:51.579936 20232 raft_consensus.cc:2454] T b39ec4b13e2a423f8c386f4c8bb6fe05 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 1.
I20240502 01:53:51.579254 20303 leader_election.cc:302] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 6a36c952c51542c0bf7ea3ec2857a9f9; no voters:
I20240502 01:53:51.581110 21009 raft_consensus.cc:2781] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:51.581279 21009 raft_consensus.cc:686] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Becoming Leader. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Running, Role: LEADER
I20240502 01:53:51.581540 21009 consensus_queue.cc:237] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:51.592423 17334 catalog_manager.cc:5484] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70). New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:51.646045 20478 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:51.659071 20232 raft_consensus.cc:1203] T 52430bbf347a4c5191edb23e1aa5d385 P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:53:51.659473 17456 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:53:51.675477 17845 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:53:51.695761 17585 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:53:51.702330 17327 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:53:51.705596 17335 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_extra_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:51.709964 20232 raft_consensus.cc:1261] T b39ec4b13e2a423f8c386f4c8bb6fe05 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:51.710431 20497 raft_consensus.cc:1261] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:51.710590 20877 consensus_queue.cc:1035] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:51.711099 20877 consensus_queue.cc:1035] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:51.719108 21172 raft_consensus.cc:1050] T 5b092dd1cc004c26a33e17c28af4d3ba P 4da690c0fa13407a85c0689ce8de9ade: attempting to promote NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 to VOTER
I20240502 01:53:51.720232 21172 consensus_queue.cc:237] T 5b092dd1cc004c26a33e17c28af4d3ba P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } }
I20240502 01:53:51.722085 20347 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:53:51.723609 20478 tablet_service.cc:1467] Processing CreateTablet for tablet aebde87a7f114537abd9c1d8b7e97004 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=c5f998b2cdaf41f6a49ea6ae65a975b8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:51.724040 20478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aebde87a7f114537abd9c1d8b7e97004. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:51.725217 20345 tablet_service.cc:1467] Processing CreateTablet for tablet 31a4ca0cc44e492fbd13a70ce38add35 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=c5f998b2cdaf41f6a49ea6ae65a975b8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:51.725607 20345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31a4ca0cc44e492fbd13a70ce38add35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:51.727710 17716 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:53:51.731154 21127 tablet_bootstrap.cc:492] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:53:51.731485 20609 tablet_service.cc:1467] Processing CreateTablet for tablet 617443200e2c47b69d63ef056294f40c (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=c5f998b2cdaf41f6a49ea6ae65a975b8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:51.731926 20609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 617443200e2c47b69d63ef056294f40c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:51.732810 21127 tablet_bootstrap.cc:654] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:51.732841 20476 tablet_service.cc:1467] Processing CreateTablet for tablet 617443200e2c47b69d63ef056294f40c (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=c5f998b2cdaf41f6a49ea6ae65a975b8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:51.733811 20476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 617443200e2c47b69d63ef056294f40c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:51.739202 20211 tablet_service.cc:1467] Processing CreateTablet for tablet 31a4ca0cc44e492fbd13a70ce38add35 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=c5f998b2cdaf41f6a49ea6ae65a975b8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:51.739234 20212 tablet_service.cc:1467] Processing CreateTablet for tablet 617443200e2c47b69d63ef056294f40c (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=c5f998b2cdaf41f6a49ea6ae65a975b8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:51.739609 20211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31a4ca0cc44e492fbd13a70ce38add35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:51.739907 17845 tablet_service.cc:1467] Processing CreateTablet for tablet aebde87a7f114537abd9c1d8b7e97004 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=c5f998b2cdaf41f6a49ea6ae65a975b8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:51.740324 17845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aebde87a7f114537abd9c1d8b7e97004. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:51.740849 17846 tablet_service.cc:1467] Processing CreateTablet for tablet 31a4ca0cc44e492fbd13a70ce38add35 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=c5f998b2cdaf41f6a49ea6ae65a975b8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:51.741261 17846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31a4ca0cc44e492fbd13a70ce38add35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:51.742322 20212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 617443200e2c47b69d63ef056294f40c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:51.752758 21129 tablet_bootstrap.cc:492] T 617443200e2c47b69d63ef056294f40c P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:53:51.753218 21148 raft_consensus.cc:1050] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3: attempting to promote NON_VOTER 2f306549407b4bb281ccf2994035e4f1 to VOTER
I20240502 01:53:51.754318 21129 tablet_bootstrap.cc:654] T 617443200e2c47b69d63ef056294f40c P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:51.754307 21148 consensus_queue.cc:237] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } }
I20240502 01:53:51.760916 20347 tablet_service.cc:1467] Processing CreateTablet for tablet aebde87a7f114537abd9c1d8b7e97004 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=c5f998b2cdaf41f6a49ea6ae65a975b8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:51.761379 20347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aebde87a7f114537abd9c1d8b7e97004. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:51.761353 17335 catalog_manager.cc:5576] TS 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623) does not have the latest schema for tablet b39ec4b13e2a423f8c386f4c8bb6fe05 (table rebalancer_dynamic_table [id=e33e661e202249fca162bb1e059c31a8]). Expected version 1 got 0
I20240502 01:53:51.761433 21217 mvcc.cc:205] Tried to move back new op lower bound from 7023062350676774912 to 7023062350160306176. Current Snapshot: MvccSnapshot[applied={T|T < 7023062350676774912}]
I20240502 01:53:51.762802 21127 tablet_bootstrap.cc:492] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:53:51.762948 21127 ts_tablet_manager.cc:1367] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.009s
I20240502 01:53:51.763716 21127 raft_consensus.cc:348] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:51.767469 21134 tablet_bootstrap.cc:492] T aebde87a7f114537abd9c1d8b7e97004 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:53:51.769156 21127 raft_consensus.cc:374] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:51.769270 21127 raft_consensus.cc:729] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:53:51.769304 21134 tablet_bootstrap.cc:654] T aebde87a7f114537abd9c1d8b7e97004 P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:51.769567 21127 consensus_queue.cc:260] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:51.770196 21127 ts_tablet_manager.cc:1398] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240502 01:53:51.770501 21127 tablet_bootstrap.cc:492] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:53:51.772209 21127 tablet_bootstrap.cc:654] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:51.774478 21129 tablet_bootstrap.cc:492] T 617443200e2c47b69d63ef056294f40c P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:53:51.774698 21129 ts_tablet_manager.cc:1367] T 617443200e2c47b69d63ef056294f40c P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.000s
I20240502 01:53:51.775856 21129 raft_consensus.cc:348] T 617443200e2c47b69d63ef056294f40c P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:51.776140 21129 raft_consensus.cc:374] T 617443200e2c47b69d63ef056294f40c P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:51.776247 21129 raft_consensus.cc:729] T 617443200e2c47b69d63ef056294f40c P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:53:51.776749 21129 consensus_queue.cc:260] T 617443200e2c47b69d63ef056294f40c P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:51.777853 17335 catalog_manager.cc:5576] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339) does not have the latest schema for tablet b39ec4b13e2a423f8c386f4c8bb6fe05 (table rebalancer_dynamic_table [id=e33e661e202249fca162bb1e059c31a8]). Expected version 1 got 0
I20240502 01:53:51.779193 21129 ts_tablet_manager.cc:1398] T 617443200e2c47b69d63ef056294f40c P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:53:51.779510 21129 tablet_bootstrap.cc:492] T 31a4ca0cc44e492fbd13a70ce38add35 P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:53:51.781201 21129 tablet_bootstrap.cc:654] T 31a4ca0cc44e492fbd13a70ce38add35 P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:51.786095 21241 tablet.cc:1722] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 6a36c952c51542c0bf7ea3ec2857a9f9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:53:51.786752 21241 tablet.cc:1620] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 6a36c952c51542c0bf7ea3ec2857a9f9: MemRowSet was empty: no flush needed.
I20240502 01:53:51.789192 21242 tablet.cc:1722] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 2f306549407b4bb281ccf2994035e4f1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:53:51.789884 21242 tablet.cc:1620] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 2f306549407b4bb281ccf2994035e4f1: MemRowSet was empty: no flush needed.
I20240502 01:53:51.798993 21246 tablet.cc:1722] T b39ec4b13e2a423f8c386f4c8bb6fe05 P fdc2ac372492454c85387589a91f63af: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:53:51.804864 17327 catalog_manager.cc:5576] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339) does not have the latest schema for tablet b39ec4b13e2a423f8c386f4c8bb6fe05 (table rebalancer_dynamic_table [id=e33e661e202249fca162bb1e059c31a8]). Expected version 1 got 0
I20240502 01:53:51.805649 20367 raft_consensus.cc:1261] T 5b092dd1cc004c26a33e17c28af4d3ba P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:51.806842 21129 tablet_bootstrap.cc:492] T 31a4ca0cc44e492fbd13a70ce38add35 P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:53:51.806910 17607 raft_consensus.cc:1261] T 5b092dd1cc004c26a33e17c28af4d3ba P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:51.806989 21129 ts_tablet_manager.cc:1367] T 31a4ca0cc44e492fbd13a70ce38add35 P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.003s
I20240502 01:53:51.807749 21129 raft_consensus.cc:348] T 31a4ca0cc44e492fbd13a70ce38add35 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:51.807967 21129 raft_consensus.cc:374] T 31a4ca0cc44e492fbd13a70ce38add35 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:51.808045 21129 raft_consensus.cc:729] T 31a4ca0cc44e492fbd13a70ce38add35 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:53:51.808391 21005 consensus_queue.cc:1035] T 5b092dd1cc004c26a33e17c28af4d3ba P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:51.808554 21246 tablet.cc:1620] T b39ec4b13e2a423f8c386f4c8bb6fe05 P fdc2ac372492454c85387589a91f63af: MemRowSet was empty: no flush needed.
I20240502 01:53:51.810178 17335 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=e33e661e202249fca162bb1e059c31a8] alter complete (version 1)
I20240502 01:53:51.810400 17332 catalog_manager.cc:5576] TS 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623) does not have the latest schema for tablet b39ec4b13e2a423f8c386f4c8bb6fe05 (table rebalancer_dynamic_table [id=e33e661e202249fca162bb1e059c31a8]). Expected version 1 got 0
I20240502 01:53:51.816959 21005 consensus_queue.cc:1035] T 5b092dd1cc004c26a33e17c28af4d3ba P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
I20240502 01:53:51.821914 21129 consensus_queue.cc:260] T 31a4ca0cc44e492fbd13a70ce38add35 P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:51.823226 17736 raft_consensus.cc:1261] T 5b092dd1cc004c26a33e17c28af4d3ba P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:51.823920 21005 consensus_queue.cc:1035] T 5b092dd1cc004c26a33e17c28af4d3ba P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:51.825389 21129 ts_tablet_manager.cc:1398] T 31a4ca0cc44e492fbd13a70ce38add35 P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20240502 01:53:51.826637 21127 tablet_bootstrap.cc:492] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:53:51.826822 21127 ts_tablet_manager.cc:1367] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.056s user 0.009s sys 0.009s
I20240502 01:53:51.827602 21127 raft_consensus.cc:348] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:51.827841 21127 raft_consensus.cc:374] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:51.827930 21127 raft_consensus.cc:729] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:53:51.828217 21127 consensus_queue.cc:260] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:51.828840 21127 ts_tablet_manager.cc:1398] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:53:51.833639 21134 tablet_bootstrap.cc:492] T aebde87a7f114537abd9c1d8b7e97004 P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:53:51.834061 21134 ts_tablet_manager.cc:1367] T aebde87a7f114537abd9c1d8b7e97004 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.067s user 0.003s sys 0.009s
I20240502 01:53:51.835014 21134 raft_consensus.cc:348] T aebde87a7f114537abd9c1d8b7e97004 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:51.835325 21134 raft_consensus.cc:374] T aebde87a7f114537abd9c1d8b7e97004 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:51.835464 21134 raft_consensus.cc:729] T aebde87a7f114537abd9c1d8b7e97004 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:53:51.835778 21134 consensus_queue.cc:260] T aebde87a7f114537abd9c1d8b7e97004 P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:51.836619 21134 ts_tablet_manager.cc:1398] T aebde87a7f114537abd9c1d8b7e97004 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:51.836946 21134 tablet_bootstrap.cc:492] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:53:51.838690 21134 tablet_bootstrap.cc:654] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:51.847657 20610 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:51.855885 21253 tablet_bootstrap.cc:492] T 617443200e2c47b69d63ef056294f40c P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:53:51.859784 21253 tablet_bootstrap.cc:654] T 617443200e2c47b69d63ef056294f40c P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:51.861835 21134 tablet_bootstrap.cc:492] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:53:51.862087 21134 ts_tablet_manager.cc:1367] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.002s
I20240502 01:53:51.862887 21134 raft_consensus.cc:348] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:51.868059 21253 tablet_bootstrap.cc:492] T 617443200e2c47b69d63ef056294f40c P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:53:51.868490 21253 ts_tablet_manager.cc:1367] T 617443200e2c47b69d63ef056294f40c P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20240502 01:53:51.869334 21253 raft_consensus.cc:348] T 617443200e2c47b69d63ef056294f40c P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:51.869637 21253 raft_consensus.cc:374] T 617443200e2c47b69d63ef056294f40c P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:51.869767 21253 raft_consensus.cc:729] T 617443200e2c47b69d63ef056294f40c P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:53:51.870081 21253 consensus_queue.cc:260] T 617443200e2c47b69d63ef056294f40c P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:51.871445 17327 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240502 01:53:51.871668 17327 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240502 01:53:51.875348 21134 raft_consensus.cc:374] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:51.875684 21134 raft_consensus.cc:729] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:53:51.875880 21253 ts_tablet_manager.cc:1398] T 617443200e2c47b69d63ef056294f40c P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 01:53:51.876281 17327 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet fa21373c6f4a43df9fd2a5ec2b8d6242
I20240502 01:53:51.876238 21134 consensus_queue.cc:260] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:51.876829 17327 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet 27815f771d5b4bbaa7a70d5ea30b279b
I20240502 01:53:51.877835 17585 tablet_service.cc:1514] Processing DeleteTablet for tablet fa21373c6f4a43df9fd2a5ec2b8d6242 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:51 UTC) from {username='slave'} at 127.0.0.1:55232
I20240502 01:53:51.877986 17587 tablet_service.cc:1514] Processing DeleteTablet for tablet 27815f771d5b4bbaa7a70d5ea30b279b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:51 UTC) from {username='slave'} at 127.0.0.1:55232
I20240502 01:53:51.878229 20737 tablet_replica.cc:330] T fa21373c6f4a43df9fd2a5ec2b8d6242 P 3fceccefba274b1dbbd8216af8b0bc29: stopping tablet replica
I20240502 01:53:51.878430 20737 raft_consensus.cc:2229] T fa21373c6f4a43df9fd2a5ec2b8d6242 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:51.878589 17327 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 4 replicas of tablet 377e804dbeed4913a7d435cfdd1e2d8e
I20240502 01:53:51.878700 20737 raft_consensus.cc:2258] T fa21373c6f4a43df9fd2a5ec2b8d6242 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:51.878780 21134 ts_tablet_manager.cc:1398] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20240502 01:53:51.879459 20737 ts_tablet_manager.cc:1875] T fa21373c6f4a43df9fd2a5ec2b8d6242 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:51.879551 17847 tablet_service.cc:1514] Processing DeleteTablet for tablet 377e804dbeed4913a7d435cfdd1e2d8e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:51 UTC) from {username='slave'} at 127.0.0.1:47226
I20240502 01:53:51.879858 20996 tablet_replica.cc:330] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3: stopping tablet replica
I20240502 01:53:51.881482 17716 tablet_service.cc:1514] Processing DeleteTablet for tablet fa21373c6f4a43df9fd2a5ec2b8d6242 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:51 UTC) from {username='slave'} at 127.0.0.1:52198
I20240502 01:53:51.881606 17715 tablet_service.cc:1514] Processing DeleteTablet for tablet 377e804dbeed4913a7d435cfdd1e2d8e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:51 UTC) from {username='slave'} at 127.0.0.1:52198
I20240502 01:53:51.883194 20478 tablet_service.cc:1514] Processing DeleteTablet for tablet 377e804dbeed4913a7d435cfdd1e2d8e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:51 UTC) from {username='slave'} at 127.0.0.1:35018
I20240502 01:53:51.884756 17455 tablet_service.cc:1514] Processing DeleteTablet for tablet 27815f771d5b4bbaa7a70d5ea30b279b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:51 UTC) from {username='slave'} at 127.0.0.1:54520
I20240502 01:53:51.884858 17456 tablet_service.cc:1514] Processing DeleteTablet for tablet fa21373c6f4a43df9fd2a5ec2b8d6242 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:51 UTC) from {username='slave'} at 127.0.0.1:54520
I20240502 01:53:51.885036 17454 tablet_service.cc:1514] Processing DeleteTablet for tablet 377e804dbeed4913a7d435cfdd1e2d8e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:51 UTC) from {username='slave'} at 127.0.0.1:54520
I20240502 01:53:51.885107 20741 tablet_replica.cc:330] T 27815f771d5b4bbaa7a70d5ea30b279b P 4da690c0fa13407a85c0689ce8de9ade: stopping tablet replica
I20240502 01:53:51.885329 20741 raft_consensus.cc:2229] T 27815f771d5b4bbaa7a70d5ea30b279b P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:51.885605 20741 raft_consensus.cc:2258] T 27815f771d5b4bbaa7a70d5ea30b279b P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:51.886373 20741 ts_tablet_manager.cc:1875] T 27815f771d5b4bbaa7a70d5ea30b279b P 4da690c0fa13407a85c0689ce8de9ade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:51.887061 21094 raft_consensus.cc:482] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:51.887221 21094 raft_consensus.cc:504] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:51.887940 21094 leader_election.cc:288] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:53:51.888489 20233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "617443200e2c47b69d63ef056294f40c" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:53:51.888701 20233 raft_consensus.cc:2454] T 617443200e2c47b69d63ef056294f40c P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 0.
I20240502 01:53:51.889068 20630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "617443200e2c47b69d63ef056294f40c" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091" is_pre_election: true
I20240502 01:53:51.889318 20630 raft_consensus.cc:2454] T 617443200e2c47b69d63ef056294f40c P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 0.
I20240502 01:53:51.889744 20433 leader_election.cc:302] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:53:51.890411 21094 raft_consensus.cc:2781] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:51.890520 21094 raft_consensus.cc:482] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:51.890594 21094 raft_consensus.cc:3037] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:51.893162 21094 raft_consensus.cc:504] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:51.893922 21094 leader_election.cc:288] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 election: Requested vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:53:51.894392 20233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "617443200e2c47b69d63ef056294f40c" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:53:51.894604 20233 raft_consensus.cc:3037] T 617443200e2c47b69d63ef056294f40c P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:51.895314 20737 ts_tablet_manager.cc:1888] T fa21373c6f4a43df9fd2a5ec2b8d6242 P 3fceccefba274b1dbbd8216af8b0bc29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:51.895512 20737 log.cc:1196] T fa21373c6f4a43df9fd2a5ec2b8d6242 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/wals/fa21373c6f4a43df9fd2a5ec2b8d6242
I20240502 01:53:51.896096 20737 ts_tablet_manager.cc:1909] T fa21373c6f4a43df9fd2a5ec2b8d6242 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting consensus metadata
I20240502 01:53:51.897323 20233 raft_consensus.cc:2454] T 617443200e2c47b69d63ef056294f40c P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 1.
I20240502 01:53:51.897336 17319 catalog_manager.cc:4830] TS 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): tablet fa21373c6f4a43df9fd2a5ec2b8d6242 (table rebalance_test_table_16 [id=f27ebc659702427c996fda2492c7e5d0]) successfully deleted
I20240502 01:53:51.897760 20737 tablet_replica.cc:330] T 27815f771d5b4bbaa7a70d5ea30b279b P 3fceccefba274b1dbbd8216af8b0bc29: stopping tablet replica
I20240502 01:53:51.897733 20434 leader_election.cc:302] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:53:51.898115 17845 tablet_service.cc:1514] Processing DeleteTablet for tablet 27815f771d5b4bbaa7a70d5ea30b279b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:51 UTC) from {username='slave'} at 127.0.0.1:47226
I20240502 01:53:51.898372 21094 raft_consensus.cc:2781] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:51.898532 21094 raft_consensus.cc:686] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Running, Role: LEADER
I20240502 01:53:51.898790 21094 consensus_queue.cc:237] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:51.900161 20741 ts_tablet_manager.cc:1888] T 27815f771d5b4bbaa7a70d5ea30b279b P 4da690c0fa13407a85c0689ce8de9ade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:51.900274 20741 log.cc:1196] T 27815f771d5b4bbaa7a70d5ea30b279b P 4da690c0fa13407a85c0689ce8de9ade: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/27815f771d5b4bbaa7a70d5ea30b279b
I20240502 01:53:51.900249 20630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "617443200e2c47b69d63ef056294f40c" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091"
I20240502 01:53:51.900493 20630 raft_consensus.cc:3037] T 617443200e2c47b69d63ef056294f40c P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:51.900751 20741 ts_tablet_manager.cc:1909] T 27815f771d5b4bbaa7a70d5ea30b279b P 4da690c0fa13407a85c0689ce8de9ade: Deleting consensus metadata
I20240502 01:53:51.903941 20630 raft_consensus.cc:2454] T 617443200e2c47b69d63ef056294f40c P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 1.
I20240502 01:53:51.908816 20741 tablet_replica.cc:330] T fa21373c6f4a43df9fd2a5ec2b8d6242 P 4da690c0fa13407a85c0689ce8de9ade: stopping tablet replica
I20240502 01:53:51.909070 20741 raft_consensus.cc:2229] T fa21373c6f4a43df9fd2a5ec2b8d6242 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:51.909057 17321 catalog_manager.cc:4830] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): tablet 27815f771d5b4bbaa7a70d5ea30b279b (table rebalance_test_table_16 [id=f27ebc659702427c996fda2492c7e5d0]) successfully deleted
I20240502 01:53:51.909324 20741 raft_consensus.cc:2258] T fa21373c6f4a43df9fd2a5ec2b8d6242 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:51.910063 20741 ts_tablet_manager.cc:1875] T fa21373c6f4a43df9fd2a5ec2b8d6242 P 4da690c0fa13407a85c0689ce8de9ade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:51.915906 21216 raft_consensus.cc:482] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:51.916039 21216 raft_consensus.cc:504] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:51.916718 21216 leader_election.cc:288] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:53:51.917522 20366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aebde87a7f114537abd9c1d8b7e97004" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:53:51.917752 20366 raft_consensus.cc:2454] T aebde87a7f114537abd9c1d8b7e97004 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 0.
I20240502 01:53:51.918210 20433 leader_election.cc:302] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 6a36c952c51542c0bf7ea3ec2857a9f9; no voters:
I20240502 01:53:51.918526 21216 raft_consensus.cc:2781] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:51.918632 21216 raft_consensus.cc:482] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:51.918731 21216 raft_consensus.cc:3037] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:51.919171 17864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aebde87a7f114537abd9c1d8b7e97004" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:53:51.921636 21216 raft_consensus.cc:504] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
W20240502 01:53:51.923310 20434 leader_election.cc:341] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:53:51.924269 20366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aebde87a7f114537abd9c1d8b7e97004" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
I20240502 01:53:51.924482 20366 raft_consensus.cc:3037] T aebde87a7f114537abd9c1d8b7e97004 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:51.925290 21257 tablet_replica.cc:330] T 377e804dbeed4913a7d435cfdd1e2d8e P 2f306549407b4bb281ccf2994035e4f1: stopping tablet replica
I20240502 01:53:51.925479 21257 raft_consensus.cc:2229] T 377e804dbeed4913a7d435cfdd1e2d8e P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Raft consensus shutting down.
I20240502 01:53:51.926939 21257 raft_consensus.cc:2258] T 377e804dbeed4913a7d435cfdd1e2d8e P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Raft consensus is shut down!
I20240502 01:53:51.927779 21257 ts_tablet_manager.cc:1875] T 377e804dbeed4913a7d435cfdd1e2d8e P 2f306549407b4bb281ccf2994035e4f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:51.928184 20366 raft_consensus.cc:2454] T aebde87a7f114537abd9c1d8b7e97004 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 1.
I20240502 01:53:51.928753 20433 leader_election.cc:302] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 6a36c952c51542c0bf7ea3ec2857a9f9; no voters:
I20240502 01:53:51.929138 21092 raft_consensus.cc:2781] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:51.932817 17864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aebde87a7f114537abd9c1d8b7e97004" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
W20240502 01:53:51.933921 20434 leader_election.cc:341] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:53:51.934154 21216 leader_election.cc:288] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 election: Requested vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:53:51.938558 20741 ts_tablet_manager.cc:1888] T fa21373c6f4a43df9fd2a5ec2b8d6242 P 4da690c0fa13407a85c0689ce8de9ade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:51.938719 20741 log.cc:1196] T fa21373c6f4a43df9fd2a5ec2b8d6242 P 4da690c0fa13407a85c0689ce8de9ade: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/fa21373c6f4a43df9fd2a5ec2b8d6242
I20240502 01:53:51.939162 20741 ts_tablet_manager.cc:1909] T fa21373c6f4a43df9fd2a5ec2b8d6242 P 4da690c0fa13407a85c0689ce8de9ade: Deleting consensus metadata
I20240502 01:53:51.940183 17321 catalog_manager.cc:4830] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): tablet fa21373c6f4a43df9fd2a5ec2b8d6242 (table rebalance_test_table_16 [id=f27ebc659702427c996fda2492c7e5d0]) successfully deleted
I20240502 01:53:51.948218 21257 ts_tablet_manager.cc:1888] T 377e804dbeed4913a7d435cfdd1e2d8e P 2f306549407b4bb281ccf2994035e4f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:53:51.948365 21257 log.cc:1196] T 377e804dbeed4913a7d435cfdd1e2d8e P 2f306549407b4bb281ccf2994035e4f1: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal/wals/377e804dbeed4913a7d435cfdd1e2d8e
I20240502 01:53:51.948814 21257 ts_tablet_manager.cc:1909] T 377e804dbeed4913a7d435cfdd1e2d8e P 2f306549407b4bb281ccf2994035e4f1: Deleting consensus metadata
I20240502 01:53:51.950299 20741 tablet_replica.cc:330] T 377e804dbeed4913a7d435cfdd1e2d8e P 4da690c0fa13407a85c0689ce8de9ade: stopping tablet replica
I20240502 01:53:51.950505 20741 raft_consensus.cc:2229] T 377e804dbeed4913a7d435cfdd1e2d8e P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:51.950776 20741 raft_consensus.cc:2258] T 377e804dbeed4913a7d435cfdd1e2d8e P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:51.951473 20741 ts_tablet_manager.cc:1875] T 377e804dbeed4913a7d435cfdd1e2d8e P 4da690c0fa13407a85c0689ce8de9ade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:51.954471 17327 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:53:51.957387 20737 raft_consensus.cc:2229] T 27815f771d5b4bbaa7a70d5ea30b279b P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:51.957788 20737 raft_consensus.cc:2258] T 27815f771d5b4bbaa7a70d5ea30b279b P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:51.958307 21092 raft_consensus.cc:686] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Running, Role: LEADER
I20240502 01:53:51.958635 20737 ts_tablet_manager.cc:1875] T 27815f771d5b4bbaa7a70d5ea30b279b P 3fceccefba274b1dbbd8216af8b0bc29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:51.958863 21092 consensus_queue.cc:237] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:51.964478 20877 raft_consensus.cc:482] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:51.964659 20877 raft_consensus.cc:504] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:51.964939 21241 tablet.cc:1722] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 6a36c952c51542c0bf7ea3ec2857a9f9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:53:51.965333 20877 leader_election.cc:288] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:53:51.965481 21241 tablet.cc:1620] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 6a36c952c51542c0bf7ea3ec2857a9f9: MemRowSet was empty: no flush needed.
I20240502 01:53:51.965718 21246 tablet.cc:1722] T b39ec4b13e2a423f8c386f4c8bb6fe05 P fdc2ac372492454c85387589a91f63af: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:53:51.966151 21246 tablet.cc:1620] T b39ec4b13e2a423f8c386f4c8bb6fe05 P fdc2ac372492454c85387589a91f63af: MemRowSet was empty: no flush needed.
I20240502 01:53:51.969307 17867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31a4ca0cc44e492fbd13a70ce38add35" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
W20240502 01:53:51.970132 20303 leader_election.cc:341] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:53:51.970430 20233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31a4ca0cc44e492fbd13a70ce38add35" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:53:51.970633 20233 raft_consensus.cc:2454] T 31a4ca0cc44e492fbd13a70ce38add35 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 0.
I20240502 01:53:51.971086 20303 leader_election.cc:302] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, fdc2ac372492454c85387589a91f63af; no voters: bb31bceba76d466db24b9341c04900c3
I20240502 01:53:51.971637 20877 raft_consensus.cc:2781] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:51.971733 20877 raft_consensus.cc:482] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:51.971829 20877 raft_consensus.cc:3037] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:51.974545 17327 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=e33e661e202249fca162bb1e059c31a8] alter complete (version 2)
I20240502 01:53:51.974620 20877 raft_consensus.cc:504] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:51.975435 20877 leader_election.cc:288] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Requested vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:53:51.976066 20233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31a4ca0cc44e492fbd13a70ce38add35" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:53:51.976286 20233 raft_consensus.cc:3037] T 31a4ca0cc44e492fbd13a70ce38add35 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:51.979383 20233 raft_consensus.cc:2454] T 31a4ca0cc44e492fbd13a70ce38add35 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 1.
I20240502 01:53:51.980547 20211 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:51.980882 21247 tablet_bootstrap.cc:492] T 31a4ca0cc44e492fbd13a70ce38add35 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:51.981171 20303 leader_election.cc:302] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, fdc2ac372492454c85387589a91f63af; no voters:
W20240502 01:53:51.983623 17670 consensus_peers.cc:485] T fa21373c6f4a43df9fd2a5ec2b8d6242 P dfa1152987644bd781b46c7e42a7a685 -> Peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): Couldn't send request to peer 3fceccefba274b1dbbd8216af8b0bc29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fa21373c6f4a43df9fd2a5ec2b8d6242. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:51.983966 20877 raft_consensus.cc:2781] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:51.984711 17865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31a4ca0cc44e492fbd13a70ce38add35" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:53:51.986501 21172 raft_consensus.cc:2932] T 5b092dd1cc004c26a33e17c28af4d3ba P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:53:51.989671 17735 raft_consensus.cc:2932] T 5b092dd1cc004c26a33e17c28af4d3ba P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:53:51.990389 17322 catalog_manager.cc:4830] TS 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): tablet 377e804dbeed4913a7d435cfdd1e2d8e (table rebalance_test_table_16 [id=f27ebc659702427c996fda2492c7e5d0]) successfully deleted
W20240502 01:53:51.992676 20303 leader_election.cc:341] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:53:51.995875 17332 catalog_manager.cc:5484] T 5b092dd1cc004c26a33e17c28af4d3ba P dfa1152987644bd781b46c7e42a7a685 reported cstate change: config changed from index 4 to 5, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:53:51.996778 21242 tablet.cc:1722] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 2f306549407b4bb281ccf2994035e4f1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:53:51.997275 21242 tablet.cc:1620] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 2f306549407b4bb281ccf2994035e4f1: MemRowSet was empty: no flush needed.
I20240502 01:53:51.997701 20996 raft_consensus.cc:2229] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Raft consensus shutting down.
W20240502 01:53:52.001729 17803 consensus_peers.cc:485] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3 -> Peer 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): Couldn't send request to peer 2f306549407b4bb281ccf2994035e4f1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 377e804dbeed4913a7d435cfdd1e2d8e. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:52.002404 17604 raft_consensus.cc:2932] T 5b092dd1cc004c26a33e17c28af4d3ba P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:53:52.008056 17475 consensus_queue.cc:237] T 5b092dd1cc004c26a33e17c28af4d3ba P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } }
W20240502 01:53:52.011336 17802 consensus_peers.cc:485] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3 -> Peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Couldn't send request to peer 4da690c0fa13407a85c0689ce8de9ade. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:52.012606 17735 raft_consensus.cc:1261] T 377e804dbeed4913a7d435cfdd1e2d8e P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:52.013419 20996 pending_rounds.cc:62] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3: Trying to abort 1 pending ops.
I20240502 01:53:52.013098 20367 raft_consensus.cc:2932] T 5b092dd1cc004c26a33e17c28af4d3ba P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:53:52.013543 20996 pending_rounds.cc:68] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023062350862913536 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "377e804dbeed4913a7d435cfdd1e2d8e" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } } }
I20240502 01:53:52.014011 20996 raft_consensus.cc:2894] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240502 01:53:52.014117 20996 raft_consensus.cc:2866] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 01:53:52.014214 20996 raft_consensus.cc:2258] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:52.022624 20741 ts_tablet_manager.cc:1888] T 377e804dbeed4913a7d435cfdd1e2d8e P 4da690c0fa13407a85c0689ce8de9ade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:53:52.022780 20741 log.cc:1196] T 377e804dbeed4913a7d435cfdd1e2d8e P 4da690c0fa13407a85c0689ce8de9ade: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/377e804dbeed4913a7d435cfdd1e2d8e
I20240502 01:53:52.023196 20741 ts_tablet_manager.cc:1909] T 377e804dbeed4913a7d435cfdd1e2d8e P 4da690c0fa13407a85c0689ce8de9ade: Deleting consensus metadata
I20240502 01:53:52.024188 17321 catalog_manager.cc:4830] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): tablet 377e804dbeed4913a7d435cfdd1e2d8e (table rebalance_test_table_16 [id=f27ebc659702427c996fda2492c7e5d0]) successfully deleted
I20240502 01:53:52.025470 21256 tablet_replica.cc:330] T fa21373c6f4a43df9fd2a5ec2b8d6242 P dfa1152987644bd781b46c7e42a7a685: stopping tablet replica
I20240502 01:53:52.025692 21256 raft_consensus.cc:2229] T fa21373c6f4a43df9fd2a5ec2b8d6242 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:52.026069 21256 raft_consensus.cc:2258] T fa21373c6f4a43df9fd2a5ec2b8d6242 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:52.026785 21256 ts_tablet_manager.cc:1875] T fa21373c6f4a43df9fd2a5ec2b8d6242 P dfa1152987644bd781b46c7e42a7a685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:52.032294 17605 raft_consensus.cc:1261] T 5b092dd1cc004c26a33e17c28af4d3ba P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:53:52.032822 21247 tablet_bootstrap.cc:654] T 31a4ca0cc44e492fbd13a70ce38add35 P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:52.033937 21261 ts_tablet_manager.cc:902] T 31aa6a26f64f49368ad43cc97c66f2f1 P bb31bceba76d466db24b9341c04900c3: Initiating tablet copy from peer 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:53:52.037420 21264 consensus_queue.cc:1035] T 5b092dd1cc004c26a33e17c28af4d3ba P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:53:52.042260 21172 raft_consensus.cc:2932] T 5b092dd1cc004c26a33e17c28af4d3ba P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:53:52.045511 17605 raft_consensus.cc:2932] T 5b092dd1cc004c26a33e17c28af4d3ba P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:53:52.050119 20737 ts_tablet_manager.cc:1888] T 27815f771d5b4bbaa7a70d5ea30b279b P 3fceccefba274b1dbbd8216af8b0bc29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:52.052779 20366 raft_consensus.cc:1261] T 5b092dd1cc004c26a33e17c28af4d3ba P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 01:53:52.055877 17800 consensus_peers.cc:485] T 27815f771d5b4bbaa7a70d5ea30b279b P bb31bceba76d466db24b9341c04900c3 -> Peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): Couldn't send request to peer 3fceccefba274b1dbbd8216af8b0bc29. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:52.055575 17327 catalog_manager.cc:5484] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71). New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:52.059509 20877 raft_consensus.cc:686] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Becoming Leader. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Running, Role: LEADER
I20240502 01:53:52.059819 20877 consensus_queue.cc:237] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:52.060197 20737 log.cc:1196] T 27815f771d5b4bbaa7a70d5ea30b279b P 3fceccefba274b1dbbd8216af8b0bc29: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/wals/27815f771d5b4bbaa7a70d5ea30b279b
I20240502 01:53:52.062391 17332 catalog_manager.cc:5484] T 5b092dd1cc004c26a33e17c28af4d3ba P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: config changed from index 5 to 6, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New cstate: current_term: 1 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:53:52.063248 20737 ts_tablet_manager.cc:1909] T 27815f771d5b4bbaa7a70d5ea30b279b P 3fceccefba274b1dbbd8216af8b0bc29: Deleting consensus metadata
I20240502 01:53:52.067401 17716 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b092dd1cc004c26a33e17c28af4d3ba with delete_type TABLET_DATA_TOMBSTONED (TS dfa1152987644bd781b46c7e42a7a685 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52198
I20240502 01:53:52.068646 17327 catalog_manager.cc:5484] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71). New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:52.069458 21172 consensus_queue.cc:1035] T 5b092dd1cc004c26a33e17c28af4d3ba P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:53:52.072144 20367 raft_consensus.cc:2932] T 5b092dd1cc004c26a33e17c28af4d3ba P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:53:52.073180 20996 ts_tablet_manager.cc:1875] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:52.079965 21040 raft_consensus.cc:1050] T 52430bbf347a4c5191edb23e1aa5d385 P dfa1152987644bd781b46c7e42a7a685: attempting to promote NON_VOTER fdc2ac372492454c85387589a91f63af to VOTER
I20240502 01:53:52.080529 21256 ts_tablet_manager.cc:1888] T fa21373c6f4a43df9fd2a5ec2b8d6242 P dfa1152987644bd781b46c7e42a7a685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:52.080660 21256 log.cc:1196] T fa21373c6f4a43df9fd2a5ec2b8d6242 P dfa1152987644bd781b46c7e42a7a685: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/wals/fa21373c6f4a43df9fd2a5ec2b8d6242
I20240502 01:53:52.081063 21040 consensus_queue.cc:237] T 52430bbf347a4c5191edb23e1aa5d385 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } }
I20240502 01:53:52.083832 21256 ts_tablet_manager.cc:1909] T fa21373c6f4a43df9fd2a5ec2b8d6242 P dfa1152987644bd781b46c7e42a7a685: Deleting consensus metadata
I20240502 01:53:52.083878 21247 tablet_bootstrap.cc:492] T 31a4ca0cc44e492fbd13a70ce38add35 P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:52.084024 21247 ts_tablet_manager.cc:1367] T 31a4ca0cc44e492fbd13a70ce38add35 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.103s user 0.006s sys 0.010s
I20240502 01:53:52.084839 21247 raft_consensus.cc:348] T 31a4ca0cc44e492fbd13a70ce38add35 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:52.085053 21247 raft_consensus.cc:374] T 31a4ca0cc44e492fbd13a70ce38add35 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:52.085135 21247 raft_consensus.cc:729] T 31a4ca0cc44e492fbd13a70ce38add35 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:52.085362 21247 consensus_queue.cc:260] T 31a4ca0cc44e492fbd13a70ce38add35 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:52.087430 21247 ts_tablet_manager.cc:1398] T 31a4ca0cc44e492fbd13a70ce38add35 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240502 01:53:52.087767 21247 tablet_bootstrap.cc:492] T aebde87a7f114537abd9c1d8b7e97004 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:52.090533 21247 tablet_bootstrap.cc:654] T aebde87a7f114537abd9c1d8b7e97004 P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:52.092237 21256 tablet_replica.cc:330] T 377e804dbeed4913a7d435cfdd1e2d8e P dfa1152987644bd781b46c7e42a7a685: stopping tablet replica
I20240502 01:53:52.092500 21256 raft_consensus.cc:2229] T 377e804dbeed4913a7d435cfdd1e2d8e P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:52.092732 21256 raft_consensus.cc:2258] T 377e804dbeed4913a7d435cfdd1e2d8e P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:52.069540 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5b092dd1cc004c26a33e17c28af4d3ba with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:52.070127 17319 catalog_manager.cc:4830] TS 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): tablet 27815f771d5b4bbaa7a70d5ea30b279b (table rebalance_test_table_16 [id=f27ebc659702427c996fda2492c7e5d0]) successfully deleted
I20240502 01:53:52.085074 17335 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:53:52.093803 17321 catalog_manager.cc:4830] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): tablet fa21373c6f4a43df9fd2a5ec2b8d6242 (table rebalance_test_table_16 [id=f27ebc659702427c996fda2492c7e5d0]) successfully deleted
I20240502 01:53:52.095718 20996 ts_tablet_manager.cc:1888] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240502 01:53:52.095610 17332 catalog_manager.cc:5484] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70). New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:52.099936 21256 ts_tablet_manager.cc:1875] T 377e804dbeed4913a7d435cfdd1e2d8e P dfa1152987644bd781b46c7e42a7a685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:52.101327 20996 log.cc:1196] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/wals/377e804dbeed4913a7d435cfdd1e2d8e
I20240502 01:53:52.102077 20996 ts_tablet_manager.cc:1909] T 377e804dbeed4913a7d435cfdd1e2d8e P bb31bceba76d466db24b9341c04900c3: Deleting consensus metadata
I20240502 01:53:52.103173 20996 tablet_replica.cc:330] T 27815f771d5b4bbaa7a70d5ea30b279b P bb31bceba76d466db24b9341c04900c3: stopping tablet replica
I20240502 01:53:52.103386 20996 raft_consensus.cc:2229] T 27815f771d5b4bbaa7a70d5ea30b279b P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:52.103693 20996 raft_consensus.cc:2258] T 27815f771d5b4bbaa7a70d5ea30b279b P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:52.104395 20996 ts_tablet_manager.cc:1875] T 27815f771d5b4bbaa7a70d5ea30b279b P bb31bceba76d466db24b9341c04900c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:52.105676 17322 catalog_manager.cc:4830] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): tablet 377e804dbeed4913a7d435cfdd1e2d8e (table rebalance_test_table_16 [id=f27ebc659702427c996fda2492c7e5d0]) successfully deleted
I20240502 01:53:52.107632 21241 tablet.cc:1722] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 6a36c952c51542c0bf7ea3ec2857a9f9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:53:52.112972 21242 tablet.cc:1722] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 2f306549407b4bb281ccf2994035e4f1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:53:52.120193 20996 ts_tablet_manager.cc:1888] T 27815f771d5b4bbaa7a70d5ea30b279b P bb31bceba76d466db24b9341c04900c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:52.121204 21246 tablet.cc:1722] T b39ec4b13e2a423f8c386f4c8bb6fe05 P fdc2ac372492454c85387589a91f63af: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:53:52.124092 20232 raft_consensus.cc:1261] T 52430bbf347a4c5191edb23e1aa5d385 P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:52.124627 17605 raft_consensus.cc:1261] T 52430bbf347a4c5191edb23e1aa5d385 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:52.127408 17332 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table_new_name [id=e33e661e202249fca162bb1e059c31a8] alter complete (version 3)
I20240502 01:53:52.125574 21261 tablet_copy_client.cc:308] T 31aa6a26f64f49368ad43cc97c66f2f1 P bb31bceba76d466db24b9341c04900c3: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.70:39155
I20240502 01:53:52.125650 21166 consensus_queue.cc:1035] T 52430bbf347a4c5191edb23e1aa5d385 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:52.145135 21275 consensus_queue.cc:1035] T 52430bbf347a4c5191edb23e1aa5d385 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:52.146940 17475 raft_consensus.cc:1261] T 52430bbf347a4c5191edb23e1aa5d385 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:52.148526 21275 consensus_queue.cc:1035] T 52430bbf347a4c5191edb23e1aa5d385 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:52.165231 20996 log.cc:1196] T 27815f771d5b4bbaa7a70d5ea30b279b P bb31bceba76d466db24b9341c04900c3: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/wals/27815f771d5b4bbaa7a70d5ea30b279b
I20240502 01:53:52.165763 20996 ts_tablet_manager.cc:1909] T 27815f771d5b4bbaa7a70d5ea30b279b P bb31bceba76d466db24b9341c04900c3: Deleting consensus metadata
I20240502 01:53:52.172672 17322 catalog_manager.cc:4830] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): tablet 27815f771d5b4bbaa7a70d5ea30b279b (table rebalance_test_table_16 [id=f27ebc659702427c996fda2492c7e5d0]) successfully deleted
I20240502 01:53:52.179877 17605 raft_consensus.cc:2932] T 52430bbf347a4c5191edb23e1aa5d385 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:53:52.182436 21275 raft_consensus.cc:2932] T 52430bbf347a4c5191edb23e1aa5d385 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:53:52.185679 17474 raft_consensus.cc:2932] T 52430bbf347a4c5191edb23e1aa5d385 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:53:52.166179 21247 tablet_bootstrap.cc:492] T aebde87a7f114537abd9c1d8b7e97004 P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:52.187763 21247 ts_tablet_manager.cc:1367] T aebde87a7f114537abd9c1d8b7e97004 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.100s user 0.009s sys 0.014s
I20240502 01:53:52.188002 20233 raft_consensus.cc:2932] T 52430bbf347a4c5191edb23e1aa5d385 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:53:52.188737 21247 raft_consensus.cc:348] T aebde87a7f114537abd9c1d8b7e97004 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:52.189108 21247 raft_consensus.cc:374] T aebde87a7f114537abd9c1d8b7e97004 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:52.189200 21247 raft_consensus.cc:729] T aebde87a7f114537abd9c1d8b7e97004 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:52.189494 21247 consensus_queue.cc:260] T aebde87a7f114537abd9c1d8b7e97004 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:52.191035 21247 ts_tablet_manager.cc:1398] T aebde87a7f114537abd9c1d8b7e97004 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:53:52.193631 17335 catalog_manager.cc:5484] T 52430bbf347a4c5191edb23e1aa5d385 P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: config changed from index 4 to 5, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:53:52.199142 17736 consensus_queue.cc:237] T 52430bbf347a4c5191edb23e1aa5d385 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } }
I20240502 01:53:52.201838 20232 raft_consensus.cc:1261] T 52430bbf347a4c5191edb23e1aa5d385 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:53:52.201936 17475 raft_consensus.cc:1261] T 52430bbf347a4c5191edb23e1aa5d385 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:53:52.202520 21277 consensus_queue.cc:1035] T 52430bbf347a4c5191edb23e1aa5d385 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:53:52.203423 21166 consensus_queue.cc:1035] T 52430bbf347a4c5191edb23e1aa5d385 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:53:52.205159 21166 raft_consensus.cc:2932] T 52430bbf347a4c5191edb23e1aa5d385 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:53:52.206282 20232 raft_consensus.cc:2932] T 52430bbf347a4c5191edb23e1aa5d385 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:53:52.210188 20387 tablet_copy_service.cc:140] P 6a36c952c51542c0bf7ea3ec2857a9f9: Received BeginTabletCopySession request for tablet 31aa6a26f64f49368ad43cc97c66f2f1 from peer bb31bceba76d466db24b9341c04900c3 ({username='slave'} at 127.11.42.68:34031)
I20240502 01:53:52.210443 20387 tablet_copy_service.cc:161] P 6a36c952c51542c0bf7ea3ec2857a9f9: Beginning new tablet copy session on tablet 31aa6a26f64f49368ad43cc97c66f2f1 from peer bb31bceba76d466db24b9341c04900c3 at {username='slave'} at 127.11.42.68:34031: session id = bb31bceba76d466db24b9341c04900c3-31aa6a26f64f49368ad43cc97c66f2f1
I20240502 01:53:52.211923 17335 catalog_manager.cc:5484] T 52430bbf347a4c5191edb23e1aa5d385 P fdc2ac372492454c85387589a91f63af reported cstate change: config changed from index 5 to 6, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:53:52.214795 20387 tablet_copy_source_session.cc:205] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:52.196575 21256 ts_tablet_manager.cc:1888] T 377e804dbeed4913a7d435cfdd1e2d8e P dfa1152987644bd781b46c7e42a7a685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:53:52.215407 21256 log.cc:1196] T 377e804dbeed4913a7d435cfdd1e2d8e P dfa1152987644bd781b46c7e42a7a685: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/wals/377e804dbeed4913a7d435cfdd1e2d8e
I20240502 01:53:52.215934 21256 ts_tablet_manager.cc:1909] T 377e804dbeed4913a7d435cfdd1e2d8e P dfa1152987644bd781b46c7e42a7a685: Deleting consensus metadata
I20240502 01:53:52.216784 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 52430bbf347a4c5191edb23e1aa5d385 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:52.216992 21261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31aa6a26f64f49368ad43cc97c66f2f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:52.217128 17321 catalog_manager.cc:4830] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): tablet 377e804dbeed4913a7d435cfdd1e2d8e (table rebalance_test_table_16 [id=f27ebc659702427c996fda2492c7e5d0]) successfully deleted
I20240502 01:53:52.217792 17475 raft_consensus.cc:2932] T 52430bbf347a4c5191edb23e1aa5d385 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:53:52.218487 21256 tablet_replica.cc:330] T 5b092dd1cc004c26a33e17c28af4d3ba P dfa1152987644bd781b46c7e42a7a685: stopping tablet replica
I20240502 01:53:52.218716 21256 raft_consensus.cc:2229] T 5b092dd1cc004c26a33e17c28af4d3ba P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:52.218971 21256 raft_consensus.cc:2258] T 5b092dd1cc004c26a33e17c28af4d3ba P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:52.220489 21256 ts_tablet_manager.cc:1875] T 5b092dd1cc004c26a33e17c28af4d3ba P dfa1152987644bd781b46c7e42a7a685: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:52.221585 17587 tablet_service.cc:1514] Processing DeleteTablet for tablet 52430bbf347a4c5191edb23e1aa5d385 with delete_type TABLET_DATA_TOMBSTONED (TS 3fceccefba274b1dbbd8216af8b0bc29 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55232
I20240502 01:53:52.222344 20737 tablet_replica.cc:330] T 52430bbf347a4c5191edb23e1aa5d385 P 3fceccefba274b1dbbd8216af8b0bc29: stopping tablet replica
I20240502 01:53:52.222558 20737 raft_consensus.cc:2229] T 52430bbf347a4c5191edb23e1aa5d385 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:52.222913 20737 raft_consensus.cc:2258] T 52430bbf347a4c5191edb23e1aa5d385 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:52.223937 20737 ts_tablet_manager.cc:1875] T 52430bbf347a4c5191edb23e1aa5d385 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:52.226594 21261 tablet_copy_client.cc:783] T 31aa6a26f64f49368ad43cc97c66f2f1 P bb31bceba76d466db24b9341c04900c3: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:52.226881 21261 tablet_copy_client.cc:647] T 31aa6a26f64f49368ad43cc97c66f2f1 P bb31bceba76d466db24b9341c04900c3: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:52.228914 21261 tablet_copy_client.cc:523] T 31aa6a26f64f49368ad43cc97c66f2f1 P bb31bceba76d466db24b9341c04900c3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:52.231040 20737 ts_tablet_manager.cc:1888] T 52430bbf347a4c5191edb23e1aa5d385 P 3fceccefba274b1dbbd8216af8b0bc29: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:53:52.231231 20737 log.cc:1196] T 52430bbf347a4c5191edb23e1aa5d385 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/wals/52430bbf347a4c5191edb23e1aa5d385
I20240502 01:53:52.232182 17319 catalog_manager.cc:4830] TS 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): tablet 52430bbf347a4c5191edb23e1aa5d385 (table rebalance_test_table_13 [id=dbf97af96f07462286eb1fabeddf25fa]) successfully deleted
I20240502 01:53:52.239727 21261 tablet_bootstrap.cc:492] T 31aa6a26f64f49368ad43cc97c66f2f1 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:52.244385 17335 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:53:52.244601 17335 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:53:52.249944 17335 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet b39ec4b13e2a423f8c386f4c8bb6fe05
I20240502 01:53:52.251009 20347 tablet_service.cc:1514] Processing DeleteTablet for tablet b39ec4b13e2a423f8c386f4c8bb6fe05 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:52 UTC) from {username='slave'} at 127.0.0.1:40196
I20240502 01:53:52.251024 20211 tablet_service.cc:1514] Processing DeleteTablet for tablet b39ec4b13e2a423f8c386f4c8bb6fe05 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:52 UTC) from {username='slave'} at 127.0.0.1:44624
I20240502 01:53:52.252972 21256 ts_tablet_manager.cc:1888] T 5b092dd1cc004c26a33e17c28af4d3ba P dfa1152987644bd781b46c7e42a7a685: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:53:52.253136 21256 log.cc:1196] T 5b092dd1cc004c26a33e17c28af4d3ba P dfa1152987644bd781b46c7e42a7a685: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/wals/5b092dd1cc004c26a33e17c28af4d3ba
I20240502 01:53:52.253813 17321 catalog_manager.cc:4830] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): tablet 5b092dd1cc004c26a33e17c28af4d3ba (table rebalance_test_table_15 [id=21fe1df2eeed4efd801d10edf592ecce]) successfully deleted
I20240502 01:53:52.257447 20478 tablet_service.cc:1514] Processing DeleteTablet for tablet b39ec4b13e2a423f8c386f4c8bb6fe05 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:52 UTC) from {username='slave'} at 127.0.0.1:35018
I20240502 01:53:52.257736 21257 tablet_replica.cc:330] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 2f306549407b4bb281ccf2994035e4f1: stopping tablet replica
I20240502 01:53:52.257937 21257 raft_consensus.cc:2229] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:52.258157 21257 raft_consensus.cc:2258] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:52.259212 21257 ts_tablet_manager.cc:1875] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 2f306549407b4bb281ccf2994035e4f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:52.262363 21282 tablet_replica.cc:330] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 6a36c952c51542c0bf7ea3ec2857a9f9: stopping tablet replica
I20240502 01:53:52.262559 21282 raft_consensus.cc:2229] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:52.262920 21282 raft_consensus.cc:2258] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:52.263767 21282 ts_tablet_manager.cc:1875] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:52.266875 21257 ts_tablet_manager.cc:1888] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 2f306549407b4bb281ccf2994035e4f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:53:52.267081 21257 log.cc:1196] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 2f306549407b4bb281ccf2994035e4f1: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal/wals/b39ec4b13e2a423f8c386f4c8bb6fe05
I20240502 01:53:52.267575 21257 ts_tablet_manager.cc:1909] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 2f306549407b4bb281ccf2994035e4f1: Deleting consensus metadata
I20240502 01:53:52.268568 17322 catalog_manager.cc:4830] TS 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): tablet b39ec4b13e2a423f8c386f4c8bb6fe05 (table rebalancer_dynamic_table_new_name [id=e33e661e202249fca162bb1e059c31a8]) successfully deleted
I20240502 01:53:52.269987 21282 ts_tablet_manager.cc:1888] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:53:52.270246 21282 log.cc:1196] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal/wals/b39ec4b13e2a423f8c386f4c8bb6fe05
I20240502 01:53:52.270799 21282 ts_tablet_manager.cc:1909] T b39ec4b13e2a423f8c386f4c8bb6fe05 P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting consensus metadata
I20240502 01:53:52.271494 21261 tablet_bootstrap.cc:492] T 31aa6a26f64f49368ad43cc97c66f2f1 P bb31bceba76d466db24b9341c04900c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:52.271863 17321 catalog_manager.cc:4830] TS 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): tablet b39ec4b13e2a423f8c386f4c8bb6fe05 (table rebalancer_dynamic_table_new_name [id=e33e661e202249fca162bb1e059c31a8]) successfully deleted
I20240502 01:53:52.272126 21261 tablet_bootstrap.cc:492] T 31aa6a26f64f49368ad43cc97c66f2f1 P bb31bceba76d466db24b9341c04900c3: Bootstrap complete.
I20240502 01:53:52.272352 21261 ts_tablet_manager.cc:1367] T 31aa6a26f64f49368ad43cc97c66f2f1 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.033s user 0.007s sys 0.003s
I20240502 01:53:52.273113 21261 raft_consensus.cc:348] T 31aa6a26f64f49368ad43cc97c66f2f1 P bb31bceba76d466db24b9341c04900c3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } }
I20240502 01:53:52.273362 21261 raft_consensus.cc:729] T 31aa6a26f64f49368ad43cc97c66f2f1 P bb31bceba76d466db24b9341c04900c3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: LEARNER
I20240502 01:53:52.273593 21261 consensus_queue.cc:260] T 31aa6a26f64f49368ad43cc97c66f2f1 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } }
I20240502 01:53:52.274930 21261 ts_tablet_manager.cc:1398] T 31aa6a26f64f49368ad43cc97c66f2f1 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:52.275743 20387 tablet_copy_service.cc:342] P 6a36c952c51542c0bf7ea3ec2857a9f9: Request end of tablet copy session bb31bceba76d466db24b9341c04900c3-31aa6a26f64f49368ad43cc97c66f2f1 received from {username='slave'} at 127.11.42.68:34031
I20240502 01:53:52.275925 20387 tablet_copy_service.cc:433] P 6a36c952c51542c0bf7ea3ec2857a9f9: ending tablet copy session bb31bceba76d466db24b9341c04900c3-31aa6a26f64f49368ad43cc97c66f2f1 on tablet 31aa6a26f64f49368ad43cc97c66f2f1 with peer bb31bceba76d466db24b9341c04900c3
I20240502 01:53:52.279850 21283 tablet_replica.cc:330] T b39ec4b13e2a423f8c386f4c8bb6fe05 P fdc2ac372492454c85387589a91f63af: stopping tablet replica
I20240502 01:53:52.280061 21283 raft_consensus.cc:2229] T b39ec4b13e2a423f8c386f4c8bb6fe05 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:52.280323 21283 raft_consensus.cc:2258] T b39ec4b13e2a423f8c386f4c8bb6fe05 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:52.284214 21260 consensus_queue.cc:1035] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:52.285584 21283 ts_tablet_manager.cc:1875] T b39ec4b13e2a423f8c386f4c8bb6fe05 P fdc2ac372492454c85387589a91f63af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:52.290822 21283 ts_tablet_manager.cc:1888] T b39ec4b13e2a423f8c386f4c8bb6fe05 P fdc2ac372492454c85387589a91f63af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:53:52.290946 21283 log.cc:1196] T b39ec4b13e2a423f8c386f4c8bb6fe05 P fdc2ac372492454c85387589a91f63af: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal/wals/b39ec4b13e2a423f8c386f4c8bb6fe05
I20240502 01:53:52.291303 21283 ts_tablet_manager.cc:1909] T b39ec4b13e2a423f8c386f4c8bb6fe05 P fdc2ac372492454c85387589a91f63af: Deleting consensus metadata
I20240502 01:53:52.292268 17322 catalog_manager.cc:4830] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): tablet b39ec4b13e2a423f8c386f4c8bb6fe05 (table rebalancer_dynamic_table_new_name [id=e33e661e202249fca162bb1e059c31a8]) successfully deleted
I20240502 01:53:52.296001 21092 consensus_queue.cc:1035] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:52.352523 20232 consensus_queue.cc:237] T 62887e332443457f8a5b41a982354dc4 P fdc2ac372492454c85387589a91f63af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } }
I20240502 01:53:52.355276 20498 raft_consensus.cc:1261] T 62887e332443457f8a5b41a982354dc4 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:52.355842 21292 consensus_queue.cc:1035] T 62887e332443457f8a5b41a982354dc4 P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:53:52.356235 20630 raft_consensus.cc:1261] T 62887e332443457f8a5b41a982354dc4 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:52.356750 20768 consensus_queue.cc:1035] T 62887e332443457f8a5b41a982354dc4 P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:53:52.358830 17332 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:53:52.359144 20768 raft_consensus.cc:2932] T 62887e332443457f8a5b41a982354dc4 P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } } }
I20240502 01:53:52.362542 20629 raft_consensus.cc:2932] T 62887e332443457f8a5b41a982354dc4 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } } }
I20240502 01:53:52.367640 17716 tablet_service.cc:1467] Processing CreateTablet for tablet 8ddbb3dba5c04928953f2b41e87d0eed (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1cfb3501e7c3445cacb6edd5bd5f14af]), partition=
I20240502 01:53:52.367833 20609 tablet_service.cc:1467] Processing CreateTablet for tablet 8ddbb3dba5c04928953f2b41e87d0eed (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1cfb3501e7c3445cacb6edd5bd5f14af]), partition=
I20240502 01:53:52.368057 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ddbb3dba5c04928953f2b41e87d0eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:52.368155 20609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ddbb3dba5c04928953f2b41e87d0eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:52.373509 21297 tablet_bootstrap.cc:492] T 8ddbb3dba5c04928953f2b41e87d0eed P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:52.374651 17332 catalog_manager.cc:5484] T 62887e332443457f8a5b41a982354dc4 P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: config changed from index -1 to 2, NON_VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) added. New cstate: current_term: 1 leader_uuid: "fdc2ac372492454c85387589a91f63af" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } } }
I20240502 01:53:52.378533 21253 tablet_bootstrap.cc:492] T 8ddbb3dba5c04928953f2b41e87d0eed P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:53:52.379997 21253 tablet_bootstrap.cc:654] T 8ddbb3dba5c04928953f2b41e87d0eed P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:52.380373 20498 raft_consensus.cc:2932] T 62887e332443457f8a5b41a982354dc4 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } } }
W20240502 01:53:52.381911 20168 consensus_peers.cc:485] T 62887e332443457f8a5b41a982354dc4 P fdc2ac372492454c85387589a91f63af -> Peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): Couldn't send request to peer bb31bceba76d466db24b9341c04900c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62887e332443457f8a5b41a982354dc4. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:52.381914 20211 tablet_service.cc:1467] Processing CreateTablet for tablet 8ddbb3dba5c04928953f2b41e87d0eed (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1cfb3501e7c3445cacb6edd5bd5f14af]), partition=
I20240502 01:53:52.382352 20211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ddbb3dba5c04928953f2b41e87d0eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:52.387763 17335 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240502 01:53:52.387992 17335 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240502 01:53:52.389686 21297 tablet_bootstrap.cc:654] T 8ddbb3dba5c04928953f2b41e87d0eed P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:52.393132 17335 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet 5c56837c0a2e49f69fab0aaf01d113bd
I20240502 01:53:52.394425 17846 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c56837c0a2e49f69fab0aaf01d113bd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:52 UTC) from {username='slave'} at 127.0.0.1:47226
I20240502 01:53:52.394726 20996 tablet_replica.cc:330] T 5c56837c0a2e49f69fab0aaf01d113bd P bb31bceba76d466db24b9341c04900c3: stopping tablet replica
I20240502 01:53:52.394943 20996 raft_consensus.cc:2229] T 5c56837c0a2e49f69fab0aaf01d113bd P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:52.395229 20996 raft_consensus.cc:2258] T 5c56837c0a2e49f69fab0aaf01d113bd P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:52.396006 20996 ts_tablet_manager.cc:1875] T 5c56837c0a2e49f69fab0aaf01d113bd P bb31bceba76d466db24b9341c04900c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:52.396965 21253 tablet_bootstrap.cc:492] T 8ddbb3dba5c04928953f2b41e87d0eed P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:53:52.397161 21253 ts_tablet_manager.cc:1367] T 8ddbb3dba5c04928953f2b41e87d0eed P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.003s
I20240502 01:53:52.398146 21253 raft_consensus.cc:348] T 8ddbb3dba5c04928953f2b41e87d0eed P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:52.398511 21253 raft_consensus.cc:374] T 8ddbb3dba5c04928953f2b41e87d0eed P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:52.398629 21253 raft_consensus.cc:729] T 8ddbb3dba5c04928953f2b41e87d0eed P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:53:52.398972 21253 consensus_queue.cc:260] T 8ddbb3dba5c04928953f2b41e87d0eed P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:52.400401 21253 ts_tablet_manager.cc:1398] T 8ddbb3dba5c04928953f2b41e87d0eed P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:52.402878 21125 raft_consensus.cc:482] T 8ddbb3dba5c04928953f2b41e87d0eed P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:52.403054 21125 raft_consensus.cc:504] T 8ddbb3dba5c04928953f2b41e87d0eed P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:52.403836 21125 leader_election.cc:288] T 8ddbb3dba5c04928953f2b41e87d0eed P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:53:52.404807 17454 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c56837c0a2e49f69fab0aaf01d113bd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:52 UTC) from {username='slave'} at 127.0.0.1:54520
I20240502 01:53:52.405148 20741 tablet_replica.cc:330] T 5c56837c0a2e49f69fab0aaf01d113bd P 4da690c0fa13407a85c0689ce8de9ade: stopping tablet replica
I20240502 01:53:52.405411 20741 raft_consensus.cc:2229] T 5c56837c0a2e49f69fab0aaf01d113bd P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:52.405711 20741 raft_consensus.cc:2258] T 5c56837c0a2e49f69fab0aaf01d113bd P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:52.406509 20741 ts_tablet_manager.cc:1875] T 5c56837c0a2e49f69fab0aaf01d113bd P 4da690c0fa13407a85c0689ce8de9ade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:52.406855 17587 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c56837c0a2e49f69fab0aaf01d113bd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:52 UTC) from {username='slave'} at 127.0.0.1:55232
I20240502 01:53:52.407160 20737 tablet_replica.cc:330] T 5c56837c0a2e49f69fab0aaf01d113bd P 3fceccefba274b1dbbd8216af8b0bc29: stopping tablet replica
I20240502 01:53:52.406919 20232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ddbb3dba5c04928953f2b41e87d0eed" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:53:52.407155 17335 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet 9fe583045886494ca8cf656ad2f83189
I20240502 01:53:52.407394 20737 raft_consensus.cc:2229] T 5c56837c0a2e49f69fab0aaf01d113bd P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Raft consensus shutting down.
W20240502 01:53:52.407683 20565 leader_election.cc:341] T 8ddbb3dba5c04928953f2b41e87d0eed P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:53:52.407775 20737 raft_consensus.cc:2258] T 5c56837c0a2e49f69fab0aaf01d113bd P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:52.408075 17846 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fe583045886494ca8cf656ad2f83189 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:52 UTC) from {username='slave'} at 127.0.0.1:47226
I20240502 01:53:52.408592 20737 ts_tablet_manager.cc:1875] T 5c56837c0a2e49f69fab0aaf01d113bd P 3fceccefba274b1dbbd8216af8b0bc29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:52.409492 17335 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet 1d4efb3b1e3e4fa3bde6305ac993596e
I20240502 01:53:52.409571 17716 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fe583045886494ca8cf656ad2f83189 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:52 UTC) from {username='slave'} at 127.0.0.1:52198
I20240502 01:53:52.409910 21256 tablet_replica.cc:330] T 9fe583045886494ca8cf656ad2f83189 P dfa1152987644bd781b46c7e42a7a685: stopping tablet replica
I20240502 01:53:52.410100 21256 raft_consensus.cc:2229] T 9fe583045886494ca8cf656ad2f83189 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:52.410148 17456 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d4efb3b1e3e4fa3bde6305ac993596e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:52 UTC) from {username='slave'} at 127.0.0.1:54520
I20240502 01:53:52.410480 21256 raft_consensus.cc:2258] T 9fe583045886494ca8cf656ad2f83189 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:52.410897 17587 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d4efb3b1e3e4fa3bde6305ac993596e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:52 UTC) from {username='slave'} at 127.0.0.1:55232
I20240502 01:53:52.411150 21256 ts_tablet_manager.cc:1875] T 9fe583045886494ca8cf656ad2f83189 P dfa1152987644bd781b46c7e42a7a685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:52.411605 17846 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d4efb3b1e3e4fa3bde6305ac993596e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:52 UTC) from {username='slave'} at 127.0.0.1:47226
I20240502 01:53:52.412760 17454 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fe583045886494ca8cf656ad2f83189 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:52 UTC) from {username='slave'} at 127.0.0.1:54520
I20240502 01:53:52.413064 20741 ts_tablet_manager.cc:1888] T 5c56837c0a2e49f69fab0aaf01d113bd P 4da690c0fa13407a85c0689ce8de9ade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:52.413244 20741 log.cc:1196] T 5c56837c0a2e49f69fab0aaf01d113bd P 4da690c0fa13407a85c0689ce8de9ade: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/5c56837c0a2e49f69fab0aaf01d113bd
I20240502 01:53:52.413699 20741 ts_tablet_manager.cc:1909] T 5c56837c0a2e49f69fab0aaf01d113bd P 4da690c0fa13407a85c0689ce8de9ade: Deleting consensus metadata
I20240502 01:53:52.414609 20741 tablet_replica.cc:330] T 1d4efb3b1e3e4fa3bde6305ac993596e P 4da690c0fa13407a85c0689ce8de9ade: stopping tablet replica
I20240502 01:53:52.414727 17321 catalog_manager.cc:4830] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): tablet 5c56837c0a2e49f69fab0aaf01d113bd (table rebalance_test_table_18 [id=fa75b6d8fec44d1998a1c84ded9c9998]) successfully deleted
I20240502 01:53:52.414803 20741 raft_consensus.cc:2229] T 1d4efb3b1e3e4fa3bde6305ac993596e P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:52.415014 20741 raft_consensus.cc:2258] T 1d4efb3b1e3e4fa3bde6305ac993596e P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:52.415325 17864 raft_consensus.cc:3037] T aebde87a7f114537abd9c1d8b7e97004 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:52.415670 20741 ts_tablet_manager.cc:1875] T 1d4efb3b1e3e4fa3bde6305ac993596e P 4da690c0fa13407a85c0689ce8de9ade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:52.418253 21256 ts_tablet_manager.cc:1888] T 9fe583045886494ca8cf656ad2f83189 P dfa1152987644bd781b46c7e42a7a685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:52.418381 21256 log.cc:1196] T 9fe583045886494ca8cf656ad2f83189 P dfa1152987644bd781b46c7e42a7a685: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/wals/9fe583045886494ca8cf656ad2f83189
I20240502 01:53:52.418785 21256 ts_tablet_manager.cc:1909] T 9fe583045886494ca8cf656ad2f83189 P dfa1152987644bd781b46c7e42a7a685: Deleting consensus metadata
I20240502 01:53:52.419818 17321 catalog_manager.cc:4830] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): tablet 9fe583045886494ca8cf656ad2f83189 (table rebalance_test_table_18 [id=fa75b6d8fec44d1998a1c84ded9c9998]) successfully deleted
I20240502 01:53:52.420079 21092 consensus_queue.cc:1035] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:52.424285 20996 ts_tablet_manager.cc:1888] T 5c56837c0a2e49f69fab0aaf01d113bd P bb31bceba76d466db24b9341c04900c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:52.424961 20996 log.cc:1196] T 5c56837c0a2e49f69fab0aaf01d113bd P bb31bceba76d466db24b9341c04900c3: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/wals/5c56837c0a2e49f69fab0aaf01d113bd
I20240502 01:53:52.425788 20737 ts_tablet_manager.cc:1888] T 5c56837c0a2e49f69fab0aaf01d113bd P 3fceccefba274b1dbbd8216af8b0bc29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:52.426229 20996 ts_tablet_manager.cc:1909] T 5c56837c0a2e49f69fab0aaf01d113bd P bb31bceba76d466db24b9341c04900c3: Deleting consensus metadata
I20240502 01:53:52.427279 17322 catalog_manager.cc:4830] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): tablet 5c56837c0a2e49f69fab0aaf01d113bd (table rebalance_test_table_18 [id=fa75b6d8fec44d1998a1c84ded9c9998]) successfully deleted
I20240502 01:53:52.428517 20996 tablet_replica.cc:330] T 9fe583045886494ca8cf656ad2f83189 P bb31bceba76d466db24b9341c04900c3: stopping tablet replica
I20240502 01:53:52.428752 20996 raft_consensus.cc:2229] T 9fe583045886494ca8cf656ad2f83189 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:52.429514 20996 raft_consensus.cc:2258] T 9fe583045886494ca8cf656ad2f83189 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:52.428861 20741 ts_tablet_manager.cc:1888] T 1d4efb3b1e3e4fa3bde6305ac993596e P 4da690c0fa13407a85c0689ce8de9ade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:52.429718 20741 log.cc:1196] T 1d4efb3b1e3e4fa3bde6305ac993596e P 4da690c0fa13407a85c0689ce8de9ade: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/1d4efb3b1e3e4fa3bde6305ac993596e
I20240502 01:53:52.430150 20741 ts_tablet_manager.cc:1909] T 1d4efb3b1e3e4fa3bde6305ac993596e P 4da690c0fa13407a85c0689ce8de9ade: Deleting consensus metadata
I20240502 01:53:52.430197 20996 ts_tablet_manager.cc:1875] T 9fe583045886494ca8cf656ad2f83189 P bb31bceba76d466db24b9341c04900c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:52.431082 17321 catalog_manager.cc:4830] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): tablet 1d4efb3b1e3e4fa3bde6305ac993596e (table rebalance_test_table_18 [id=fa75b6d8fec44d1998a1c84ded9c9998]) successfully deleted
I20240502 01:53:52.432859 20741 tablet_replica.cc:330] T 9fe583045886494ca8cf656ad2f83189 P 4da690c0fa13407a85c0689ce8de9ade: stopping tablet replica
I20240502 01:53:52.433183 20737 log.cc:1196] T 5c56837c0a2e49f69fab0aaf01d113bd P 3fceccefba274b1dbbd8216af8b0bc29: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/wals/5c56837c0a2e49f69fab0aaf01d113bd
I20240502 01:53:52.433881 20737 ts_tablet_manager.cc:1909] T 5c56837c0a2e49f69fab0aaf01d113bd P 3fceccefba274b1dbbd8216af8b0bc29: Deleting consensus metadata
I20240502 01:53:52.434983 17319 catalog_manager.cc:4830] TS 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): tablet 5c56837c0a2e49f69fab0aaf01d113bd (table rebalance_test_table_18 [id=fa75b6d8fec44d1998a1c84ded9c9998]) successfully deleted
I20240502 01:53:52.435137 20741 raft_consensus.cc:2229] T 9fe583045886494ca8cf656ad2f83189 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:52.435375 20741 raft_consensus.cc:2258] T 9fe583045886494ca8cf656ad2f83189 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:52.435470 20737 tablet_replica.cc:330] T 1d4efb3b1e3e4fa3bde6305ac993596e P 3fceccefba274b1dbbd8216af8b0bc29: stopping tablet replica
I20240502 01:53:52.435746 20737 raft_consensus.cc:2229] T 1d4efb3b1e3e4fa3bde6305ac993596e P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:52.436025 20737 raft_consensus.cc:2258] T 1d4efb3b1e3e4fa3bde6305ac993596e P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:52.436774 20741 ts_tablet_manager.cc:1875] T 9fe583045886494ca8cf656ad2f83189 P 4da690c0fa13407a85c0689ce8de9ade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:52.436776 20737 ts_tablet_manager.cc:1875] T 1d4efb3b1e3e4fa3bde6305ac993596e P 3fceccefba274b1dbbd8216af8b0bc29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:52.443143 21297 tablet_bootstrap.cc:492] T 8ddbb3dba5c04928953f2b41e87d0eed P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:52.443369 21297 ts_tablet_manager.cc:1367] T 8ddbb3dba5c04928953f2b41e87d0eed P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.070s user 0.005s sys 0.000s
I20240502 01:53:52.444294 20996 ts_tablet_manager.cc:1888] T 9fe583045886494ca8cf656ad2f83189 P bb31bceba76d466db24b9341c04900c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:52.444444 20996 log.cc:1196] T 9fe583045886494ca8cf656ad2f83189 P bb31bceba76d466db24b9341c04900c3: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/wals/9fe583045886494ca8cf656ad2f83189
I20240502 01:53:52.444226 21297 raft_consensus.cc:348] T 8ddbb3dba5c04928953f2b41e87d0eed P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:52.444859 21297 raft_consensus.cc:374] T 8ddbb3dba5c04928953f2b41e87d0eed P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:52.445005 21297 raft_consensus.cc:729] T 8ddbb3dba5c04928953f2b41e87d0eed P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:52.445317 21297 consensus_queue.cc:260] T 8ddbb3dba5c04928953f2b41e87d0eed P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:52.446606 21259 raft_consensus.cc:482] T 31a4ca0cc44e492fbd13a70ce38add35 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:52.446776 21259 raft_consensus.cc:504] T 31a4ca0cc44e492fbd13a70ce38add35 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:52.447269 21297 ts_tablet_manager.cc:1398] T 8ddbb3dba5c04928953f2b41e87d0eed P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:53:52.447734 21259 leader_election.cc:288] T 31a4ca0cc44e492fbd13a70ce38add35 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:53:52.449509 20367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31a4ca0cc44e492fbd13a70ce38add35" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:53:52.455179 20996 ts_tablet_manager.cc:1909] T 9fe583045886494ca8cf656ad2f83189 P bb31bceba76d466db24b9341c04900c3: Deleting consensus metadata
I20240502 01:53:52.457468 21300 tablet_bootstrap.cc:492] T 8ddbb3dba5c04928953f2b41e87d0eed P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:53:52.458258 17322 catalog_manager.cc:4830] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): tablet 9fe583045886494ca8cf656ad2f83189 (table rebalance_test_table_18 [id=fa75b6d8fec44d1998a1c84ded9c9998]) successfully deleted
I20240502 01:53:52.458922 20996 tablet_replica.cc:330] T 1d4efb3b1e3e4fa3bde6305ac993596e P bb31bceba76d466db24b9341c04900c3: stopping tablet replica
I20240502 01:53:52.457548 20232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31a4ca0cc44e492fbd13a70ce38add35" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:53:52.459124 20996 raft_consensus.cc:2229] T 1d4efb3b1e3e4fa3bde6305ac993596e P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:52.459247 20232 raft_consensus.cc:2379] T 31a4ca0cc44e492fbd13a70ce38add35 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bb31bceba76d466db24b9341c04900c3 in current term 1: Already voted for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in this term.
I20240502 01:53:52.459448 20996 raft_consensus.cc:2258] T 1d4efb3b1e3e4fa3bde6305ac993596e P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:52.459661 17803 leader_election.cc:302] T 31a4ca0cc44e492fbd13a70ce38add35 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb31bceba76d466db24b9341c04900c3; no voters: 6a36c952c51542c0bf7ea3ec2857a9f9, fdc2ac372492454c85387589a91f63af
I20240502 01:53:52.460847 21259 raft_consensus.cc:3037] T 31a4ca0cc44e492fbd13a70ce38add35 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:52.462216 21300 tablet_bootstrap.cc:654] T 8ddbb3dba5c04928953f2b41e87d0eed P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:52.462288 20741 ts_tablet_manager.cc:1888] T 9fe583045886494ca8cf656ad2f83189 P 4da690c0fa13407a85c0689ce8de9ade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:52.462442 20741 log.cc:1196] T 9fe583045886494ca8cf656ad2f83189 P 4da690c0fa13407a85c0689ce8de9ade: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/9fe583045886494ca8cf656ad2f83189
I20240502 01:53:52.462925 20741 ts_tablet_manager.cc:1909] T 9fe583045886494ca8cf656ad2f83189 P 4da690c0fa13407a85c0689ce8de9ade: Deleting consensus metadata
I20240502 01:53:52.463459 21259 raft_consensus.cc:2726] T 31a4ca0cc44e492fbd13a70ce38add35 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:53:52.463984 17321 catalog_manager.cc:4830] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): tablet 9fe583045886494ca8cf656ad2f83189 (table rebalance_test_table_18 [id=fa75b6d8fec44d1998a1c84ded9c9998]) successfully deleted
I20240502 01:53:52.465586 20996 ts_tablet_manager.cc:1875] T 1d4efb3b1e3e4fa3bde6305ac993596e P bb31bceba76d466db24b9341c04900c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:52.471010 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ddbb3dba5c04928953f2b41e87d0eed" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:53:52.471238 17736 raft_consensus.cc:2454] T 8ddbb3dba5c04928953f2b41e87d0eed P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 0.
I20240502 01:53:52.473918 20564 leader_election.cc:302] T 8ddbb3dba5c04928953f2b41e87d0eed P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dfa1152987644bd781b46c7e42a7a685, f19cc6cb1e76460aa6711e49a5775091; no voters: fdc2ac372492454c85387589a91f63af
I20240502 01:53:52.474774 21125 raft_consensus.cc:2781] T 8ddbb3dba5c04928953f2b41e87d0eed P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:52.474900 21125 raft_consensus.cc:482] T 8ddbb3dba5c04928953f2b41e87d0eed P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:52.474989 21125 raft_consensus.cc:3037] T 8ddbb3dba5c04928953f2b41e87d0eed P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:52.475071 20996 ts_tablet_manager.cc:1888] T 1d4efb3b1e3e4fa3bde6305ac993596e P bb31bceba76d466db24b9341c04900c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:52.475183 20996 log.cc:1196] T 1d4efb3b1e3e4fa3bde6305ac993596e P bb31bceba76d466db24b9341c04900c3: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/wals/1d4efb3b1e3e4fa3bde6305ac993596e
I20240502 01:53:52.475515 20996 ts_tablet_manager.cc:1909] T 1d4efb3b1e3e4fa3bde6305ac993596e P bb31bceba76d466db24b9341c04900c3: Deleting consensus metadata
I20240502 01:53:52.476297 17322 catalog_manager.cc:4830] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): tablet 1d4efb3b1e3e4fa3bde6305ac993596e (table rebalance_test_table_18 [id=fa75b6d8fec44d1998a1c84ded9c9998]) successfully deleted
I20240502 01:53:52.477109 21125 raft_consensus.cc:504] T 8ddbb3dba5c04928953f2b41e87d0eed P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:52.478767 20232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ddbb3dba5c04928953f2b41e87d0eed" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:53:52.479102 20737 ts_tablet_manager.cc:1888] T 1d4efb3b1e3e4fa3bde6305ac993596e P 3fceccefba274b1dbbd8216af8b0bc29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 01:53:52.479275 20737 log.cc:1196] T 1d4efb3b1e3e4fa3bde6305ac993596e P 3fceccefba274b1dbbd8216af8b0bc29: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/wals/1d4efb3b1e3e4fa3bde6305ac993596e
I20240502 01:53:52.479602 20737 ts_tablet_manager.cc:1909] T 1d4efb3b1e3e4fa3bde6305ac993596e P 3fceccefba274b1dbbd8216af8b0bc29: Deleting consensus metadata
I20240502 01:53:52.478432 21125 leader_election.cc:288] T 8ddbb3dba5c04928953f2b41e87d0eed P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 election: Requested vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
W20240502 01:53:52.479447 20565 leader_election.cc:341] T 8ddbb3dba5c04928953f2b41e87d0eed P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:53:52.479712 21260 consensus_queue.cc:1035] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:52.480654 17319 catalog_manager.cc:4830] TS 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): tablet 1d4efb3b1e3e4fa3bde6305ac993596e (table rebalance_test_table_18 [id=fa75b6d8fec44d1998a1c84ded9c9998]) successfully deleted
I20240502 01:53:52.483729 21300 tablet_bootstrap.cc:492] T 8ddbb3dba5c04928953f2b41e87d0eed P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:53:52.483915 21300 ts_tablet_manager.cc:1367] T 8ddbb3dba5c04928953f2b41e87d0eed P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.002s
I20240502 01:53:52.484902 21300 raft_consensus.cc:348] T 8ddbb3dba5c04928953f2b41e87d0eed P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:52.485158 21300 raft_consensus.cc:374] T 8ddbb3dba5c04928953f2b41e87d0eed P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:52.485282 21300 raft_consensus.cc:729] T 8ddbb3dba5c04928953f2b41e87d0eed P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:53:52.485492 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ddbb3dba5c04928953f2b41e87d0eed" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:53:52.485553 21300 consensus_queue.cc:260] T 8ddbb3dba5c04928953f2b41e87d0eed P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:52.485699 17736 raft_consensus.cc:3037] T 8ddbb3dba5c04928953f2b41e87d0eed P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:52.486819 21300 ts_tablet_manager.cc:1398] T 8ddbb3dba5c04928953f2b41e87d0eed P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:52.487161 17865 consensus_queue.cc:237] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:53:52.490331 17735 raft_consensus.cc:1261] T 4bb42883b1a0475b863d008bd01fa70a P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:52.490720 17736 raft_consensus.cc:2454] T 8ddbb3dba5c04928953f2b41e87d0eed P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 1.
I20240502 01:53:52.490759 17605 raft_consensus.cc:1261] T 4bb42883b1a0475b863d008bd01fa70a P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:52.491253 20564 leader_election.cc:302] T 8ddbb3dba5c04928953f2b41e87d0eed P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dfa1152987644bd781b46c7e42a7a685, f19cc6cb1e76460aa6711e49a5775091; no voters: fdc2ac372492454c85387589a91f63af
I20240502 01:53:52.491426 21263 consensus_queue.cc:1035] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:52.491577 21125 raft_consensus.cc:2781] T 8ddbb3dba5c04928953f2b41e87d0eed P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:52.492110 21259 consensus_queue.cc:1035] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:52.492838 21125 raft_consensus.cc:686] T 8ddbb3dba5c04928953f2b41e87d0eed P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Becoming Leader. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Running, Role: LEADER
I20240502 01:53:52.493232 21125 consensus_queue.cc:237] T 8ddbb3dba5c04928953f2b41e87d0eed P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:52.499337 21259 raft_consensus.cc:2932] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:53:52.506191 17335 catalog_manager.cc:5484] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 reported cstate change: config changed from index -1 to 4, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:52.507136 17327 catalog_manager.cc:5484] T 8ddbb3dba5c04928953f2b41e87d0eed P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: term changed from 0 to 1, leader changed from <none> to f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72). New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:52.508862 17604 raft_consensus.cc:2932] T 4bb42883b1a0475b863d008bd01fa70a P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:53:52.509619 20813 consensus_queue.cc:1035] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:52.514593 17865 raft_consensus.cc:1203] T 31aa6a26f64f49368ad43cc97c66f2f1 P bb31bceba76d466db24b9341c04900c3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:53:52.518800 17736 raft_consensus.cc:2932] T 4bb42883b1a0475b863d008bd01fa70a P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
W20240502 01:53:52.520951 17802 consensus_peers.cc:485] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4bb42883b1a0475b863d008bd01fa70a. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:52.524015 20877 consensus_queue.cc:1035] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:52.547279 21296 raft_consensus.cc:482] T 8ddbb3dba5c04928953f2b41e87d0eed P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:52.547434 21296 raft_consensus.cc:504] T 8ddbb3dba5c04928953f2b41e87d0eed P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:52.548345 21296 leader_election.cc:288] T 8ddbb3dba5c04928953f2b41e87d0eed P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:53:52.549005 20630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ddbb3dba5c04928953f2b41e87d0eed" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091" is_pre_election: true
I20240502 01:53:52.549211 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ddbb3dba5c04928953f2b41e87d0eed" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:53:52.549477 17736 raft_consensus.cc:2379] T 8ddbb3dba5c04928953f2b41e87d0eed P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdc2ac372492454c85387589a91f63af in current term 1: Already voted for candidate f19cc6cb1e76460aa6711e49a5775091 in this term.
I20240502 01:53:52.549713 17604 consensus_queue.cc:237] T 52ac2775e02844d195677035796c3847 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } }
I20240502 01:53:52.549861 20167 leader_election.cc:302] T 8ddbb3dba5c04928953f2b41e87d0eed P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdc2ac372492454c85387589a91f63af; no voters: dfa1152987644bd781b46c7e42a7a685, f19cc6cb1e76460aa6711e49a5775091
I20240502 01:53:52.550230 21296 raft_consensus.cc:3037] T 8ddbb3dba5c04928953f2b41e87d0eed P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:52.552598 17865 raft_consensus.cc:1261] T 52ac2775e02844d195677035796c3847 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:52.553426 17475 raft_consensus.cc:1261] T 52ac2775e02844d195677035796c3847 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:53:52.553905 17542 consensus_peers.cc:485] T 52ac2775e02844d195677035796c3847 P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): Couldn't send request to peer 6a36c952c51542c0bf7ea3ec2857a9f9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 52ac2775e02844d195677035796c3847. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:52.554457 21296 raft_consensus.cc:2726] T 8ddbb3dba5c04928953f2b41e87d0eed P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:53:52.555720 20868 consensus_queue.cc:1035] T 52ac2775e02844d195677035796c3847 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:52.556221 20868 consensus_queue.cc:1035] T 52ac2775e02844d195677035796c3847 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:52.558622 20868 raft_consensus.cc:2932] T 52ac2775e02844d195677035796c3847 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } } }
I20240502 01:53:52.559247 17475 raft_consensus.cc:2932] T 52ac2775e02844d195677035796c3847 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } } }
I20240502 01:53:52.565415 17333 catalog_manager.cc:5484] T 52ac2775e02844d195677035796c3847 P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: config changed from index -1 to 4, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } } }
I20240502 01:53:52.572400 17865 raft_consensus.cc:2932] T 52ac2775e02844d195677035796c3847 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } } }
I20240502 01:53:52.586797 20630 consensus_queue.cc:237] T 975f253ff03447ff96fe84169c05097e P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } }
I20240502 01:53:52.589598 20232 raft_consensus.cc:1261] T 975f253ff03447ff96fe84169c05097e P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:52.590554 17475 raft_consensus.cc:1261] T 975f253ff03447ff96fe84169c05097e P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:52.591387 21314 consensus_queue.cc:1035] T 975f253ff03447ff96fe84169c05097e P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:53:52.592103 21314 consensus_queue.cc:1035] T 975f253ff03447ff96fe84169c05097e P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:53:52.597981 20562 consensus_peers.cc:485] T 975f253ff03447ff96fe84169c05097e P f19cc6cb1e76460aa6711e49a5775091 -> Peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): Couldn't send request to peer 3fceccefba274b1dbbd8216af8b0bc29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 975f253ff03447ff96fe84169c05097e. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:52.599606 21314 raft_consensus.cc:2932] T 975f253ff03447ff96fe84169c05097e P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } } }
I20240502 01:53:52.600769 17475 raft_consensus.cc:2932] T 975f253ff03447ff96fe84169c05097e P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } } }
I20240502 01:53:52.603436 20232 raft_consensus.cc:2932] T 975f253ff03447ff96fe84169c05097e P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } } }
I20240502 01:53:52.605873 17335 catalog_manager.cc:5484] T 975f253ff03447ff96fe84169c05097e P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: config changed from index -1 to 2, NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) added. New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:52.622114 17604 consensus_queue.cc:237] T 6aacf7c4f19546f581c9a54ad171916b P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
I20240502 01:53:52.624424 17865 raft_consensus.cc:1261] T 6aacf7c4f19546f581c9a54ad171916b P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:53:52.624606 17543 consensus_peers.cc:485] T 6aacf7c4f19546f581c9a54ad171916b P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): Couldn't send request to peer 2f306549407b4bb281ccf2994035e4f1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6aacf7c4f19546f581c9a54ad171916b. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:52.624650 17736 raft_consensus.cc:1261] T 6aacf7c4f19546f581c9a54ad171916b P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:52.624959 20868 consensus_queue.cc:1035] T 6aacf7c4f19546f581c9a54ad171916b P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:52.625290 21329 consensus_queue.cc:1035] T 6aacf7c4f19546f581c9a54ad171916b P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:52.627987 20868 raft_consensus.cc:2932] T 6aacf7c4f19546f581c9a54ad171916b P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:53:52.628736 17736 raft_consensus.cc:2932] T 6aacf7c4f19546f581c9a54ad171916b P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:53:52.629472 17865 raft_consensus.cc:2932] T 6aacf7c4f19546f581c9a54ad171916b P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:53:52.633759 17327 catalog_manager.cc:5484] T 6aacf7c4f19546f581c9a54ad171916b P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: config changed from index -1 to 4, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:52.646129 17335 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:53:52.649235 17475 consensus_queue.cc:237] T 9642b9b37a784cf881bfc4971d8c9cb6 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:53:52.652508 20232 raft_consensus.cc:1261] T 8ddbb3dba5c04928953f2b41e87d0eed P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:52.652909 17736 raft_consensus.cc:1261] T 8ddbb3dba5c04928953f2b41e87d0eed P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:52.653050 17735 raft_consensus.cc:1261] T 9642b9b37a784cf881bfc4971d8c9cb6 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:52.653669 21314 consensus_queue.cc:1035] T 8ddbb3dba5c04928953f2b41e87d0eed P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:52.654603 21125 consensus_queue.cc:1035] T 8ddbb3dba5c04928953f2b41e87d0eed P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:52.654820 17604 raft_consensus.cc:1261] T 9642b9b37a784cf881bfc4971d8c9cb6 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:52.655273 17335 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_extra_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:52.655401 21264 consensus_queue.cc:1035] T 9642b9b37a784cf881bfc4971d8c9cb6 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:52.657534 21264 consensus_queue.cc:1035] T 9642b9b37a784cf881bfc4971d8c9cb6 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:53:52.662900 17411 consensus_peers.cc:485] T 9642b9b37a784cf881bfc4971d8c9cb6 P 4da690c0fa13407a85c0689ce8de9ade -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9642b9b37a784cf881bfc4971d8c9cb6. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:52.672127 20211 tablet_service.cc:1467] Processing CreateTablet for tablet 1b14ac3413004127b72904b631ec1b94 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=96aabcc026484fffa9ea0c0dc8d37340]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:52.672581 20211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b14ac3413004127b72904b631ec1b94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:52.673604 17587 tablet_service.cc:1467] Processing CreateTablet for tablet 1b14ac3413004127b72904b631ec1b94 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=96aabcc026484fffa9ea0c0dc8d37340]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:52.674077 17587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b14ac3413004127b72904b631ec1b94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:52.674065 20347 tablet_service.cc:1467] Processing CreateTablet for tablet 1d8e2e0f6ed34e4cb004e1c5309c0f1f (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=96aabcc026484fffa9ea0c0dc8d37340]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:52.674469 20347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d8e2e0f6ed34e4cb004e1c5309c0f1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:52.677882 20609 tablet_service.cc:1467] Processing CreateTablet for tablet 1b14ac3413004127b72904b631ec1b94 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=96aabcc026484fffa9ea0c0dc8d37340]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:52.678103 20608 tablet_service.cc:1467] Processing CreateTablet for tablet 1d8e2e0f6ed34e4cb004e1c5309c0f1f (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=96aabcc026484fffa9ea0c0dc8d37340]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:52.678335 20609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b14ac3413004127b72904b631ec1b94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:52.679559 17846 tablet_service.cc:1467] Processing CreateTablet for tablet 1a837e7dce89462596b2b6049069ec8e (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=96aabcc026484fffa9ea0c0dc8d37340]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:52.679960 17846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a837e7dce89462596b2b6049069ec8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:52.681257 20610 tablet_service.cc:1467] Processing CreateTablet for tablet 1a837e7dce89462596b2b6049069ec8e (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=96aabcc026484fffa9ea0c0dc8d37340]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:52.681663 20610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a837e7dce89462596b2b6049069ec8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:52.683771 21264 raft_consensus.cc:2932] T 9642b9b37a784cf881bfc4971d8c9cb6 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:53:52.684351 20608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d8e2e0f6ed34e4cb004e1c5309c0f1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:52.685416 20478 tablet_service.cc:1467] Processing CreateTablet for tablet 1d8e2e0f6ed34e4cb004e1c5309c0f1f (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=96aabcc026484fffa9ea0c0dc8d37340]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:52.685849 20478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d8e2e0f6ed34e4cb004e1c5309c0f1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:52.688226 20212 tablet_service.cc:1467] Processing CreateTablet for tablet 1a837e7dce89462596b2b6049069ec8e (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=96aabcc026484fffa9ea0c0dc8d37340]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:52.688652 20212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a837e7dce89462596b2b6049069ec8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:52.696537 21300 tablet_bootstrap.cc:492] T 1a837e7dce89462596b2b6049069ec8e P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:53:52.697839 21253 tablet_bootstrap.cc:492] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:53:52.698359 21300 tablet_bootstrap.cc:654] T 1a837e7dce89462596b2b6049069ec8e P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:52.699465 21253 tablet_bootstrap.cc:654] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:52.700194 17604 raft_consensus.cc:2932] T 9642b9b37a784cf881bfc4971d8c9cb6 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:53:52.712630 17333 catalog_manager.cc:5576] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339) does not have the latest schema for tablet 8ddbb3dba5c04928953f2b41e87d0eed (table rebalancer_dynamic_table [id=1cfb3501e7c3445cacb6edd5bd5f14af]). Expected version 1 got 0
I20240502 01:53:52.714118 21370 tablet.cc:1722] T 8ddbb3dba5c04928953f2b41e87d0eed P f19cc6cb1e76460aa6711e49a5775091: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:53:52.714807 21370 tablet.cc:1620] T 8ddbb3dba5c04928953f2b41e87d0eed P f19cc6cb1e76460aa6711e49a5775091: MemRowSet was empty: no flush needed.
I20240502 01:53:52.715628 17333 catalog_manager.cc:5576] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411) does not have the latest schema for tablet 8ddbb3dba5c04928953f2b41e87d0eed (table rebalancer_dynamic_table [id=1cfb3501e7c3445cacb6edd5bd5f14af]). Expected version 1 got 0
I20240502 01:53:52.717777 21362 tablet_bootstrap.cc:492] T 1b14ac3413004127b72904b631ec1b94 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:52.718043 17333 catalog_manager.cc:5576] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115) does not have the latest schema for tablet 8ddbb3dba5c04928953f2b41e87d0eed (table rebalancer_dynamic_table [id=1cfb3501e7c3445cacb6edd5bd5f14af]). Expected version 1 got 0
I20240502 01:53:52.719834 21300 tablet_bootstrap.cc:492] T 1a837e7dce89462596b2b6049069ec8e P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:53:52.720386 21300 ts_tablet_manager.cc:1367] T 1a837e7dce89462596b2b6049069ec8e P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20240502 01:53:52.721801 17327 catalog_manager.cc:5484] T 9642b9b37a784cf881bfc4971d8c9cb6 P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: config changed from index -1 to 4, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New cstate: current_term: 1 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:53:52.722572 21362 tablet_bootstrap.cc:654] T 1b14ac3413004127b72904b631ec1b94 P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:52.722951 17333 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=1cfb3501e7c3445cacb6edd5bd5f14af] alter complete (version 1)
I20240502 01:53:52.721558 21371 tablet.cc:1722] T 8ddbb3dba5c04928953f2b41e87d0eed P fdc2ac372492454c85387589a91f63af: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:53:52.728847 21300 raft_consensus.cc:348] T 1a837e7dce89462596b2b6049069ec8e P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:52.729157 17333 catalog_manager.cc:5576] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339) does not have the latest schema for tablet 8ddbb3dba5c04928953f2b41e87d0eed (table rebalancer_dynamic_table [id=1cfb3501e7c3445cacb6edd5bd5f14af]). Expected version 1 got 0
I20240502 01:53:52.729198 21300 raft_consensus.cc:374] T 1a837e7dce89462596b2b6049069ec8e P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:52.729351 21300 raft_consensus.cc:729] T 1a837e7dce89462596b2b6049069ec8e P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:53:52.730197 21300 consensus_queue.cc:260] T 1a837e7dce89462596b2b6049069ec8e P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:52.731812 21300 ts_tablet_manager.cc:1398] T 1a837e7dce89462596b2b6049069ec8e P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:53:52.732411 21371 tablet.cc:1620] T 8ddbb3dba5c04928953f2b41e87d0eed P fdc2ac372492454c85387589a91f63af: MemRowSet was empty: no flush needed.
W20240502 01:53:52.735694 17411 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.42.70:39155, user_credentials={real_user=slave}} blocked reactor thread for 47648.8us
I20240502 01:53:52.735883 21368 tablet_bootstrap.cc:492] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:53:52.737614 21253 tablet_bootstrap.cc:492] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:53:52.739320 21362 tablet_bootstrap.cc:492] T 1b14ac3413004127b72904b631ec1b94 P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:52.739408 21368 tablet_bootstrap.cc:654] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:52.739518 21362 ts_tablet_manager.cc:1367] T 1b14ac3413004127b72904b631ec1b94 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.005s
I20240502 01:53:52.739647 21253 ts_tablet_manager.cc:1367] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.042s user 0.007s sys 0.005s
I20240502 01:53:52.740329 21362 raft_consensus.cc:348] T 1b14ac3413004127b72904b631ec1b94 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:52.740593 21362 raft_consensus.cc:374] T 1b14ac3413004127b72904b631ec1b94 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:52.740485 21253 raft_consensus.cc:348] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:52.740685 21253 raft_consensus.cc:374] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:52.740756 21362 raft_consensus.cc:729] T 1b14ac3413004127b72904b631ec1b94 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:52.740868 21253 raft_consensus.cc:729] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:53:52.741089 21362 consensus_queue.cc:260] T 1b14ac3413004127b72904b631ec1b94 P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:52.741933 21362 ts_tablet_manager.cc:1398] T 1b14ac3413004127b72904b631ec1b94 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:53:52.743316 21375 tablet_bootstrap.cc:492] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:53:52.743425 21253 consensus_queue.cc:260] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:52.745661 21253 ts_tablet_manager.cc:1398] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20240502 01:53:52.745966 21253 tablet_bootstrap.cc:492] T 1a837e7dce89462596b2b6049069ec8e P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:53:52.745841 17736 raft_consensus.cc:2932] T 9642b9b37a784cf881bfc4971d8c9cb6 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:53:52.749140 21375 tablet_bootstrap.cc:654] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:52.751062 21368 tablet_bootstrap.cc:492] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:53:52.752135 21368 ts_tablet_manager.cc:1367] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20240502 01:53:52.753167 21368 raft_consensus.cc:348] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:52.754221 21296 raft_consensus.cc:482] T 1a837e7dce89462596b2b6049069ec8e P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:52.754416 21296 raft_consensus.cc:504] T 1a837e7dce89462596b2b6049069ec8e P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:52.754981 21253 tablet_bootstrap.cc:654] T 1a837e7dce89462596b2b6049069ec8e P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:52.755052 21296 leader_election.cc:288] T 1a837e7dce89462596b2b6049069ec8e P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:53:52.756203 21368 raft_consensus.cc:374] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:52.755645 20630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a837e7dce89462596b2b6049069ec8e" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091" is_pre_election: true
I20240502 01:53:52.756482 17865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a837e7dce89462596b2b6049069ec8e" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
W20240502 01:53:52.756908 20167 leader_election.cc:341] T 1a837e7dce89462596b2b6049069ec8e P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Illegal state: must be running to vote when last-logged opid is not known
W20240502 01:53:52.757362 20168 leader_election.cc:341] T 1a837e7dce89462596b2b6049069ec8e P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:53:52.757488 20168 leader_election.cc:302] T 1a837e7dce89462596b2b6049069ec8e P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdc2ac372492454c85387589a91f63af; no voters: bb31bceba76d466db24b9341c04900c3, f19cc6cb1e76460aa6711e49a5775091
I20240502 01:53:52.757923 21368 raft_consensus.cc:729] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:53:52.757973 21296 raft_consensus.cc:2726] T 1a837e7dce89462596b2b6049069ec8e P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:53:52.760638 21375 tablet_bootstrap.cc:492] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:53:52.758554 21368 consensus_queue.cc:260] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:52.762578 21375 ts_tablet_manager.cc:1367] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20240502 01:53:52.763103 21374 tablet_bootstrap.cc:492] T 1a837e7dce89462596b2b6049069ec8e P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:52.763592 21253 tablet_bootstrap.cc:492] T 1a837e7dce89462596b2b6049069ec8e P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:53:52.761246 21368 ts_tablet_manager.cc:1398] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 01:53:52.764681 21374 tablet_bootstrap.cc:654] T 1a837e7dce89462596b2b6049069ec8e P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:52.765609 21253 ts_tablet_manager.cc:1367] T 1a837e7dce89462596b2b6049069ec8e P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20240502 01:53:52.765617 21372 tablet.cc:1722] T 8ddbb3dba5c04928953f2b41e87d0eed P dfa1152987644bd781b46c7e42a7a685: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:53:52.766752 21372 tablet.cc:1620] T 8ddbb3dba5c04928953f2b41e87d0eed P dfa1152987644bd781b46c7e42a7a685: MemRowSet was empty: no flush needed.
I20240502 01:53:52.764644 21375 raft_consensus.cc:348] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:52.767053 21375 raft_consensus.cc:374] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:52.767258 21375 raft_consensus.cc:729] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:53:52.767374 21253 raft_consensus.cc:348] T 1a837e7dce89462596b2b6049069ec8e P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:52.767736 21253 raft_consensus.cc:374] T 1a837e7dce89462596b2b6049069ec8e P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:52.770268 21253 raft_consensus.cc:729] T 1a837e7dce89462596b2b6049069ec8e P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:53:52.770579 21253 consensus_queue.cc:260] T 1a837e7dce89462596b2b6049069ec8e P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:52.768117 21375 consensus_queue.cc:260] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:52.770936 20813 raft_consensus.cc:482] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:52.771121 20813 raft_consensus.cc:504] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:52.771459 21300 tablet_bootstrap.cc:492] T 1b14ac3413004127b72904b631ec1b94 P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:53:52.771994 20813 leader_election.cc:288] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:53:52.772940 21300 tablet_bootstrap.cc:654] T 1b14ac3413004127b72904b631ec1b94 P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:52.773810 20498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d8e2e0f6ed34e4cb004e1c5309c0f1f" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
I20240502 01:53:52.774021 20498 raft_consensus.cc:2454] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 0.
I20240502 01:53:52.774586 21375 ts_tablet_manager.cc:1398] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240502 01:53:52.772732 20630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d8e2e0f6ed34e4cb004e1c5309c0f1f" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091" is_pre_election: true
I20240502 01:53:52.774741 20303 leader_election.cc:302] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 6a36c952c51542c0bf7ea3ec2857a9f9; no voters:
I20240502 01:53:52.775082 20813 raft_consensus.cc:2781] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:52.775185 20813 raft_consensus.cc:482] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:52.775297 20813 raft_consensus.cc:3037] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:52.776170 21253 ts_tablet_manager.cc:1398] T 1a837e7dce89462596b2b6049069ec8e P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240502 01:53:52.776441 21253 tablet_bootstrap.cc:492] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
W20240502 01:53:52.775161 20302 leader_election.cc:341] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:53:52.779218 21253 tablet_bootstrap.cc:654] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:52.780201 17335 catalog_manager.cc:5576] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115) does not have the latest schema for tablet 8ddbb3dba5c04928953f2b41e87d0eed (table rebalancer_dynamic_table [id=1cfb3501e7c3445cacb6edd5bd5f14af]). Expected version 1 got 0
I20240502 01:53:52.778295 20813 raft_consensus.cc:504] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:52.781373 20498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d8e2e0f6ed34e4cb004e1c5309c0f1f" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
I20240502 01:53:52.781559 20498 raft_consensus.cc:3037] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:52.783699 20498 raft_consensus.cc:2454] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 1.
I20240502 01:53:52.784971 20303 leader_election.cc:302] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 6a36c952c51542c0bf7ea3ec2857a9f9; no voters:
I20240502 01:53:52.785356 20877 raft_consensus.cc:2781] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:52.785511 20877 raft_consensus.cc:686] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Becoming Leader. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Running, Role: LEADER
I20240502 01:53:52.785740 20877 consensus_queue.cc:237] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:52.787492 21374 tablet_bootstrap.cc:492] T 1a837e7dce89462596b2b6049069ec8e P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:52.789590 21374 ts_tablet_manager.cc:1367] T 1a837e7dce89462596b2b6049069ec8e P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.002s
I20240502 01:53:52.790231 20813 leader_election.cc:288] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Requested vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:53:52.790330 20630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d8e2e0f6ed34e4cb004e1c5309c0f1f" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091"
W20240502 01:53:52.791971 20302 leader_election.cc:341] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:53:52.801514 21253 tablet_bootstrap.cc:492] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:53:52.801676 21253 ts_tablet_manager.cc:1367] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20240502 01:53:52.802512 21253 raft_consensus.cc:348] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:52.802731 21253 raft_consensus.cc:374] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:52.802824 21253 raft_consensus.cc:729] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:53:52.803045 21253 consensus_queue.cc:260] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:52.803642 21253 ts_tablet_manager.cc:1398] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:53:52.810150 21374 raft_consensus.cc:348] T 1a837e7dce89462596b2b6049069ec8e P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:52.811338 21374 raft_consensus.cc:374] T 1a837e7dce89462596b2b6049069ec8e P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:52.811610 21374 raft_consensus.cc:729] T 1a837e7dce89462596b2b6049069ec8e P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:52.812036 21374 consensus_queue.cc:260] T 1a837e7dce89462596b2b6049069ec8e P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:52.812271 21300 tablet_bootstrap.cc:492] T 1b14ac3413004127b72904b631ec1b94 P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:53:52.812414 21300 ts_tablet_manager.cc:1367] T 1b14ac3413004127b72904b631ec1b94 P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.041s user 0.015s sys 0.012s
I20240502 01:53:52.813124 21300 raft_consensus.cc:348] T 1b14ac3413004127b72904b631ec1b94 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:52.813326 21300 raft_consensus.cc:374] T 1b14ac3413004127b72904b631ec1b94 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:52.813405 21300 raft_consensus.cc:729] T 1b14ac3413004127b72904b631ec1b94 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:53:52.814119 21374 ts_tablet_manager.cc:1398] T 1a837e7dce89462596b2b6049069ec8e P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.024s user 0.001s sys 0.001s
I20240502 01:53:52.813623 21300 consensus_queue.cc:260] T 1b14ac3413004127b72904b631ec1b94 P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:52.817723 21125 raft_consensus.cc:482] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:52.817878 21125 raft_consensus.cc:504] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:52.818431 21125 leader_election.cc:288] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:53:52.819202 20498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d8e2e0f6ed34e4cb004e1c5309c0f1f" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
I20240502 01:53:52.819346 20367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d8e2e0f6ed34e4cb004e1c5309c0f1f" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:53:52.819469 20498 raft_consensus.cc:2379] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f19cc6cb1e76460aa6711e49a5775091 in current term 1: Already voted for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in this term.
I20240502 01:53:52.820101 20565 leader_election.cc:302] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f19cc6cb1e76460aa6711e49a5775091; no voters: 2f306549407b4bb281ccf2994035e4f1, 6a36c952c51542c0bf7ea3ec2857a9f9
I20240502 01:53:52.820520 21125 raft_consensus.cc:3037] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:52.820269 17335 catalog_manager.cc:5484] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70). New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:52.823673 21125 raft_consensus.cc:2726] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:53:52.824244 21384 ts_tablet_manager.cc:902] T 62887e332443457f8a5b41a982354dc4 P bb31bceba76d466db24b9341c04900c3: Initiating tablet copy from peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:53:52.824618 21300 ts_tablet_manager.cc:1398] T 1b14ac3413004127b72904b631ec1b94 P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240502 01:53:52.841603 20367 consensus_queue.cc:237] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } }
I20240502 01:53:52.845916 21384 tablet_copy_client.cc:308] T 62887e332443457f8a5b41a982354dc4 P bb31bceba76d466db24b9341c04900c3: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.69:34339
I20240502 01:53:52.846738 20253 tablet_copy_service.cc:140] P fdc2ac372492454c85387589a91f63af: Received BeginTabletCopySession request for tablet 62887e332443457f8a5b41a982354dc4 from peer bb31bceba76d466db24b9341c04900c3 ({username='slave'} at 127.11.42.68:59527)
I20240502 01:53:52.846916 20253 tablet_copy_service.cc:161] P fdc2ac372492454c85387589a91f63af: Beginning new tablet copy session on tablet 62887e332443457f8a5b41a982354dc4 from peer bb31bceba76d466db24b9341c04900c3 at {username='slave'} at 127.11.42.68:59527: session id = bb31bceba76d466db24b9341c04900c3-62887e332443457f8a5b41a982354dc4
I20240502 01:53:52.850701 20253 tablet_copy_source_session.cc:205] T 62887e332443457f8a5b41a982354dc4 P fdc2ac372492454c85387589a91f63af: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:52.854612 21384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62887e332443457f8a5b41a982354dc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:52.854574 17335 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:53:52.860100 20232 raft_consensus.cc:1261] T e42126e16cda4d908b00e2e4a6807d6b P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:53:52.861810 20303 consensus_peers.cc:485] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): Couldn't send request to peer bb31bceba76d466db24b9341c04900c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e42126e16cda4d908b00e2e4a6807d6b. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:52.862828 20813 consensus_queue.cc:1035] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:53:52.863673 21370 tablet.cc:1722] T 8ddbb3dba5c04928953f2b41e87d0eed P f19cc6cb1e76460aa6711e49a5775091: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:53:52.864202 21370 tablet.cc:1620] T 8ddbb3dba5c04928953f2b41e87d0eed P f19cc6cb1e76460aa6711e49a5775091: MemRowSet was empty: no flush needed.
I20240502 01:53:52.865046 17475 raft_consensus.cc:1261] T e42126e16cda4d908b00e2e4a6807d6b P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:52.865149 21372 tablet.cc:1722] T 8ddbb3dba5c04928953f2b41e87d0eed P dfa1152987644bd781b46c7e42a7a685: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:53:52.865593 21372 tablet.cc:1620] T 8ddbb3dba5c04928953f2b41e87d0eed P dfa1152987644bd781b46c7e42a7a685: MemRowSet was empty: no flush needed.
I20240502 01:53:52.865630 20813 consensus_queue.cc:1035] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:53:52.867619 21384 tablet_copy_client.cc:783] T 62887e332443457f8a5b41a982354dc4 P bb31bceba76d466db24b9341c04900c3: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:52.868317 21371 tablet.cc:1722] T 8ddbb3dba5c04928953f2b41e87d0eed P fdc2ac372492454c85387589a91f63af: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:53:52.868714 21371 tablet.cc:1620] T 8ddbb3dba5c04928953f2b41e87d0eed P fdc2ac372492454c85387589a91f63af: MemRowSet was empty: no flush needed.
I20240502 01:53:52.868878 21384 tablet_copy_client.cc:647] T 62887e332443457f8a5b41a982354dc4 P bb31bceba76d466db24b9341c04900c3: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:52.871161 20813 raft_consensus.cc:2932] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } } }
I20240502 01:53:52.873353 21384 tablet_copy_client.cc:523] T 62887e332443457f8a5b41a982354dc4 P bb31bceba76d466db24b9341c04900c3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:52.876856 21384 tablet_bootstrap.cc:492] T 62887e332443457f8a5b41a982354dc4 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:52.882350 17327 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=1cfb3501e7c3445cacb6edd5bd5f14af] alter complete (version 2)
I20240502 01:53:52.883555 21384 tablet_bootstrap.cc:492] T 62887e332443457f8a5b41a982354dc4 P bb31bceba76d466db24b9341c04900c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:52.883890 21384 tablet_bootstrap.cc:492] T 62887e332443457f8a5b41a982354dc4 P bb31bceba76d466db24b9341c04900c3: Bootstrap complete.
I20240502 01:53:52.884059 21384 ts_tablet_manager.cc:1367] T 62887e332443457f8a5b41a982354dc4 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20240502 01:53:52.875929 17475 raft_consensus.cc:2932] T e42126e16cda4d908b00e2e4a6807d6b P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } } }
I20240502 01:53:52.884872 21384 raft_consensus.cc:348] T 62887e332443457f8a5b41a982354dc4 P bb31bceba76d466db24b9341c04900c3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } }
I20240502 01:53:52.885140 21384 raft_consensus.cc:729] T 62887e332443457f8a5b41a982354dc4 P bb31bceba76d466db24b9341c04900c3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: LEARNER
I20240502 01:53:52.885357 21384 consensus_queue.cc:260] T 62887e332443457f8a5b41a982354dc4 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } }
I20240502 01:53:52.886265 21384 ts_tablet_manager.cc:1398] T 62887e332443457f8a5b41a982354dc4 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:52.888087 20232 raft_consensus.cc:2932] T e42126e16cda4d908b00e2e4a6807d6b P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } } }
I20240502 01:53:52.888638 20253 tablet_copy_service.cc:342] P fdc2ac372492454c85387589a91f63af: Request end of tablet copy session bb31bceba76d466db24b9341c04900c3-62887e332443457f8a5b41a982354dc4 received from {username='slave'} at 127.11.42.68:59527
I20240502 01:53:52.888790 20253 tablet_copy_service.cc:433] P fdc2ac372492454c85387589a91f63af: ending tablet copy session bb31bceba76d466db24b9341c04900c3-62887e332443457f8a5b41a982354dc4 on tablet 62887e332443457f8a5b41a982354dc4 with peer bb31bceba76d466db24b9341c04900c3
I20240502 01:53:52.889128 17333 catalog_manager.cc:5484] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: config changed from index -1 to 2, NON_VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) added. New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:52.905424 17865 consensus_queue.cc:237] T 65bdab2c9fba48a6828a864278319cc0 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } }
I20240502 01:53:52.907685 17604 raft_consensus.cc:1261] T 65bdab2c9fba48a6828a864278319cc0 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:52.907721 17735 raft_consensus.cc:1261] T 65bdab2c9fba48a6828a864278319cc0 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:53:52.908262 17802 consensus_peers.cc:485] T 65bdab2c9fba48a6828a864278319cc0 P bb31bceba76d466db24b9341c04900c3 -> Peer 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): Couldn't send request to peer 6a36c952c51542c0bf7ea3ec2857a9f9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 65bdab2c9fba48a6828a864278319cc0. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:52.908370 21263 consensus_queue.cc:1035] T 65bdab2c9fba48a6828a864278319cc0 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:52.908645 21148 consensus_queue.cc:1035] T 65bdab2c9fba48a6828a864278319cc0 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:52.911582 21265 raft_consensus.cc:2932] T 65bdab2c9fba48a6828a864278319cc0 P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } } }
I20240502 01:53:52.912124 17604 raft_consensus.cc:2932] T 65bdab2c9fba48a6828a864278319cc0 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } } }
I20240502 01:53:52.915016 17735 raft_consensus.cc:2932] T 65bdab2c9fba48a6828a864278319cc0 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } } }
I20240502 01:53:52.918644 17333 catalog_manager.cc:5484] T 65bdab2c9fba48a6828a864278319cc0 P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: config changed from index -1 to 4, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } } }
I20240502 01:53:52.933192 20367 consensus_queue.cc:237] T ebb9bca9f1bd41b0857939909b1110e4 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
I20240502 01:53:52.935771 17735 raft_consensus.cc:1261] T ebb9bca9f1bd41b0857939909b1110e4 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:53:52.935806 20303 consensus_peers.cc:485] T ebb9bca9f1bd41b0857939909b1110e4 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): Couldn't send request to peer 2f306549407b4bb281ccf2994035e4f1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ebb9bca9f1bd41b0857939909b1110e4. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:52.935863 20232 raft_consensus.cc:1261] T ebb9bca9f1bd41b0857939909b1110e4 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:52.936388 20813 consensus_queue.cc:1035] T ebb9bca9f1bd41b0857939909b1110e4 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:53:52.936691 20877 consensus_queue.cc:1035] T ebb9bca9f1bd41b0857939909b1110e4 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:53:52.939339 21009 raft_consensus.cc:2932] T ebb9bca9f1bd41b0857939909b1110e4 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:53:52.939821 20232 raft_consensus.cc:2932] T ebb9bca9f1bd41b0857939909b1110e4 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:53:52.940030 17735 raft_consensus.cc:2932] T ebb9bca9f1bd41b0857939909b1110e4 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:53:52.944938 17335 catalog_manager.cc:5484] T ebb9bca9f1bd41b0857939909b1110e4 P dfa1152987644bd781b46c7e42a7a685 reported cstate change: config changed from index -1 to 2, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:53:52.961217 17604 consensus_queue.cc:237] T 1f01949cf70440b38e59a5020fd48c44 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:53:52.963343 17735 raft_consensus.cc:1261] T 1f01949cf70440b38e59a5020fd48c44 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:53:52.963666 17542 consensus_peers.cc:485] T 1f01949cf70440b38e59a5020fd48c44 P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1f01949cf70440b38e59a5020fd48c44. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:52.963706 17865 raft_consensus.cc:1261] T 1f01949cf70440b38e59a5020fd48c44 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:52.963979 21329 consensus_queue.cc:1035] T 1f01949cf70440b38e59a5020fd48c44 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:52.964344 20868 consensus_queue.cc:1035] T 1f01949cf70440b38e59a5020fd48c44 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:52.967734 21329 raft_consensus.cc:2932] T 1f01949cf70440b38e59a5020fd48c44 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:53:52.971405 17865 raft_consensus.cc:2932] T 1f01949cf70440b38e59a5020fd48c44 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:53:52.979820 21421 ts_tablet_manager.cc:902] T 4bb42883b1a0475b863d008bd01fa70a P f19cc6cb1e76460aa6711e49a5775091: Initiating tablet copy from peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:53:52.982252 17333 catalog_manager.cc:5484] T 1f01949cf70440b38e59a5020fd48c44 P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: config changed from index -1 to 4, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:52.983594 21125 raft_consensus.cc:482] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:52.983798 21125 raft_consensus.cc:504] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:52.984519 21125 leader_election.cc:288] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:52.987040 20232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b14ac3413004127b72904b631ec1b94" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:53:52.987342 20232 raft_consensus.cc:2454] T 1b14ac3413004127b72904b631ec1b94 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 0.
I20240502 01:53:52.988618 20565 leader_election.cc:302] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f19cc6cb1e76460aa6711e49a5775091, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:53:52.989498 17736 raft_consensus.cc:2932] T 1f01949cf70440b38e59a5020fd48c44 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:53:52.990722 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b14ac3413004127b72904b631ec1b94" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:53:52.991009 17604 raft_consensus.cc:2454] T 1b14ac3413004127b72904b631ec1b94 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 0.
I20240502 01:53:52.991804 21125 raft_consensus.cc:2781] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:52.991947 21125 raft_consensus.cc:482] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:52.992098 21125 raft_consensus.cc:3037] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:52.993413 17327 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:53:52.995750 21125 raft_consensus.cc:504] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:52.999670 20232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b14ac3413004127b72904b631ec1b94" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:53:52.999898 20232 raft_consensus.cc:3037] T 1b14ac3413004127b72904b631ec1b94 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:53.002575 20232 raft_consensus.cc:2454] T 1b14ac3413004127b72904b631ec1b94 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 1.
I20240502 01:53:53.003890 21370 tablet.cc:1722] T 8ddbb3dba5c04928953f2b41e87d0eed P f19cc6cb1e76460aa6711e49a5775091: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:53:53.004092 20565 leader_election.cc:302] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f19cc6cb1e76460aa6711e49a5775091, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:53:53.005219 21372 tablet.cc:1722] T 8ddbb3dba5c04928953f2b41e87d0eed P dfa1152987644bd781b46c7e42a7a685: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:53:53.006398 21421 tablet_copy_client.cc:308] T 4bb42883b1a0475b863d008bd01fa70a P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.68:44083
I20240502 01:53:53.010277 21429 raft_consensus.cc:2781] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:53.010807 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b14ac3413004127b72904b631ec1b94" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:53:53.011063 17604 raft_consensus.cc:3037] T 1b14ac3413004127b72904b631ec1b94 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:53.014312 17604 raft_consensus.cc:2454] T 1b14ac3413004127b72904b631ec1b94 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 1.
I20240502 01:53:53.025624 17887 tablet_copy_service.cc:140] P bb31bceba76d466db24b9341c04900c3: Received BeginTabletCopySession request for tablet 4bb42883b1a0475b863d008bd01fa70a from peer f19cc6cb1e76460aa6711e49a5775091 ({username='slave'} at 127.11.42.72:53629)
I20240502 01:53:53.025803 17887 tablet_copy_service.cc:161] P bb31bceba76d466db24b9341c04900c3: Beginning new tablet copy session on tablet 4bb42883b1a0475b863d008bd01fa70a from peer f19cc6cb1e76460aa6711e49a5775091 at {username='slave'} at 127.11.42.72:53629: session id = f19cc6cb1e76460aa6711e49a5775091-4bb42883b1a0475b863d008bd01fa70a
I20240502 01:53:53.027623 17335 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table_new_name [id=1cfb3501e7c3445cacb6edd5bd5f14af] alter complete (version 3)
I20240502 01:53:53.028697 17887 tablet_copy_source_session.cc:205] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:53.036186 21421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bb42883b1a0475b863d008bd01fa70a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:53.040747 21125 leader_election.cc:288] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 election: Requested vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:53.043340 21421 tablet_copy_client.cc:783] T 4bb42883b1a0475b863d008bd01fa70a P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:53.043677 21421 tablet_copy_client.cc:647] T 4bb42883b1a0475b863d008bd01fa70a P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:53.046516 21421 tablet_copy_client.cc:523] T 4bb42883b1a0475b863d008bd01fa70a P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:53.050699 21371 tablet.cc:1722] T 8ddbb3dba5c04928953f2b41e87d0eed P fdc2ac372492454c85387589a91f63af: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:53:53.053436 21421 tablet_bootstrap.cc:492] T 4bb42883b1a0475b863d008bd01fa70a P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
W20240502 01:53:53.068593 20564 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.42.67:36115, user_credentials={real_user=slave}} blocked reactor thread for 62540us
I20240502 01:53:53.078378 21434 ts_tablet_manager.cc:902] T 9642b9b37a784cf881bfc4971d8c9cb6 P f19cc6cb1e76460aa6711e49a5775091: Initiating tablet copy from peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:53.080025 21432 ts_tablet_manager.cc:902] T 6aacf7c4f19546f581c9a54ad171916b P 2f306549407b4bb281ccf2994035e4f1: Initiating tablet copy from peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:53.084154 21432 tablet_copy_client.cc:308] T 6aacf7c4f19546f581c9a54ad171916b P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.66:38531
I20240502 01:53:53.085734 17627 tablet_copy_service.cc:140] P 3fceccefba274b1dbbd8216af8b0bc29: Received BeginTabletCopySession request for tablet 6aacf7c4f19546f581c9a54ad171916b from peer 2f306549407b4bb281ccf2994035e4f1 ({username='slave'} at 127.11.42.71:49447)
I20240502 01:53:53.085927 17627 tablet_copy_service.cc:161] P 3fceccefba274b1dbbd8216af8b0bc29: Beginning new tablet copy session on tablet 6aacf7c4f19546f581c9a54ad171916b from peer 2f306549407b4bb281ccf2994035e4f1 at {username='slave'} at 127.11.42.71:49447: session id = 2f306549407b4bb281ccf2994035e4f1-6aacf7c4f19546f581c9a54ad171916b
I20240502 01:53:53.089938 17627 tablet_copy_source_session.cc:205] T 6aacf7c4f19546f581c9a54ad171916b P 3fceccefba274b1dbbd8216af8b0bc29: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:53.093555 21432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6aacf7c4f19546f581c9a54ad171916b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:53.099577 21429 raft_consensus.cc:686] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Becoming Leader. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Running, Role: LEADER
I20240502 01:53:53.100044 21429 consensus_queue.cc:237] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:53:53.106068 21430 raft_consensus.cc:482] T 1a837e7dce89462596b2b6049069ec8e P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:53.106292 21430 raft_consensus.cc:504] T 1a837e7dce89462596b2b6049069ec8e P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:53.107798 20232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a837e7dce89462596b2b6049069ec8e" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:53:53.108057 20232 raft_consensus.cc:2454] T 1a837e7dce89462596b2b6049069ec8e P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 0.
I20240502 01:53:53.108299 17864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a837e7dce89462596b2b6049069ec8e" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:53:53.108539 17864 raft_consensus.cc:2454] T 1a837e7dce89462596b2b6049069ec8e P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 0.
I20240502 01:53:53.108671 20565 leader_election.cc:302] T 1a837e7dce89462596b2b6049069ec8e P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f19cc6cb1e76460aa6711e49a5775091, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:53:53.108990 21431 raft_consensus.cc:2781] T 1a837e7dce89462596b2b6049069ec8e P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:53.109117 21431 raft_consensus.cc:482] T 1a837e7dce89462596b2b6049069ec8e P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:53.109244 21431 raft_consensus.cc:3037] T 1a837e7dce89462596b2b6049069ec8e P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:53.109203 21430 leader_election.cc:288] T 1a837e7dce89462596b2b6049069ec8e P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:53:53.110234 21432 tablet_copy_client.cc:783] T 6aacf7c4f19546f581c9a54ad171916b P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:53.110693 21432 tablet_copy_client.cc:647] T 6aacf7c4f19546f581c9a54ad171916b P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:53.112418 21431 raft_consensus.cc:504] T 1a837e7dce89462596b2b6049069ec8e P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:53.112962 21432 tablet_copy_client.cc:523] T 6aacf7c4f19546f581c9a54ad171916b P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:53.113615 20232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a837e7dce89462596b2b6049069ec8e" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:53:53.113827 20232 raft_consensus.cc:3037] T 1a837e7dce89462596b2b6049069ec8e P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:53.116534 20232 raft_consensus.cc:2454] T 1a837e7dce89462596b2b6049069ec8e P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 1.
I20240502 01:53:53.116971 20565 leader_election.cc:302] T 1a837e7dce89462596b2b6049069ec8e P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f19cc6cb1e76460aa6711e49a5775091, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:53:53.117229 21430 raft_consensus.cc:2781] T 1a837e7dce89462596b2b6049069ec8e P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:53.117496 21430 raft_consensus.cc:686] T 1a837e7dce89462596b2b6049069ec8e P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Becoming Leader. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Running, Role: LEADER
I20240502 01:53:53.117705 21431 leader_election.cc:288] T 1a837e7dce89462596b2b6049069ec8e P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 election: Requested vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:53:53.117918 17864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a837e7dce89462596b2b6049069ec8e" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:53:53.118121 17864 raft_consensus.cc:3037] T 1a837e7dce89462596b2b6049069ec8e P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:53.120440 17864 raft_consensus.cc:2454] T 1a837e7dce89462596b2b6049069ec8e P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 1.
I20240502 01:53:53.120823 21430 consensus_queue.cc:237] T 1a837e7dce89462596b2b6049069ec8e P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:53.122978 21432 tablet_bootstrap.cc:492] T 6aacf7c4f19546f581c9a54ad171916b P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:53:53.131724 17335 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:53:53.131984 17335 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:53:53.135607 17335 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet 8ddbb3dba5c04928953f2b41e87d0eed
I20240502 01:53:53.136802 17716 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ddbb3dba5c04928953f2b41e87d0eed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:53 UTC) from {username='slave'} at 127.0.0.1:52198
I20240502 01:53:53.137135 20211 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ddbb3dba5c04928953f2b41e87d0eed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:53 UTC) from {username='slave'} at 127.0.0.1:44624
I20240502 01:53:53.139106 21009 raft_consensus.cc:1050] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9: attempting to promote NON_VOTER bb31bceba76d466db24b9341c04900c3 to VOTER
I20240502 01:53:53.140121 21009 consensus_queue.cc:237] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } }
I20240502 01:53:53.142633 21421 tablet_bootstrap.cc:492] T 4bb42883b1a0475b863d008bd01fa70a P f19cc6cb1e76460aa6711e49a5775091: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:53.142709 20608 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ddbb3dba5c04928953f2b41e87d0eed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:53 UTC) from {username='slave'} at 127.0.0.1:33698
I20240502 01:53:53.143143 21421 tablet_bootstrap.cc:492] T 4bb42883b1a0475b863d008bd01fa70a P f19cc6cb1e76460aa6711e49a5775091: Bootstrap complete.
I20240502 01:53:53.143441 21421 ts_tablet_manager.cc:1367] T 4bb42883b1a0475b863d008bd01fa70a P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.090s user 0.049s sys 0.004s
I20240502 01:53:53.144294 21421 raft_consensus.cc:348] T 4bb42883b1a0475b863d008bd01fa70a P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:53:53.144551 21421 raft_consensus.cc:729] T 4bb42883b1a0475b863d008bd01fa70a P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: LEARNER
I20240502 01:53:53.144804 21421 consensus_queue.cc:260] T 4bb42883b1a0475b863d008bd01fa70a P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:53:53.145403 21421 ts_tablet_manager.cc:1398] T 4bb42883b1a0475b863d008bd01fa70a P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:53.146153 17887 tablet_copy_service.cc:342] P bb31bceba76d466db24b9341c04900c3: Request end of tablet copy session f19cc6cb1e76460aa6711e49a5775091-4bb42883b1a0475b863d008bd01fa70a received from {username='slave'} at 127.11.42.72:53629
I20240502 01:53:53.146279 17887 tablet_copy_service.cc:433] P bb31bceba76d466db24b9341c04900c3: ending tablet copy session f19cc6cb1e76460aa6711e49a5775091-4bb42883b1a0475b863d008bd01fa70a on tablet 4bb42883b1a0475b863d008bd01fa70a with peer f19cc6cb1e76460aa6711e49a5775091
I20240502 01:53:53.148861 17335 catalog_manager.cc:5484] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: term changed from 0 to 1, leader changed from <none> to f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72). New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:53.155150 21434 tablet_copy_client.cc:308] T 9642b9b37a784cf881bfc4971d8c9cb6 P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.65:38321
I20240502 01:53:53.155393 17335 catalog_manager.cc:5484] T 1a837e7dce89462596b2b6049069ec8e P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: term changed from 0 to 1, leader changed from <none> to f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72). New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:53.157305 17864 raft_consensus.cc:1261] T 31aa6a26f64f49368ad43cc97c66f2f1 P bb31bceba76d466db24b9341c04900c3 [term 1 LEARNER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:53:53.157253 17496 tablet_copy_service.cc:140] P 4da690c0fa13407a85c0689ce8de9ade: Received BeginTabletCopySession request for tablet 9642b9b37a784cf881bfc4971d8c9cb6 from peer f19cc6cb1e76460aa6711e49a5775091 ({username='slave'} at 127.11.42.72:56975)
I20240502 01:53:53.157454 17496 tablet_copy_service.cc:161] P 4da690c0fa13407a85c0689ce8de9ade: Beginning new tablet copy session on tablet 9642b9b37a784cf881bfc4971d8c9cb6 from peer f19cc6cb1e76460aa6711e49a5775091 at {username='slave'} at 127.11.42.72:56975: session id = f19cc6cb1e76460aa6711e49a5775091-9642b9b37a784cf881bfc4971d8c9cb6
I20240502 01:53:53.157629 20630 raft_consensus.cc:1261] T 31aa6a26f64f49368ad43cc97c66f2f1 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:53:53.158047 20813 consensus_queue.cc:1035] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:53:53.158421 20877 consensus_queue.cc:1035] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:53:53.160274 17496 tablet_copy_source_session.cc:205] T 9642b9b37a784cf881bfc4971d8c9cb6 P 4da690c0fa13407a85c0689ce8de9ade: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:53.163858 21434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9642b9b37a784cf881bfc4971d8c9cb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:53.166162 20232 raft_consensus.cc:1261] T 31aa6a26f64f49368ad43cc97c66f2f1 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:53:53.166730 20813 consensus_queue.cc:1035] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:53:53.180130 21434 tablet_copy_client.cc:783] T 9642b9b37a784cf881bfc4971d8c9cb6 P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:53.180382 21434 tablet_copy_client.cc:647] T 9642b9b37a784cf881bfc4971d8c9cb6 P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:53.205078 21434 tablet_copy_client.cc:523] T 9642b9b37a784cf881bfc4971d8c9cb6 P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:53.209520 21434 tablet_bootstrap.cc:492] T 9642b9b37a784cf881bfc4971d8c9cb6 P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:53:53.210312 21457 tablet_replica.cc:330] T 8ddbb3dba5c04928953f2b41e87d0eed P f19cc6cb1e76460aa6711e49a5775091: stopping tablet replica
I20240502 01:53:53.210598 21457 raft_consensus.cc:2229] T 8ddbb3dba5c04928953f2b41e87d0eed P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:53.211179 21457 raft_consensus.cc:2258] T 8ddbb3dba5c04928953f2b41e87d0eed P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:53.212224 21457 ts_tablet_manager.cc:1875] T 8ddbb3dba5c04928953f2b41e87d0eed P f19cc6cb1e76460aa6711e49a5775091: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:53.213559 21462 ts_tablet_manager.cc:902] T 975f253ff03447ff96fe84169c05097e P 3fceccefba274b1dbbd8216af8b0bc29: Initiating tablet copy from peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:53:53.215708 20813 raft_consensus.cc:2932] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:53:53.224498 20232 raft_consensus.cc:2932] T 31aa6a26f64f49368ad43cc97c66f2f1 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:53:53.225451 17865 raft_consensus.cc:2932] T 31aa6a26f64f49368ad43cc97c66f2f1 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:53:53.227442 21454 tablet_replica.cc:330] T 8ddbb3dba5c04928953f2b41e87d0eed P dfa1152987644bd781b46c7e42a7a685: stopping tablet replica
I20240502 01:53:53.228433 21454 raft_consensus.cc:2229] T 8ddbb3dba5c04928953f2b41e87d0eed P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:53.228698 21454 raft_consensus.cc:2258] T 8ddbb3dba5c04928953f2b41e87d0eed P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:53.230095 17335 catalog_manager.cc:5484] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: config changed from index 2 to 3, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:53.231451 21453 ts_tablet_manager.cc:902] T 52ac2775e02844d195677035796c3847 P 6a36c952c51542c0bf7ea3ec2857a9f9: Initiating tablet copy from peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:53.232795 21454 ts_tablet_manager.cc:1875] T 8ddbb3dba5c04928953f2b41e87d0eed P dfa1152987644bd781b46c7e42a7a685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:53.238183 21455 tablet_replica.cc:330] T 8ddbb3dba5c04928953f2b41e87d0eed P fdc2ac372492454c85387589a91f63af: stopping tablet replica
I20240502 01:53:53.239821 17327 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:53:53.242116 20367 consensus_queue.cc:237] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } }
I20240502 01:53:53.243465 21384 ts_tablet_manager.cc:902] T e42126e16cda4d908b00e2e4a6807d6b P bb31bceba76d466db24b9341c04900c3: Initiating tablet copy from peer 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:53:53.249250 17454 tablet_service.cc:1467] Processing CreateTablet for tablet b7274d4a59be4949a42f218c3edf743d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5b70d9fa9fb44c838d736b49154a72f5]), partition=
I20240502 01:53:53.249617 17454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7274d4a59be4949a42f218c3edf743d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:53.249912 20478 tablet_service.cc:1467] Processing CreateTablet for tablet b7274d4a59be4949a42f218c3edf743d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5b70d9fa9fb44c838d736b49154a72f5]), partition=
I20240502 01:53:53.250300 20478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7274d4a59be4949a42f218c3edf743d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:53.255748 17865 raft_consensus.cc:1261] T 31aa6a26f64f49368ad43cc97c66f2f1 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:53.256451 20813 consensus_queue.cc:1035] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:53.256650 21368 tablet_bootstrap.cc:492] T b7274d4a59be4949a42f218c3edf743d P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:53:53.258271 21368 tablet_bootstrap.cc:654] T b7274d4a59be4949a42f218c3edf743d P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:53.258937 20813 raft_consensus.cc:2932] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:53:53.261792 17865 raft_consensus.cc:2932] T 31aa6a26f64f49368ad43cc97c66f2f1 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:53:53.266304 20608 tablet_service.cc:1467] Processing CreateTablet for tablet b7274d4a59be4949a42f218c3edf743d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5b70d9fa9fb44c838d736b49154a72f5]), partition=
I20240502 01:53:53.267731 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 31aa6a26f64f49368ad43cc97c66f2f1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:53.268229 20608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7274d4a59be4949a42f218c3edf743d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:53.267925 17327 catalog_manager.cc:5484] T 31aa6a26f64f49368ad43cc97c66f2f1 P bb31bceba76d466db24b9341c04900c3 reported cstate change: config changed from index 3 to 4, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:53:53.269656 20232 raft_consensus.cc:1261] T 31aa6a26f64f49368ad43cc97c66f2f1 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:53.270419 20877 consensus_queue.cc:1035] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:53.271986 20232 raft_consensus.cc:2932] T 31aa6a26f64f49368ad43cc97c66f2f1 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:53:53.273734 21455 raft_consensus.cc:2229] T 8ddbb3dba5c04928953f2b41e87d0eed P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:53.274189 21455 raft_consensus.cc:2258] T 8ddbb3dba5c04928953f2b41e87d0eed P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:53.275532 21455 ts_tablet_manager.cc:1875] T 8ddbb3dba5c04928953f2b41e87d0eed P fdc2ac372492454c85387589a91f63af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:53.276031 20610 tablet_service.cc:1514] Processing DeleteTablet for tablet 31aa6a26f64f49368ad43cc97c66f2f1 with delete_type TABLET_DATA_TOMBSTONED (TS f19cc6cb1e76460aa6711e49a5775091 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33698
I20240502 01:53:53.282882 21457 ts_tablet_manager.cc:1888] T 8ddbb3dba5c04928953f2b41e87d0eed P f19cc6cb1e76460aa6711e49a5775091: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:53:53.294420 21469 tablet_bootstrap.cc:492] T b7274d4a59be4949a42f218c3edf743d P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:53.296053 21009 consensus_queue.cc:1035] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:53.296430 21469 tablet_bootstrap.cc:654] T b7274d4a59be4949a42f218c3edf743d P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:53.300374 21453 tablet_copy_client.cc:308] T 52ac2775e02844d195677035796c3847 P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.66:38531
I20240502 01:53:53.286639 21253 tablet_bootstrap.cc:492] T b7274d4a59be4949a42f218c3edf743d P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:53:53.310534 21253 tablet_bootstrap.cc:654] T b7274d4a59be4949a42f218c3edf743d P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:53.312438 21457 log.cc:1196] T 8ddbb3dba5c04928953f2b41e87d0eed P f19cc6cb1e76460aa6711e49a5775091: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal/wals/8ddbb3dba5c04928953f2b41e87d0eed
I20240502 01:53:53.313051 21457 ts_tablet_manager.cc:1909] T 8ddbb3dba5c04928953f2b41e87d0eed P f19cc6cb1e76460aa6711e49a5775091: Deleting consensus metadata
I20240502 01:53:53.313236 21253 tablet_bootstrap.cc:492] T b7274d4a59be4949a42f218c3edf743d P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:53:53.313217 21462 tablet_copy_client.cc:308] T 975f253ff03447ff96fe84169c05097e P 3fceccefba274b1dbbd8216af8b0bc29: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.72:42411
I20240502 01:53:53.313491 21253 ts_tablet_manager.cc:1367] T b7274d4a59be4949a42f218c3edf743d P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.001s
I20240502 01:53:53.314217 17321 catalog_manager.cc:4830] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): tablet 8ddbb3dba5c04928953f2b41e87d0eed (table rebalancer_dynamic_table_new_name [id=1cfb3501e7c3445cacb6edd5bd5f14af]) successfully deleted
I20240502 01:53:53.314234 20650 tablet_copy_service.cc:140] P f19cc6cb1e76460aa6711e49a5775091: Received BeginTabletCopySession request for tablet 975f253ff03447ff96fe84169c05097e from peer 3fceccefba274b1dbbd8216af8b0bc29 ({username='slave'} at 127.11.42.66:54577)
I20240502 01:53:53.314461 20650 tablet_copy_service.cc:161] P f19cc6cb1e76460aa6711e49a5775091: Beginning new tablet copy session on tablet 975f253ff03447ff96fe84169c05097e from peer 3fceccefba274b1dbbd8216af8b0bc29 at {username='slave'} at 127.11.42.66:54577: session id = 3fceccefba274b1dbbd8216af8b0bc29-975f253ff03447ff96fe84169c05097e
I20240502 01:53:53.314400 21253 raft_consensus.cc:348] T b7274d4a59be4949a42f218c3edf743d P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:53.314618 21253 raft_consensus.cc:374] T b7274d4a59be4949a42f218c3edf743d P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:53.314689 21253 raft_consensus.cc:729] T b7274d4a59be4949a42f218c3edf743d P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:53:53.314946 21253 consensus_queue.cc:260] T b7274d4a59be4949a42f218c3edf743d P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:53.315580 21253 ts_tablet_manager.cc:1398] T b7274d4a59be4949a42f218c3edf743d P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:53.318349 20650 tablet_copy_source_session.cc:205] T 975f253ff03447ff96fe84169c05097e P f19cc6cb1e76460aa6711e49a5775091: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:53.320818 21457 tablet_replica.cc:330] T 31aa6a26f64f49368ad43cc97c66f2f1 P f19cc6cb1e76460aa6711e49a5775091: stopping tablet replica
I20240502 01:53:53.321110 21457 raft_consensus.cc:2229] T 31aa6a26f64f49368ad43cc97c66f2f1 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:53.321570 21368 tablet_bootstrap.cc:492] T b7274d4a59be4949a42f218c3edf743d P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:53:53.321880 21368 ts_tablet_manager.cc:1367] T b7274d4a59be4949a42f218c3edf743d P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.065s user 0.004s sys 0.010s
I20240502 01:53:53.322706 21368 raft_consensus.cc:348] T b7274d4a59be4949a42f218c3edf743d P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:53.322999 21368 raft_consensus.cc:374] T b7274d4a59be4949a42f218c3edf743d P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:53.323068 21368 raft_consensus.cc:729] T b7274d4a59be4949a42f218c3edf743d P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:53:53.323380 21368 consensus_queue.cc:260] T b7274d4a59be4949a42f218c3edf743d P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:53.323905 21368 ts_tablet_manager.cc:1398] T b7274d4a59be4949a42f218c3edf743d P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:53:53.325525 21457 pending_rounds.cc:62] T 31aa6a26f64f49368ad43cc97c66f2f1 P f19cc6cb1e76460aa6711e49a5775091: Trying to abort 1 pending ops.
I20240502 01:53:53.325656 21457 pending_rounds.cc:68] T 31aa6a26f64f49368ad43cc97c66f2f1 P f19cc6cb1e76460aa6711e49a5775091: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7023062356539568128 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "31aa6a26f64f49368ad43cc97c66f2f1" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } } }
I20240502 01:53:53.326159 21457 raft_consensus.cc:2894] T 31aa6a26f64f49368ad43cc97c66f2f1 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240502 01:53:53.329202 21457 raft_consensus.cc:2866] T 31aa6a26f64f49368ad43cc97c66f2f1 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 01:53:53.332434 21457 raft_consensus.cc:2258] T 31aa6a26f64f49368ad43cc97c66f2f1 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:53.333464 21469 tablet_bootstrap.cc:492] T b7274d4a59be4949a42f218c3edf743d P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:53.333642 21469 ts_tablet_manager.cc:1367] T b7274d4a59be4949a42f218c3edf743d P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.000s
I20240502 01:53:53.334399 21469 raft_consensus.cc:348] T b7274d4a59be4949a42f218c3edf743d P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:53.334630 21469 raft_consensus.cc:374] T b7274d4a59be4949a42f218c3edf743d P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:53.334738 21469 raft_consensus.cc:729] T b7274d4a59be4949a42f218c3edf743d P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:53.334964 21469 consensus_queue.cc:260] T b7274d4a59be4949a42f218c3edf743d P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:53.336539 21469 ts_tablet_manager.cc:1398] T b7274d4a59be4949a42f218c3edf743d P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:53:53.336880 21457 ts_tablet_manager.cc:1875] T 31aa6a26f64f49368ad43cc97c66f2f1 P f19cc6cb1e76460aa6711e49a5775091: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:53.345386 21454 ts_tablet_manager.cc:1888] T 8ddbb3dba5c04928953f2b41e87d0eed P dfa1152987644bd781b46c7e42a7a685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:53:53.345597 21454 log.cc:1196] T 8ddbb3dba5c04928953f2b41e87d0eed P dfa1152987644bd781b46c7e42a7a685: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/wals/8ddbb3dba5c04928953f2b41e87d0eed
I20240502 01:53:53.346886 21454 ts_tablet_manager.cc:1909] T 8ddbb3dba5c04928953f2b41e87d0eed P dfa1152987644bd781b46c7e42a7a685: Deleting consensus metadata
I20240502 01:53:53.347674 21384 tablet_copy_client.cc:308] T e42126e16cda4d908b00e2e4a6807d6b P bb31bceba76d466db24b9341c04900c3: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.70:39155
I20240502 01:53:53.348348 20387 tablet_copy_service.cc:140] P 6a36c952c51542c0bf7ea3ec2857a9f9: Received BeginTabletCopySession request for tablet e42126e16cda4d908b00e2e4a6807d6b from peer bb31bceba76d466db24b9341c04900c3 ({username='slave'} at 127.11.42.68:34031)
I20240502 01:53:53.348505 20387 tablet_copy_service.cc:161] P 6a36c952c51542c0bf7ea3ec2857a9f9: Beginning new tablet copy session on tablet e42126e16cda4d908b00e2e4a6807d6b from peer bb31bceba76d466db24b9341c04900c3 at {username='slave'} at 127.11.42.68:34031: session id = bb31bceba76d466db24b9341c04900c3-e42126e16cda4d908b00e2e4a6807d6b
I20240502 01:53:53.351418 20387 tablet_copy_source_session.cc:205] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:53.354192 21384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e42126e16cda4d908b00e2e4a6807d6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:53.359186 21462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 975f253ff03447ff96fe84169c05097e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:53.359158 17321 catalog_manager.cc:4830] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): tablet 8ddbb3dba5c04928953f2b41e87d0eed (table rebalancer_dynamic_table_new_name [id=1cfb3501e7c3445cacb6edd5bd5f14af]) successfully deleted
I20240502 01:53:53.362438 21457 ts_tablet_manager.cc:1888] T 31aa6a26f64f49368ad43cc97c66f2f1 P f19cc6cb1e76460aa6711e49a5775091: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:53:53.362576 21457 log.cc:1196] T 31aa6a26f64f49368ad43cc97c66f2f1 P f19cc6cb1e76460aa6711e49a5775091: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal/wals/31aa6a26f64f49368ad43cc97c66f2f1
I20240502 01:53:53.363329 17321 catalog_manager.cc:4830] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): tablet 31aa6a26f64f49368ad43cc97c66f2f1 (table rebalancer_extra_table_0 [id=2da12dd1e8bc4159bc3e1040e3db33fe]) successfully deleted
I20240502 01:53:53.365039 21455 ts_tablet_manager.cc:1888] T 8ddbb3dba5c04928953f2b41e87d0eed P fdc2ac372492454c85387589a91f63af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:53:53.366223 17627 tablet_copy_service.cc:140] P 3fceccefba274b1dbbd8216af8b0bc29: Received BeginTabletCopySession request for tablet 52ac2775e02844d195677035796c3847 from peer 6a36c952c51542c0bf7ea3ec2857a9f9 ({username='slave'} at 127.11.42.70:53247)
I20240502 01:53:53.366400 17627 tablet_copy_service.cc:161] P 3fceccefba274b1dbbd8216af8b0bc29: Beginning new tablet copy session on tablet 52ac2775e02844d195677035796c3847 from peer 6a36c952c51542c0bf7ea3ec2857a9f9 at {username='slave'} at 127.11.42.70:53247: session id = 6a36c952c51542c0bf7ea3ec2857a9f9-52ac2775e02844d195677035796c3847
I20240502 01:53:53.369707 17627 tablet_copy_source_session.cc:205] T 52ac2775e02844d195677035796c3847 P 3fceccefba274b1dbbd8216af8b0bc29: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:53.371308 21453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52ac2775e02844d195677035796c3847. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:53.372485 20877 consensus_queue.cc:1035] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:53.379359 21455 log.cc:1196] T 8ddbb3dba5c04928953f2b41e87d0eed P fdc2ac372492454c85387589a91f63af: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal/wals/8ddbb3dba5c04928953f2b41e87d0eed
I20240502 01:53:53.380009 21455 ts_tablet_manager.cc:1909] T 8ddbb3dba5c04928953f2b41e87d0eed P fdc2ac372492454c85387589a91f63af: Deleting consensus metadata
I20240502 01:53:53.386272 17322 catalog_manager.cc:4830] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): tablet 8ddbb3dba5c04928953f2b41e87d0eed (table rebalancer_dynamic_table_new_name [id=1cfb3501e7c3445cacb6edd5bd5f14af]) successfully deleted
I20240502 01:53:53.391120 21453 tablet_copy_client.cc:783] T 52ac2775e02844d195677035796c3847 P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:53.391491 21453 tablet_copy_client.cc:647] T 52ac2775e02844d195677035796c3847 P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:53.393769 21453 tablet_copy_client.cc:523] T 52ac2775e02844d195677035796c3847 P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:53.398684 21384 tablet_copy_client.cc:783] T e42126e16cda4d908b00e2e4a6807d6b P bb31bceba76d466db24b9341c04900c3: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:53.399075 21384 tablet_copy_client.cc:647] T e42126e16cda4d908b00e2e4a6807d6b P bb31bceba76d466db24b9341c04900c3: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:53.402710 17865 raft_consensus.cc:1203] T 62887e332443457f8a5b41a982354dc4 P bb31bceba76d466db24b9341c04900c3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:53:53.403010 21384 tablet_copy_client.cc:523] T e42126e16cda4d908b00e2e4a6807d6b P bb31bceba76d466db24b9341c04900c3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:53.409394 21453 tablet_bootstrap.cc:492] T 52ac2775e02844d195677035796c3847 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:53:53.416155 21462 tablet_copy_client.cc:783] T 975f253ff03447ff96fe84169c05097e P 3fceccefba274b1dbbd8216af8b0bc29: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:53.416502 21462 tablet_copy_client.cc:647] T 975f253ff03447ff96fe84169c05097e P 3fceccefba274b1dbbd8216af8b0bc29: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:53.417222 21384 tablet_bootstrap.cc:492] T e42126e16cda4d908b00e2e4a6807d6b P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:53.419330 21462 tablet_copy_client.cc:523] T 975f253ff03447ff96fe84169c05097e P 3fceccefba274b1dbbd8216af8b0bc29: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:53.424651 21462 tablet_bootstrap.cc:492] T 975f253ff03447ff96fe84169c05097e P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:53.425102 21421 ts_tablet_manager.cc:902] T 1f01949cf70440b38e59a5020fd48c44 P f19cc6cb1e76460aa6711e49a5775091: Initiating tablet copy from peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:53.433110 20630 raft_consensus.cc:1203] T 4bb42883b1a0475b863d008bd01fa70a P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:53:53.440982 21384 tablet_bootstrap.cc:492] T e42126e16cda4d908b00e2e4a6807d6b P bb31bceba76d466db24b9341c04900c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:53.441654 21384 tablet_bootstrap.cc:492] T e42126e16cda4d908b00e2e4a6807d6b P bb31bceba76d466db24b9341c04900c3: Bootstrap complete.
I20240502 01:53:53.442024 21384 ts_tablet_manager.cc:1367] T e42126e16cda4d908b00e2e4a6807d6b P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.000s
I20240502 01:53:53.442401 21462 tablet_bootstrap.cc:492] T 975f253ff03447ff96fe84169c05097e P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:53.442883 21462 tablet_bootstrap.cc:492] T 975f253ff03447ff96fe84169c05097e P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap complete.
I20240502 01:53:53.443130 21462 ts_tablet_manager.cc:1367] T 975f253ff03447ff96fe84169c05097e P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.001s
I20240502 01:53:53.443030 21384 raft_consensus.cc:348] T e42126e16cda4d908b00e2e4a6807d6b P bb31bceba76d466db24b9341c04900c3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } }
I20240502 01:53:53.443367 21384 raft_consensus.cc:729] T e42126e16cda4d908b00e2e4a6807d6b P bb31bceba76d466db24b9341c04900c3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: LEARNER
I20240502 01:53:53.443826 21384 consensus_queue.cc:260] T e42126e16cda4d908b00e2e4a6807d6b P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } }
I20240502 01:53:53.444259 21462 raft_consensus.cc:348] T 975f253ff03447ff96fe84169c05097e P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } }
I20240502 01:53:53.444526 21462 raft_consensus.cc:729] T 975f253ff03447ff96fe84169c05097e P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: LEARNER
I20240502 01:53:53.444772 21462 consensus_queue.cc:260] T 975f253ff03447ff96fe84169c05097e P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } }
I20240502 01:53:53.445151 21384 ts_tablet_manager.cc:1398] T e42126e16cda4d908b00e2e4a6807d6b P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:53.446216 20387 tablet_copy_service.cc:342] P 6a36c952c51542c0bf7ea3ec2857a9f9: Request end of tablet copy session bb31bceba76d466db24b9341c04900c3-e42126e16cda4d908b00e2e4a6807d6b received from {username='slave'} at 127.11.42.68:34031
I20240502 01:53:53.446359 20387 tablet_copy_service.cc:433] P 6a36c952c51542c0bf7ea3ec2857a9f9: ending tablet copy session bb31bceba76d466db24b9341c04900c3-e42126e16cda4d908b00e2e4a6807d6b on tablet e42126e16cda4d908b00e2e4a6807d6b with peer bb31bceba76d466db24b9341c04900c3
I20240502 01:53:53.448565 21462 ts_tablet_manager.cc:1398] T 975f253ff03447ff96fe84169c05097e P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 01:53:53.452051 20650 tablet_copy_service.cc:342] P f19cc6cb1e76460aa6711e49a5775091: Request end of tablet copy session 3fceccefba274b1dbbd8216af8b0bc29-975f253ff03447ff96fe84169c05097e received from {username='slave'} at 127.11.42.66:54577
I20240502 01:53:53.452219 20650 tablet_copy_service.cc:433] P f19cc6cb1e76460aa6711e49a5775091: ending tablet copy session 3fceccefba274b1dbbd8216af8b0bc29-975f253ff03447ff96fe84169c05097e on tablet 975f253ff03447ff96fe84169c05097e with peer 3fceccefba274b1dbbd8216af8b0bc29
I20240502 01:53:53.455650 21363 raft_consensus.cc:482] T b7274d4a59be4949a42f218c3edf743d P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:53.455809 21363 raft_consensus.cc:504] T b7274d4a59be4949a42f218c3edf743d P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:53.456533 21363 leader_election.cc:288] T b7274d4a59be4949a42f218c3edf743d P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:53:53.456935 20498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7274d4a59be4949a42f218c3edf743d" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
I20240502 01:53:53.457129 20630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7274d4a59be4949a42f218c3edf743d" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091" is_pre_election: true
I20240502 01:53:53.457218 20498 raft_consensus.cc:2454] T b7274d4a59be4949a42f218c3edf743d P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 0.
I20240502 01:53:53.457329 20630 raft_consensus.cc:2454] T b7274d4a59be4949a42f218c3edf743d P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 0.
I20240502 01:53:53.458665 17411 leader_election.cc:302] T b7274d4a59be4949a42f218c3edf743d P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:53:53.458974 21363 raft_consensus.cc:2781] T b7274d4a59be4949a42f218c3edf743d P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:53.459115 21363 raft_consensus.cc:482] T b7274d4a59be4949a42f218c3edf743d P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:53.459209 21363 raft_consensus.cc:3037] T b7274d4a59be4949a42f218c3edf743d P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:53.461860 21363 raft_consensus.cc:504] T b7274d4a59be4949a42f218c3edf743d P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:53.462805 21363 leader_election.cc:288] T b7274d4a59be4949a42f218c3edf743d P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Requested vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:53:53.463078 20498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7274d4a59be4949a42f218c3edf743d" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
I20240502 01:53:53.463248 20498 raft_consensus.cc:3037] T b7274d4a59be4949a42f218c3edf743d P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:53.464962 20498 raft_consensus.cc:2454] T b7274d4a59be4949a42f218c3edf743d P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
I20240502 01:53:53.465322 20630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7274d4a59be4949a42f218c3edf743d" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091"
I20240502 01:53:53.465538 20630 raft_consensus.cc:3037] T b7274d4a59be4949a42f218c3edf743d P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:53.467615 20630 raft_consensus.cc:2454] T b7274d4a59be4949a42f218c3edf743d P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
I20240502 01:53:53.468006 17412 leader_election.cc:302] T b7274d4a59be4949a42f218c3edf743d P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 4da690c0fa13407a85c0689ce8de9ade; no voters:
I20240502 01:53:53.468298 21363 raft_consensus.cc:2781] T b7274d4a59be4949a42f218c3edf743d P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:53.472039 21363 raft_consensus.cc:686] T b7274d4a59be4949a42f218c3edf743d P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Becoming Leader. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Running, Role: LEADER
I20240502 01:53:53.472371 21363 consensus_queue.cc:237] T b7274d4a59be4949a42f218c3edf743d P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:53.472513 21421 tablet_copy_client.cc:308] T 1f01949cf70440b38e59a5020fd48c44 P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.66:38531
I20240502 01:53:53.473244 17627 tablet_copy_service.cc:140] P 3fceccefba274b1dbbd8216af8b0bc29: Received BeginTabletCopySession request for tablet 1f01949cf70440b38e59a5020fd48c44 from peer f19cc6cb1e76460aa6711e49a5775091 ({username='slave'} at 127.11.42.72:53885)
I20240502 01:53:53.473413 17627 tablet_copy_service.cc:161] P 3fceccefba274b1dbbd8216af8b0bc29: Beginning new tablet copy session on tablet 1f01949cf70440b38e59a5020fd48c44 from peer f19cc6cb1e76460aa6711e49a5775091 at {username='slave'} at 127.11.42.72:53885: session id = f19cc6cb1e76460aa6711e49a5775091-1f01949cf70440b38e59a5020fd48c44
I20240502 01:53:53.475540 17627 tablet_copy_source_session.cc:205] T 1f01949cf70440b38e59a5020fd48c44 P 3fceccefba274b1dbbd8216af8b0bc29: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:53.477190 21421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f01949cf70440b38e59a5020fd48c44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:53.484695 21421 tablet_copy_client.cc:783] T 1f01949cf70440b38e59a5020fd48c44 P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:53.484921 21421 tablet_copy_client.cc:647] T 1f01949cf70440b38e59a5020fd48c44 P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:53.486610 21421 tablet_copy_client.cc:523] T 1f01949cf70440b38e59a5020fd48c44 P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:53.487730 21434 tablet_bootstrap.cc:492] T 9642b9b37a784cf881bfc4971d8c9cb6 P f19cc6cb1e76460aa6711e49a5775091: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:53.493399 21434 tablet_bootstrap.cc:492] T 9642b9b37a784cf881bfc4971d8c9cb6 P f19cc6cb1e76460aa6711e49a5775091: Bootstrap complete.
I20240502 01:53:53.493692 21434 ts_tablet_manager.cc:1367] T 9642b9b37a784cf881bfc4971d8c9cb6 P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.284s user 0.043s sys 0.005s
I20240502 01:53:53.493707 17327 catalog_manager.cc:5484] T b7274d4a59be4949a42f218c3edf743d P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: term changed from 0 to 1, leader changed from <none> to 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65). New cstate: current_term: 1 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:53.494390 21421 tablet_bootstrap.cc:492] T 1f01949cf70440b38e59a5020fd48c44 P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:53:53.501097 21432 tablet_bootstrap.cc:492] T 6aacf7c4f19546f581c9a54ad171916b P 2f306549407b4bb281ccf2994035e4f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:53.501586 21432 tablet_bootstrap.cc:492] T 6aacf7c4f19546f581c9a54ad171916b P 2f306549407b4bb281ccf2994035e4f1: Bootstrap complete.
I20240502 01:53:53.501960 21432 ts_tablet_manager.cc:1367] T 6aacf7c4f19546f581c9a54ad171916b P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.379s user 0.048s sys 0.002s
I20240502 01:53:53.503926 21506 ts_tablet_manager.cc:902] T ebb9bca9f1bd41b0857939909b1110e4 P 2f306549407b4bb281ccf2994035e4f1: Initiating tablet copy from peer 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:53:53.503805 21434 raft_consensus.cc:348] T 9642b9b37a784cf881bfc4971d8c9cb6 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:53:53.505779 21434 raft_consensus.cc:729] T 9642b9b37a784cf881bfc4971d8c9cb6 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: LEARNER
I20240502 01:53:53.506067 21434 consensus_queue.cc:260] T 9642b9b37a784cf881bfc4971d8c9cb6 P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:53:53.507936 21434 ts_tablet_manager.cc:1398] T 9642b9b37a784cf881bfc4971d8c9cb6 P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240502 01:53:53.508327 21432 raft_consensus.cc:348] T 6aacf7c4f19546f581c9a54ad171916b P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
I20240502 01:53:53.508594 21432 raft_consensus.cc:729] T 6aacf7c4f19546f581c9a54ad171916b P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: LEARNER
I20240502 01:53:53.508807 21432 consensus_queue.cc:260] T 6aacf7c4f19546f581c9a54ad171916b P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
I20240502 01:53:53.511708 21432 ts_tablet_manager.cc:1398] T 6aacf7c4f19546f581c9a54ad171916b P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240502 01:53:53.512421 17627 tablet_copy_service.cc:342] P 3fceccefba274b1dbbd8216af8b0bc29: Request end of tablet copy session 2f306549407b4bb281ccf2994035e4f1-6aacf7c4f19546f581c9a54ad171916b received from {username='slave'} at 127.11.42.71:49447
I20240502 01:53:53.512612 17627 tablet_copy_service.cc:433] P 3fceccefba274b1dbbd8216af8b0bc29: ending tablet copy session 2f306549407b4bb281ccf2994035e4f1-6aacf7c4f19546f581c9a54ad171916b on tablet 6aacf7c4f19546f581c9a54ad171916b with peer 2f306549407b4bb281ccf2994035e4f1
I20240502 01:53:53.513813 17496 tablet_copy_service.cc:342] P 4da690c0fa13407a85c0689ce8de9ade: Request end of tablet copy session f19cc6cb1e76460aa6711e49a5775091-9642b9b37a784cf881bfc4971d8c9cb6 received from {username='slave'} at 127.11.42.72:56975
I20240502 01:53:53.513947 17496 tablet_copy_service.cc:433] P 4da690c0fa13407a85c0689ce8de9ade: ending tablet copy session f19cc6cb1e76460aa6711e49a5775091-9642b9b37a784cf881bfc4971d8c9cb6 on tablet 9642b9b37a784cf881bfc4971d8c9cb6 with peer f19cc6cb1e76460aa6711e49a5775091
I20240502 01:53:53.532133 21513 ts_tablet_manager.cc:902] T 65bdab2c9fba48a6828a864278319cc0 P 6a36c952c51542c0bf7ea3ec2857a9f9: Initiating tablet copy from peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:53:53.545675 21453 tablet_bootstrap.cc:492] T 52ac2775e02844d195677035796c3847 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:53.546152 21453 tablet_bootstrap.cc:492] T 52ac2775e02844d195677035796c3847 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap complete.
I20240502 01:53:53.546530 21453 ts_tablet_manager.cc:1367] T 52ac2775e02844d195677035796c3847 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.137s user 0.048s sys 0.000s
I20240502 01:53:53.547537 21453 raft_consensus.cc:348] T 52ac2775e02844d195677035796c3847 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } }
I20240502 01:53:53.547828 21453 raft_consensus.cc:729] T 52ac2775e02844d195677035796c3847 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: LEARNER
I20240502 01:53:53.548060 21453 consensus_queue.cc:260] T 52ac2775e02844d195677035796c3847 P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } }
I20240502 01:53:53.552353 21453 ts_tablet_manager.cc:1398] T 52ac2775e02844d195677035796c3847 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:53.553695 17627 tablet_copy_service.cc:342] P 3fceccefba274b1dbbd8216af8b0bc29: Request end of tablet copy session 6a36c952c51542c0bf7ea3ec2857a9f9-52ac2775e02844d195677035796c3847 received from {username='slave'} at 127.11.42.70:53247
I20240502 01:53:53.553858 17627 tablet_copy_service.cc:433] P 3fceccefba274b1dbbd8216af8b0bc29: ending tablet copy session 6a36c952c51542c0bf7ea3ec2857a9f9-52ac2775e02844d195677035796c3847 on tablet 52ac2775e02844d195677035796c3847 with peer 6a36c952c51542c0bf7ea3ec2857a9f9
I20240502 01:53:53.562530 21506 tablet_copy_client.cc:308] T ebb9bca9f1bd41b0857939909b1110e4 P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.70:39155
I20240502 01:53:53.562556 21421 tablet_bootstrap.cc:492] T 1f01949cf70440b38e59a5020fd48c44 P f19cc6cb1e76460aa6711e49a5775091: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:53.562983 21421 tablet_bootstrap.cc:492] T 1f01949cf70440b38e59a5020fd48c44 P f19cc6cb1e76460aa6711e49a5775091: Bootstrap complete.
I20240502 01:53:53.563254 21421 ts_tablet_manager.cc:1367] T 1f01949cf70440b38e59a5020fd48c44 P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.069s user 0.039s sys 0.010s
I20240502 01:53:53.564075 21421 raft_consensus.cc:348] T 1f01949cf70440b38e59a5020fd48c44 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:53:53.564311 21421 raft_consensus.cc:729] T 1f01949cf70440b38e59a5020fd48c44 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: LEARNER
I20240502 01:53:53.564532 21421 consensus_queue.cc:260] T 1f01949cf70440b38e59a5020fd48c44 P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:53:53.565132 21421 ts_tablet_manager.cc:1398] T 1f01949cf70440b38e59a5020fd48c44 P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 01:53:53.565711 20387 tablet_copy_service.cc:140] P 6a36c952c51542c0bf7ea3ec2857a9f9: Received BeginTabletCopySession request for tablet ebb9bca9f1bd41b0857939909b1110e4 from peer 2f306549407b4bb281ccf2994035e4f1 ({username='slave'} at 127.11.42.71:39255)
I20240502 01:53:53.565938 20387 tablet_copy_service.cc:161] P 6a36c952c51542c0bf7ea3ec2857a9f9: Beginning new tablet copy session on tablet ebb9bca9f1bd41b0857939909b1110e4 from peer 2f306549407b4bb281ccf2994035e4f1 at {username='slave'} at 127.11.42.71:39255: session id = 2f306549407b4bb281ccf2994035e4f1-ebb9bca9f1bd41b0857939909b1110e4
I20240502 01:53:53.569197 20387 tablet_copy_source_session.cc:205] T ebb9bca9f1bd41b0857939909b1110e4 P 6a36c952c51542c0bf7ea3ec2857a9f9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:53.569471 21430 consensus_queue.cc:1035] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:53.569751 17627 tablet_copy_service.cc:342] P 3fceccefba274b1dbbd8216af8b0bc29: Request end of tablet copy session f19cc6cb1e76460aa6711e49a5775091-1f01949cf70440b38e59a5020fd48c44 received from {username='slave'} at 127.11.42.72:53885
I20240502 01:53:53.569890 17627 tablet_copy_service.cc:433] P 3fceccefba274b1dbbd8216af8b0bc29: ending tablet copy session f19cc6cb1e76460aa6711e49a5775091-1f01949cf70440b38e59a5020fd48c44 on tablet 1f01949cf70440b38e59a5020fd48c44 with peer f19cc6cb1e76460aa6711e49a5775091
I20240502 01:53:53.573927 21506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebb9bca9f1bd41b0857939909b1110e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:53.581305 21430 consensus_queue.cc:1035] T 1a837e7dce89462596b2b6049069ec8e P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:53.607784 21429 consensus_queue.cc:1035] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:53.608631 21506 tablet_copy_client.cc:783] T ebb9bca9f1bd41b0857939909b1110e4 P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:53.609050 21506 tablet_copy_client.cc:647] T ebb9bca9f1bd41b0857939909b1110e4 P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:53.609714 20630 raft_consensus.cc:1203] T 9642b9b37a784cf881bfc4971d8c9cb6 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:53:53.610806 21506 tablet_copy_client.cc:523] T ebb9bca9f1bd41b0857939909b1110e4 P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:53.613389 21430 consensus_queue.cc:1035] T 1a837e7dce89462596b2b6049069ec8e P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:53.619446 21506 tablet_bootstrap.cc:492] T ebb9bca9f1bd41b0857939909b1110e4 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:53:53.628314 20498 raft_consensus.cc:1203] T 6aacf7c4f19546f581c9a54ad171916b P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:53:53.638667 21506 tablet_bootstrap.cc:492] T ebb9bca9f1bd41b0857939909b1110e4 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:53.639197 21506 tablet_bootstrap.cc:492] T ebb9bca9f1bd41b0857939909b1110e4 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap complete.
I20240502 01:53:53.639467 21506 ts_tablet_manager.cc:1367] T ebb9bca9f1bd41b0857939909b1110e4 P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.006s
I20240502 01:53:53.640290 21506 raft_consensus.cc:348] T ebb9bca9f1bd41b0857939909b1110e4 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
I20240502 01:53:53.640530 21506 raft_consensus.cc:729] T ebb9bca9f1bd41b0857939909b1110e4 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: LEARNER
I20240502 01:53:53.642491 21506 consensus_queue.cc:260] T ebb9bca9f1bd41b0857939909b1110e4 P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
I20240502 01:53:53.646458 21506 ts_tablet_manager.cc:1398] T ebb9bca9f1bd41b0857939909b1110e4 P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20240502 01:53:53.647146 20387 tablet_copy_service.cc:342] P 6a36c952c51542c0bf7ea3ec2857a9f9: Request end of tablet copy session 2f306549407b4bb281ccf2994035e4f1-ebb9bca9f1bd41b0857939909b1110e4 received from {username='slave'} at 127.11.42.71:39255
I20240502 01:53:53.647326 20387 tablet_copy_service.cc:433] P 6a36c952c51542c0bf7ea3ec2857a9f9: ending tablet copy session 2f306549407b4bb281ccf2994035e4f1-ebb9bca9f1bd41b0857939909b1110e4 on tablet ebb9bca9f1bd41b0857939909b1110e4 with peer 2f306549407b4bb281ccf2994035e4f1
I20240502 01:53:53.653959 17866 raft_consensus.cc:1203] T e42126e16cda4d908b00e2e4a6807d6b P bb31bceba76d466db24b9341c04900c3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:53:53.657886 21513 tablet_copy_client.cc:308] T 65bdab2c9fba48a6828a864278319cc0 P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.68:44083
I20240502 01:53:53.658468 17887 tablet_copy_service.cc:140] P bb31bceba76d466db24b9341c04900c3: Received BeginTabletCopySession request for tablet 65bdab2c9fba48a6828a864278319cc0 from peer 6a36c952c51542c0bf7ea3ec2857a9f9 ({username='slave'} at 127.11.42.70:41203)
I20240502 01:53:53.658615 17887 tablet_copy_service.cc:161] P bb31bceba76d466db24b9341c04900c3: Beginning new tablet copy session on tablet 65bdab2c9fba48a6828a864278319cc0 from peer 6a36c952c51542c0bf7ea3ec2857a9f9 at {username='slave'} at 127.11.42.70:41203: session id = 6a36c952c51542c0bf7ea3ec2857a9f9-65bdab2c9fba48a6828a864278319cc0
I20240502 01:53:53.661468 17887 tablet_copy_source_session.cc:205] T 65bdab2c9fba48a6828a864278319cc0 P bb31bceba76d466db24b9341c04900c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:53.661386 17334 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:53:53.662642 21513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65bdab2c9fba48a6828a864278319cc0. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 01:53:53.674080 17672 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.42.65:38321, user_credentials={real_user=slave}} blocked reactor thread for 67412us
I20240502 01:53:53.676460 21513 tablet_copy_client.cc:783] T 65bdab2c9fba48a6828a864278319cc0 P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:53.676707 21513 tablet_copy_client.cc:647] T 65bdab2c9fba48a6828a864278319cc0 P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:53.688550 21513 tablet_copy_client.cc:523] T 65bdab2c9fba48a6828a864278319cc0 P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:53.697429 21513 tablet_bootstrap.cc:492] T 65bdab2c9fba48a6828a864278319cc0 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:53:53.698873 17327 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_extra_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:53.707448 20367 raft_consensus.cc:1203] T 52ac2775e02844d195677035796c3847 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:53:53.717265 20630 raft_consensus.cc:1261] T b7274d4a59be4949a42f218c3edf743d P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:53.717952 21363 consensus_queue.cc:1035] T b7274d4a59be4949a42f218c3edf743d P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:53.719230 20498 raft_consensus.cc:1261] T b7274d4a59be4949a42f218c3edf743d P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:53.720203 21363 consensus_queue.cc:1035] T b7274d4a59be4949a42f218c3edf743d P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:53.733379 17334 catalog_manager.cc:5576] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411) does not have the latest schema for tablet b7274d4a59be4949a42f218c3edf743d (table rebalancer_dynamic_table [id=5b70d9fa9fb44c838d736b49154a72f5]). Expected version 1 got 0
I20240502 01:53:53.733781 20211 tablet_service.cc:1467] Processing CreateTablet for tablet 8eed067f639542d487de971761e91931 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=ecb981650f8c425798b67a5194ad793c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:53.734040 20347 tablet_service.cc:1467] Processing CreateTablet for tablet 8eed067f639542d487de971761e91931 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=ecb981650f8c425798b67a5194ad793c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:53.734215 20211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eed067f639542d487de971761e91931. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:53.734428 20347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eed067f639542d487de971761e91931. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:53.739217 20610 tablet_service.cc:1467] Processing CreateTablet for tablet 139304da92ab4f48b8a08fa24ed60c95 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=ecb981650f8c425798b67a5194ad793c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:53.739614 20610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 139304da92ab4f48b8a08fa24ed60c95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:53.739640 20345 tablet_service.cc:1467] Processing CreateTablet for tablet 472d7827b3074236a81a3ede098f6c0a (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=ecb981650f8c425798b67a5194ad793c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:53.739951 20345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 472d7827b3074236a81a3ede098f6c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:53.741506 20608 tablet_service.cc:1467] Processing CreateTablet for tablet 8eed067f639542d487de971761e91931 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=ecb981650f8c425798b67a5194ad793c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:53.741904 20608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eed067f639542d487de971761e91931. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:53.745592 17454 tablet_service.cc:1467] Processing CreateTablet for tablet 472d7827b3074236a81a3ede098f6c0a (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=ecb981650f8c425798b67a5194ad793c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:53.746001 17454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 472d7827b3074236a81a3ede098f6c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:53.746493 21253 tablet_bootstrap.cc:492] T 139304da92ab4f48b8a08fa24ed60c95 P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:53:53.748081 21253 tablet_bootstrap.cc:654] T 139304da92ab4f48b8a08fa24ed60c95 P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:53.751164 20478 tablet_service.cc:1467] Processing CreateTablet for tablet 139304da92ab4f48b8a08fa24ed60c95 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=ecb981650f8c425798b67a5194ad793c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:53.751607 20478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 139304da92ab4f48b8a08fa24ed60c95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:53.752377 20212 tablet_service.cc:1467] Processing CreateTablet for tablet 139304da92ab4f48b8a08fa24ed60c95 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=ecb981650f8c425798b67a5194ad793c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:53.752810 20212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 139304da92ab4f48b8a08fa24ed60c95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:53.757397 20213 tablet_service.cc:1467] Processing CreateTablet for tablet 472d7827b3074236a81a3ede098f6c0a (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=ecb981650f8c425798b67a5194ad793c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:53.757750 20213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 472d7827b3074236a81a3ede098f6c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:53.760551 21368 tablet_bootstrap.cc:492] T 139304da92ab4f48b8a08fa24ed60c95 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:53:53.762076 21368 tablet_bootstrap.cc:654] T 139304da92ab4f48b8a08fa24ed60c95 P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:53.766134 17326 catalog_manager.cc:5576] TS 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623) does not have the latest schema for tablet b7274d4a59be4949a42f218c3edf743d (table rebalancer_dynamic_table [id=5b70d9fa9fb44c838d736b49154a72f5]). Expected version 1 got 0
I20240502 01:53:53.770116 21469 tablet_bootstrap.cc:492] T 472d7827b3074236a81a3ede098f6c0a P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:53.771813 21469 tablet_bootstrap.cc:654] T 472d7827b3074236a81a3ede098f6c0a P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:53.785622 17605 raft_consensus.cc:1203] T 975f253ff03447ff96fe84169c05097e P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:53:53.793877 21368 tablet_bootstrap.cc:492] T 139304da92ab4f48b8a08fa24ed60c95 P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:53:53.794133 21368 ts_tablet_manager.cc:1367] T 139304da92ab4f48b8a08fa24ed60c95 P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.008s
I20240502 01:53:53.794226 21469 tablet_bootstrap.cc:492] T 472d7827b3074236a81a3ede098f6c0a P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:53.796430 21469 ts_tablet_manager.cc:1367] T 472d7827b3074236a81a3ede098f6c0a P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.005s
I20240502 01:53:53.797511 21554 tablet.cc:1722] T b7274d4a59be4949a42f218c3edf743d P 2f306549407b4bb281ccf2994035e4f1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:53:53.798204 21554 tablet.cc:1620] T b7274d4a59be4949a42f218c3edf743d P 2f306549407b4bb281ccf2994035e4f1: MemRowSet was empty: no flush needed.
I20240502 01:53:53.798602 17333 catalog_manager.cc:5576] TS 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623) does not have the latest schema for tablet b7274d4a59be4949a42f218c3edf743d (table rebalancer_dynamic_table [id=5b70d9fa9fb44c838d736b49154a72f5]). Expected version 1 got 0
I20240502 01:53:53.799882 21552 tablet_bootstrap.cc:492] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:53:53.799969 21368 raft_consensus.cc:348] T 139304da92ab4f48b8a08fa24ed60c95 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:53.800266 21368 raft_consensus.cc:374] T 139304da92ab4f48b8a08fa24ed60c95 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:53.800410 21368 raft_consensus.cc:729] T 139304da92ab4f48b8a08fa24ed60c95 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:53:53.801030 21553 tablet_bootstrap.cc:492] T 8eed067f639542d487de971761e91931 P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:53:53.800786 21368 consensus_queue.cc:260] T 139304da92ab4f48b8a08fa24ed60c95 P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:53.801585 21548 tablet.cc:1722] T b7274d4a59be4949a42f218c3edf743d P 4da690c0fa13407a85c0689ce8de9ade: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:53:53.802521 21556 tablet.cc:1722] T b7274d4a59be4949a42f218c3edf743d P f19cc6cb1e76460aa6711e49a5775091: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:53:53.803980 21556 tablet.cc:1620] T b7274d4a59be4949a42f218c3edf743d P f19cc6cb1e76460aa6711e49a5775091: MemRowSet was empty: no flush needed.
I20240502 01:53:53.804370 17326 catalog_manager.cc:5576] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321) does not have the latest schema for tablet b7274d4a59be4949a42f218c3edf743d (table rebalancer_dynamic_table [id=5b70d9fa9fb44c838d736b49154a72f5]). Expected version 1 got 0
I20240502 01:53:53.804369 21368 ts_tablet_manager.cc:1398] T 139304da92ab4f48b8a08fa24ed60c95 P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240502 01:53:53.805634 17333 catalog_manager.cc:5576] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411) does not have the latest schema for tablet b7274d4a59be4949a42f218c3edf743d (table rebalancer_dynamic_table [id=5b70d9fa9fb44c838d736b49154a72f5]). Expected version 1 got 0
I20240502 01:53:53.805255 21253 tablet_bootstrap.cc:492] T 139304da92ab4f48b8a08fa24ed60c95 P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:53:53.807009 21253 ts_tablet_manager.cc:1367] T 139304da92ab4f48b8a08fa24ed60c95 P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.061s user 0.000s sys 0.015s
I20240502 01:53:53.808674 21552 tablet_bootstrap.cc:654] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:53.802070 21469 raft_consensus.cc:348] T 472d7827b3074236a81a3ede098f6c0a P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:53.808905 21469 raft_consensus.cc:374] T 472d7827b3074236a81a3ede098f6c0a P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:53.809039 21469 raft_consensus.cc:729] T 472d7827b3074236a81a3ede098f6c0a P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:53.809345 21469 consensus_queue.cc:260] T 472d7827b3074236a81a3ede098f6c0a P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:53.809967 21469 ts_tablet_manager.cc:1398] T 472d7827b3074236a81a3ede098f6c0a P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240502 01:53:53.811146 21548 tablet.cc:1620] T b7274d4a59be4949a42f218c3edf743d P 4da690c0fa13407a85c0689ce8de9ade: MemRowSet was empty: no flush needed.
I20240502 01:53:53.811875 21553 tablet_bootstrap.cc:654] T 8eed067f639542d487de971761e91931 P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:53.812697 17334 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=5b70d9fa9fb44c838d736b49154a72f5] alter complete (version 1)
I20240502 01:53:53.814666 21253 raft_consensus.cc:348] T 139304da92ab4f48b8a08fa24ed60c95 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:53.814949 21253 raft_consensus.cc:374] T 139304da92ab4f48b8a08fa24ed60c95 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:53.815095 21253 raft_consensus.cc:729] T 139304da92ab4f48b8a08fa24ed60c95 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:53:53.815414 21253 consensus_queue.cc:260] T 139304da92ab4f48b8a08fa24ed60c95 P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:53.816378 21253 ts_tablet_manager.cc:1398] T 139304da92ab4f48b8a08fa24ed60c95 P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240502 01:53:53.820580 21253 tablet_bootstrap.cc:492] T 8eed067f639542d487de971761e91931 P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:53:53.822324 21253 tablet_bootstrap.cc:654] T 8eed067f639542d487de971761e91931 P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:53.836557 21553 tablet_bootstrap.cc:492] T 8eed067f639542d487de971761e91931 P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:53:53.836768 21553 ts_tablet_manager.cc:1367] T 8eed067f639542d487de971761e91931 P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.002s
I20240502 01:53:53.837723 21553 raft_consensus.cc:348] T 8eed067f639542d487de971761e91931 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:53.838395 21553 raft_consensus.cc:374] T 8eed067f639542d487de971761e91931 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:53.838864 21552 tablet_bootstrap.cc:492] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:53:53.839218 21552 ts_tablet_manager.cc:1367] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.046s user 0.004s sys 0.000s
I20240502 01:53:53.841564 21552 raft_consensus.cc:348] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:53.842150 21552 raft_consensus.cc:374] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:53.842351 21552 raft_consensus.cc:729] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:53:53.842682 21253 tablet_bootstrap.cc:492] T 8eed067f639542d487de971761e91931 P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:53:53.842862 21253 ts_tablet_manager.cc:1367] T 8eed067f639542d487de971761e91931 P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20240502 01:53:53.842691 21552 consensus_queue.cc:260] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:53.843075 21553 raft_consensus.cc:729] T 8eed067f639542d487de971761e91931 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:53:53.843341 21553 consensus_queue.cc:260] T 8eed067f639542d487de971761e91931 P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:53.843609 21253 raft_consensus.cc:348] T 8eed067f639542d487de971761e91931 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:53.843847 21253 raft_consensus.cc:374] T 8eed067f639542d487de971761e91931 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:53.843943 21253 raft_consensus.cc:729] T 8eed067f639542d487de971761e91931 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:53:53.843945 21553 ts_tablet_manager.cc:1398] T 8eed067f639542d487de971761e91931 P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240502 01:53:53.844195 21253 consensus_queue.cc:260] T 8eed067f639542d487de971761e91931 P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:53.844579 21552 ts_tablet_manager.cc:1398] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:53:53.844906 21552 tablet_bootstrap.cc:492] T 472d7827b3074236a81a3ede098f6c0a P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:53:53.845512 21253 ts_tablet_manager.cc:1398] T 8eed067f639542d487de971761e91931 P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:53:53.847047 21552 tablet_bootstrap.cc:654] T 472d7827b3074236a81a3ede098f6c0a P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:53.850764 20630 raft_consensus.cc:1203] T 1f01949cf70440b38e59a5020fd48c44 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:53:53.853605 21553 tablet_bootstrap.cc:492] T 472d7827b3074236a81a3ede098f6c0a P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:53:53.855523 21553 tablet_bootstrap.cc:654] T 472d7827b3074236a81a3ede098f6c0a P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:53.865545 21259 raft_consensus.cc:1050] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3: attempting to promote NON_VOTER f19cc6cb1e76460aa6711e49a5775091 to VOTER
I20240502 01:53:53.866412 21259 consensus_queue.cc:237] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } }
I20240502 01:53:53.869041 21553 tablet_bootstrap.cc:492] T 472d7827b3074236a81a3ede098f6c0a P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:53:53.869251 21553 ts_tablet_manager.cc:1367] T 472d7827b3074236a81a3ede098f6c0a P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.002s
I20240502 01:53:53.870297 21553 raft_consensus.cc:348] T 472d7827b3074236a81a3ede098f6c0a P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:53.870659 21553 raft_consensus.cc:374] T 472d7827b3074236a81a3ede098f6c0a P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:53.870769 21553 raft_consensus.cc:729] T 472d7827b3074236a81a3ede098f6c0a P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:53:53.871098 21553 consensus_queue.cc:260] T 472d7827b3074236a81a3ede098f6c0a P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:53.871681 21553 ts_tablet_manager.cc:1398] T 472d7827b3074236a81a3ede098f6c0a P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:53.871994 21553 tablet_bootstrap.cc:492] T 139304da92ab4f48b8a08fa24ed60c95 P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:53:53.872445 17735 raft_consensus.cc:1261] T 4bb42883b1a0475b863d008bd01fa70a P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:53.873368 20630 raft_consensus.cc:1261] T 4bb42883b1a0475b863d008bd01fa70a P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:53.873363 21265 consensus_queue.cc:1035] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:53.874084 21265 consensus_queue.cc:1035] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:53.874437 21552 tablet_bootstrap.cc:492] T 472d7827b3074236a81a3ede098f6c0a P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:53:53.874641 21552 ts_tablet_manager.cc:1367] T 472d7827b3074236a81a3ede098f6c0a P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
I20240502 01:53:53.875344 21553 tablet_bootstrap.cc:654] T 139304da92ab4f48b8a08fa24ed60c95 P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:53.875445 21552 raft_consensus.cc:348] T 472d7827b3074236a81a3ede098f6c0a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:53.875717 21552 raft_consensus.cc:374] T 472d7827b3074236a81a3ede098f6c0a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:53.875829 21552 raft_consensus.cc:729] T 472d7827b3074236a81a3ede098f6c0a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:53:53.874615 21513 tablet_bootstrap.cc:492] T 65bdab2c9fba48a6828a864278319cc0 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:53.876133 21552 consensus_queue.cc:260] T 472d7827b3074236a81a3ede098f6c0a P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:53.876713 21513 tablet_bootstrap.cc:492] T 65bdab2c9fba48a6828a864278319cc0 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap complete.
I20240502 01:53:53.877005 21513 ts_tablet_manager.cc:1367] T 65bdab2c9fba48a6828a864278319cc0 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.180s user 0.042s sys 0.007s
I20240502 01:53:53.877712 21552 ts_tablet_manager.cc:1398] T 472d7827b3074236a81a3ede098f6c0a P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:53:53.877862 21513 raft_consensus.cc:348] T 65bdab2c9fba48a6828a864278319cc0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } }
I20240502 01:53:53.878190 21513 raft_consensus.cc:729] T 65bdab2c9fba48a6828a864278319cc0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: LEARNER
I20240502 01:53:53.878415 21513 consensus_queue.cc:260] T 65bdab2c9fba48a6828a864278319cc0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } }
I20240502 01:53:53.879014 21513 ts_tablet_manager.cc:1398] T 65bdab2c9fba48a6828a864278319cc0 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:53.879657 17887 tablet_copy_service.cc:342] P bb31bceba76d466db24b9341c04900c3: Request end of tablet copy session 6a36c952c51542c0bf7ea3ec2857a9f9-65bdab2c9fba48a6828a864278319cc0 received from {username='slave'} at 127.11.42.70:41203
I20240502 01:53:53.879804 17887 tablet_copy_service.cc:433] P bb31bceba76d466db24b9341c04900c3: ending tablet copy session 6a36c952c51542c0bf7ea3ec2857a9f9-65bdab2c9fba48a6828a864278319cc0 on tablet 65bdab2c9fba48a6828a864278319cc0 with peer 6a36c952c51542c0bf7ea3ec2857a9f9
I20240502 01:53:53.884793 21553 tablet_bootstrap.cc:492] T 139304da92ab4f48b8a08fa24ed60c95 P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:53:53.884958 21553 ts_tablet_manager.cc:1367] T 139304da92ab4f48b8a08fa24ed60c95 P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20240502 01:53:53.885692 21553 raft_consensus.cc:348] T 139304da92ab4f48b8a08fa24ed60c95 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:53.886014 21553 raft_consensus.cc:374] T 139304da92ab4f48b8a08fa24ed60c95 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:53.886144 21553 raft_consensus.cc:729] T 139304da92ab4f48b8a08fa24ed60c95 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:53:53.886437 21553 consensus_queue.cc:260] T 139304da92ab4f48b8a08fa24ed60c95 P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:53.886972 21553 ts_tablet_manager.cc:1398] T 139304da92ab4f48b8a08fa24ed60c95 P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:53.891142 21265 raft_consensus.cc:2932] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:53.895501 20877 raft_consensus.cc:482] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:53.895347 17735 raft_consensus.cc:2932] T 4bb42883b1a0475b863d008bd01fa70a P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:53.895663 20877 raft_consensus.cc:504] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:53.896363 20877 leader_election.cc:288] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:53:53.896831 20630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eed067f639542d487de971761e91931" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091" is_pre_election: true
I20240502 01:53:53.896956 20233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eed067f639542d487de971761e91931" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:53:53.897179 20630 raft_consensus.cc:2454] T 8eed067f639542d487de971761e91931 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 0.
I20240502 01:53:53.897322 20233 raft_consensus.cc:2454] T 8eed067f639542d487de971761e91931 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 0.
I20240502 01:53:53.897596 20302 leader_election.cc:302] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:53:53.898093 20877 raft_consensus.cc:2781] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:53.898201 20877 raft_consensus.cc:482] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:53.898283 20877 raft_consensus.cc:3037] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:53.899485 17605 raft_consensus.cc:1261] T 4bb42883b1a0475b863d008bd01fa70a P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:53.900069 21259 consensus_queue.cc:1035] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:53.900959 20877 raft_consensus.cc:504] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:53.902187 20233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eed067f639542d487de971761e91931" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:53:53.902403 20233 raft_consensus.cc:3037] T 8eed067f639542d487de971761e91931 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:53.905495 20233 raft_consensus.cc:2454] T 8eed067f639542d487de971761e91931 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 1.
I20240502 01:53:53.906354 20303 leader_election.cc:302] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:53:53.906672 21009 raft_consensus.cc:2781] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:53.906816 21009 raft_consensus.cc:686] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Becoming Leader. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Running, Role: LEADER
I20240502 01:53:53.907111 21009 consensus_queue.cc:237] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:53.912896 17605 raft_consensus.cc:2932] T 4bb42883b1a0475b863d008bd01fa70a P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:53.915784 17333 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:53:53.923605 20877 leader_election.cc:288] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Requested vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:53:53.924050 20629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eed067f639542d487de971761e91931" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091"
I20240502 01:53:53.924299 20629 raft_consensus.cc:3037] T 8eed067f639542d487de971761e91931 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:53.926069 21548 tablet.cc:1722] T b7274d4a59be4949a42f218c3edf743d P 4da690c0fa13407a85c0689ce8de9ade: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:53:53.926604 21548 tablet.cc:1620] T b7274d4a59be4949a42f218c3edf743d P 4da690c0fa13407a85c0689ce8de9ade: MemRowSet was empty: no flush needed.
I20240502 01:53:53.927101 20629 raft_consensus.cc:2454] T 8eed067f639542d487de971761e91931 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 1.
I20240502 01:53:53.927839 21556 tablet.cc:1722] T b7274d4a59be4949a42f218c3edf743d P f19cc6cb1e76460aa6711e49a5775091: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:53:53.928280 21556 tablet.cc:1620] T b7274d4a59be4949a42f218c3edf743d P f19cc6cb1e76460aa6711e49a5775091: MemRowSet was empty: no flush needed.
I20240502 01:53:53.931608 21554 tablet.cc:1722] T b7274d4a59be4949a42f218c3edf743d P 2f306549407b4bb281ccf2994035e4f1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:53:53.932052 21554 tablet.cc:1620] T b7274d4a59be4949a42f218c3edf743d P 2f306549407b4bb281ccf2994035e4f1: MemRowSet was empty: no flush needed.
I20240502 01:53:53.936668 20629 raft_consensus.cc:2932] T 4bb42883b1a0475b863d008bd01fa70a P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:53.939587 17334 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=5b70d9fa9fb44c838d736b49154a72f5] alter complete (version 2)
I20240502 01:53:53.944097 17326 catalog_manager.cc:5484] T 4bb42883b1a0475b863d008bd01fa70a P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: config changed from index 4 to 5, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:53.950335 17587 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:53:53.951182 20478 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:53:54.001998 17333 catalog_manager.cc:5484] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70). New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:54.012048 21580 raft_consensus.cc:1050] T 62887e332443457f8a5b41a982354dc4 P fdc2ac372492454c85387589a91f63af: attempting to promote NON_VOTER bb31bceba76d466db24b9341c04900c3 to VOTER
I20240502 01:53:54.011873 17864 consensus_queue.cc:237] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } }
I20240502 01:53:54.013317 21580 consensus_queue.cc:237] T 62887e332443457f8a5b41a982354dc4 P fdc2ac372492454c85387589a91f63af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } }
I20240502 01:53:54.022745 20629 raft_consensus.cc:1261] T 4bb42883b1a0475b863d008bd01fa70a P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:53:54.023394 21571 consensus_queue.cc:1035] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:53:54.023559 17605 raft_consensus.cc:1261] T 4bb42883b1a0475b863d008bd01fa70a P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:53:54.024125 21571 consensus_queue.cc:1035] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:53:54.026620 21571 raft_consensus.cc:2932] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:54.031525 20629 raft_consensus.cc:2932] T 4bb42883b1a0475b863d008bd01fa70a P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:54.037127 17326 catalog_manager.cc:5484] T 4bb42883b1a0475b863d008bd01fa70a P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: config changed from index 5 to 6, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:54.038734 17322 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4bb42883b1a0475b863d008bd01fa70a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:54.040510 17864 raft_consensus.cc:1261] T 62887e332443457f8a5b41a982354dc4 P bb31bceba76d466db24b9341c04900c3 [term 1 LEARNER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:53:54.042300 17716 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bb42883b1a0475b863d008bd01fa70a with delete_type TABLET_DATA_TOMBSTONED (TS dfa1152987644bd781b46c7e42a7a685 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52198
I20240502 01:53:54.042337 21487 consensus_queue.cc:1035] T 62887e332443457f8a5b41a982354dc4 P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:53:54.042577 17333 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:53:54.043506 17605 raft_consensus.cc:2932] T 4bb42883b1a0475b863d008bd01fa70a P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:54.043740 20629 raft_consensus.cc:1261] T 62887e332443457f8a5b41a982354dc4 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:53:54.047385 21584 consensus_queue.cc:1035] T 62887e332443457f8a5b41a982354dc4 P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:53:54.050602 21548 tablet.cc:1722] T b7274d4a59be4949a42f218c3edf743d P 4da690c0fa13407a85c0689ce8de9ade: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:53:54.051188 21556 tablet.cc:1722] T b7274d4a59be4949a42f218c3edf743d P f19cc6cb1e76460aa6711e49a5775091: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:53:54.053966 21554 tablet.cc:1722] T b7274d4a59be4949a42f218c3edf743d P 2f306549407b4bb281ccf2994035e4f1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:53:54.062026 17326 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table_new_name [id=5b70d9fa9fb44c838d736b49154a72f5] alter complete (version 3)
I20240502 01:53:54.065320 21216 raft_consensus.cc:482] T 139304da92ab4f48b8a08fa24ed60c95 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:54.065479 21216 raft_consensus.cc:504] T 139304da92ab4f48b8a08fa24ed60c95 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:54.065814 20497 raft_consensus.cc:1261] T 62887e332443457f8a5b41a982354dc4 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:53:54.066176 21216 leader_election.cc:288] T 139304da92ab4f48b8a08fa24ed60c95 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:53:54.067107 20630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "139304da92ab4f48b8a08fa24ed60c95" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091" is_pre_election: true
I20240502 01:53:54.067742 20630 raft_consensus.cc:2454] T 139304da92ab4f48b8a08fa24ed60c95 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 0.
I20240502 01:53:54.068408 20433 leader_election.cc:302] T 139304da92ab4f48b8a08fa24ed60c95 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:53:54.068552 21487 raft_consensus.cc:482] T 139304da92ab4f48b8a08fa24ed60c95 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:54.068782 21216 raft_consensus.cc:2781] T 139304da92ab4f48b8a08fa24ed60c95 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:54.068704 21487 raft_consensus.cc:504] T 139304da92ab4f48b8a08fa24ed60c95 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:54.068893 21216 raft_consensus.cc:482] T 139304da92ab4f48b8a08fa24ed60c95 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:54.068969 21216 raft_consensus.cc:3037] T 139304da92ab4f48b8a08fa24ed60c95 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:54.069790 20629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "139304da92ab4f48b8a08fa24ed60c95" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091" is_pre_election: true
I20240502 01:53:54.069969 20629 raft_consensus.cc:2454] T 139304da92ab4f48b8a08fa24ed60c95 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 0.
I20240502 01:53:54.070331 20167 leader_election.cc:302] T 139304da92ab4f48b8a08fa24ed60c95 P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f19cc6cb1e76460aa6711e49a5775091, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:53:54.070787 21487 leader_election.cc:288] T 139304da92ab4f48b8a08fa24ed60c95 P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:53:54.070968 21487 raft_consensus.cc:2781] T 139304da92ab4f48b8a08fa24ed60c95 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:54.071039 21487 raft_consensus.cc:482] T 139304da92ab4f48b8a08fa24ed60c95 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:54.071125 21487 raft_consensus.cc:3037] T 139304da92ab4f48b8a08fa24ed60c95 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:54.071509 21216 raft_consensus.cc:504] T 139304da92ab4f48b8a08fa24ed60c95 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:54.072841 21216 leader_election.cc:288] T 139304da92ab4f48b8a08fa24ed60c95 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 election: Requested vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:53:54.072942 20629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "139304da92ab4f48b8a08fa24ed60c95" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091"
I20240502 01:53:54.073208 20629 raft_consensus.cc:3037] T 139304da92ab4f48b8a08fa24ed60c95 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:54.076144 20629 raft_consensus.cc:2454] T 139304da92ab4f48b8a08fa24ed60c95 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 1.
I20240502 01:53:54.076637 20433 leader_election.cc:302] T 139304da92ab4f48b8a08fa24ed60c95 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:53:54.076998 21216 raft_consensus.cc:2781] T 139304da92ab4f48b8a08fa24ed60c95 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:54.078534 21216 raft_consensus.cc:686] T 139304da92ab4f48b8a08fa24ed60c95 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Running, Role: LEADER
I20240502 01:53:54.078866 21216 consensus_queue.cc:237] T 139304da92ab4f48b8a08fa24ed60c95 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:54.081761 21591 tablet_replica.cc:330] T 4bb42883b1a0475b863d008bd01fa70a P dfa1152987644bd781b46c7e42a7a685: stopping tablet replica
I20240502 01:53:54.082136 20497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "139304da92ab4f48b8a08fa24ed60c95" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
I20240502 01:53:54.082814 21591 raft_consensus.cc:2229] T 4bb42883b1a0475b863d008bd01fa70a P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:54.083273 21591 raft_consensus.cc:2258] T 4bb42883b1a0475b863d008bd01fa70a P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:54.083709 20233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "139304da92ab4f48b8a08fa24ed60c95" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:53:54.084407 20231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "139304da92ab4f48b8a08fa24ed60c95" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:53:54.085388 21584 consensus_queue.cc:1035] T 62887e332443457f8a5b41a982354dc4 P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:53:54.086033 21584 raft_consensus.cc:2932] T 62887e332443457f8a5b41a982354dc4 P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:53:54.087883 17454 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:53:54.088413 21591 ts_tablet_manager.cc:1875] T 4bb42883b1a0475b863d008bd01fa70a P dfa1152987644bd781b46c7e42a7a685: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:54.089075 21487 raft_consensus.cc:504] T 139304da92ab4f48b8a08fa24ed60c95 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:54.089546 20233 raft_consensus.cc:2379] T 139304da92ab4f48b8a08fa24ed60c95 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2f306549407b4bb281ccf2994035e4f1 in current term 1: Already voted for candidate fdc2ac372492454c85387589a91f63af in this term.
I20240502 01:53:54.091001 20629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "139304da92ab4f48b8a08fa24ed60c95" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091"
I20240502 01:53:54.091405 20629 raft_consensus.cc:2379] T 139304da92ab4f48b8a08fa24ed60c95 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate fdc2ac372492454c85387589a91f63af in current term 1: Already voted for candidate 2f306549407b4bb281ccf2994035e4f1 in this term.
I20240502 01:53:54.094368 17334 catalog_manager.cc:5484] T 139304da92ab4f48b8a08fa24ed60c95 P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71). New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:54.098721 21487 leader_election.cc:288] T 139304da92ab4f48b8a08fa24ed60c95 P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 election: Requested vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:53:54.099467 17864 raft_consensus.cc:2932] T 62887e332443457f8a5b41a982354dc4 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:53:54.100418 20497 raft_consensus.cc:1203] T ebb9bca9f1bd41b0857939909b1110e4 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:53:54.107093 17326 catalog_manager.cc:5484] T 62887e332443457f8a5b41a982354dc4 P bb31bceba76d466db24b9341c04900c3 reported cstate change: config changed from index 2 to 3, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fdc2ac372492454c85387589a91f63af" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:53:54.109464 20497 raft_consensus.cc:2932] T 62887e332443457f8a5b41a982354dc4 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:53:54.113226 21591 ts_tablet_manager.cc:1888] T 4bb42883b1a0475b863d008bd01fa70a P dfa1152987644bd781b46c7e42a7a685: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:53:54.113368 21591 log.cc:1196] T 4bb42883b1a0475b863d008bd01fa70a P dfa1152987644bd781b46c7e42a7a685: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/wals/4bb42883b1a0475b863d008bd01fa70a
I20240502 01:53:54.114152 17321 catalog_manager.cc:4830] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): tablet 4bb42883b1a0475b863d008bd01fa70a (table rebalance_test_table_17 [id=c6474d0c5256497eb6fa5844f3530b62]) successfully deleted
I20240502 01:53:54.118250 20497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "139304da92ab4f48b8a08fa24ed60c95" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
I20240502 01:53:54.118922 20168 leader_election.cc:302] T 139304da92ab4f48b8a08fa24ed60c95 P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdc2ac372492454c85387589a91f63af; no voters: 2f306549407b4bb281ccf2994035e4f1, f19cc6cb1e76460aa6711e49a5775091
I20240502 01:53:54.119588 21580 raft_consensus.cc:2726] T 139304da92ab4f48b8a08fa24ed60c95 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 01:53:54.119979 20347 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:53:54.122570 20629 raft_consensus.cc:2932] T 62887e332443457f8a5b41a982354dc4 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:53:54.133952 20367 raft_consensus.cc:1203] T 65bdab2c9fba48a6828a864278319cc0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:53:54.137542 20233 consensus_queue.cc:237] T 62887e332443457f8a5b41a982354dc4 P fdc2ac372492454c85387589a91f63af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } }
I20240502 01:53:54.139564 20629 raft_consensus.cc:1261] T 62887e332443457f8a5b41a982354dc4 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:54.140201 21584 consensus_queue.cc:1035] T 62887e332443457f8a5b41a982354dc4 P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:54.140638 21543 raft_consensus.cc:1050] T 52ac2775e02844d195677035796c3847 P 3fceccefba274b1dbbd8216af8b0bc29: attempting to promote NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 to VOTER
I20240502 01:53:54.141786 21543 consensus_queue.cc:237] T 52ac2775e02844d195677035796c3847 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } }
I20240502 01:53:54.143688 17864 raft_consensus.cc:1261] T 62887e332443457f8a5b41a982354dc4 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:54.145907 21584 consensus_queue.cc:1035] T 62887e332443457f8a5b41a982354dc4 P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:54.146605 20868 raft_consensus.cc:1050] T 6aacf7c4f19546f581c9a54ad171916b P 3fceccefba274b1dbbd8216af8b0bc29: attempting to promote NON_VOTER 2f306549407b4bb281ccf2994035e4f1 to VOTER
I20240502 01:53:54.147405 20868 consensus_queue.cc:237] T 6aacf7c4f19546f581c9a54ad171916b P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } }
I20240502 01:53:54.148509 21580 raft_consensus.cc:2932] T 62887e332443457f8a5b41a982354dc4 P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:53:54.152527 21544 raft_consensus.cc:1050] T 9642b9b37a784cf881bfc4971d8c9cb6 P 4da690c0fa13407a85c0689ce8de9ade: attempting to promote NON_VOTER f19cc6cb1e76460aa6711e49a5775091 to VOTER
I20240502 01:53:54.153474 21544 consensus_queue.cc:237] T 9642b9b37a784cf881bfc4971d8c9cb6 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } }
I20240502 01:53:54.155203 17322 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 62887e332443457f8a5b41a982354dc4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:54.156229 17864 raft_consensus.cc:2932] T 62887e332443457f8a5b41a982354dc4 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:53:54.164122 17334 catalog_manager.cc:5484] T 62887e332443457f8a5b41a982354dc4 P fdc2ac372492454c85387589a91f63af reported cstate change: config changed from index 3 to 4, VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) evicted. New cstate: current_term: 1 leader_uuid: "fdc2ac372492454c85387589a91f63af" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:54.165674 17326 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:53:54.165872 17326 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:53:54.169292 17326 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet b7274d4a59be4949a42f218c3edf743d
I20240502 01:53:54.170545 17454 tablet_service.cc:1514] Processing DeleteTablet for tablet b7274d4a59be4949a42f218c3edf743d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:54 UTC) from {username='slave'} at 127.0.0.1:54520
I20240502 01:53:54.174161 20478 tablet_service.cc:1514] Processing DeleteTablet for tablet b7274d4a59be4949a42f218c3edf743d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:54 UTC) from {username='slave'} at 127.0.0.1:35018
I20240502 01:53:54.184358 20476 tablet_service.cc:1514] Processing DeleteTablet for tablet 62887e332443457f8a5b41a982354dc4 with delete_type TABLET_DATA_TOMBSTONED (TS 2f306549407b4bb281ccf2994035e4f1 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35018
I20240502 01:53:54.189149 21608 tablet_replica.cc:330] T b7274d4a59be4949a42f218c3edf743d P 4da690c0fa13407a85c0689ce8de9ade: stopping tablet replica
I20240502 01:53:54.190167 20497 raft_consensus.cc:1261] T 6aacf7c4f19546f581c9a54ad171916b P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:54.190898 21329 consensus_queue.cc:1035] T 6aacf7c4f19546f581c9a54ad171916b P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:54.190881 20629 raft_consensus.cc:2932] T 62887e332443457f8a5b41a982354dc4 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:53:54.200384 20367 raft_consensus.cc:1261] T 52ac2775e02844d195677035796c3847 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:54.201054 21329 consensus_queue.cc:1035] T 52ac2775e02844d195677035796c3847 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:54.208346 20630 raft_consensus.cc:1261] T 9642b9b37a784cf881bfc4971d8c9cb6 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:54.214128 21609 tablet_replica.cc:330] T b7274d4a59be4949a42f218c3edf743d P 2f306549407b4bb281ccf2994035e4f1: stopping tablet replica
I20240502 01:53:54.214447 21609 raft_consensus.cc:2229] T b7274d4a59be4949a42f218c3edf743d P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:54.215565 21609 raft_consensus.cc:2258] T b7274d4a59be4949a42f218c3edf743d P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:54.216463 17846 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:53:54.216463 17864 raft_consensus.cc:1261] T 52ac2775e02844d195677035796c3847 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:54.217073 21609 ts_tablet_manager.cc:1875] T b7274d4a59be4949a42f218c3edf743d P 2f306549407b4bb281ccf2994035e4f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:54.217200 17735 raft_consensus.cc:1261] T 6aacf7c4f19546f581c9a54ad171916b P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:54.218058 21329 consensus_queue.cc:1035] T 6aacf7c4f19546f581c9a54ad171916b P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:54.218361 17865 raft_consensus.cc:1261] T 6aacf7c4f19546f581c9a54ad171916b P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:54.218655 21329 consensus_queue.cc:1035] T 52ac2775e02844d195677035796c3847 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:54.219906 21329 consensus_queue.cc:1035] T 6aacf7c4f19546f581c9a54ad171916b P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20240502 01:53:54.229179 21609 ts_tablet_manager.cc:1888] T b7274d4a59be4949a42f218c3edf743d P 2f306549407b4bb281ccf2994035e4f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:53:54.229389 21609 log.cc:1196] T b7274d4a59be4949a42f218c3edf743d P 2f306549407b4bb281ccf2994035e4f1: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal/wals/b7274d4a59be4949a42f218c3edf743d
I20240502 01:53:54.229893 21609 ts_tablet_manager.cc:1909] T b7274d4a59be4949a42f218c3edf743d P 2f306549407b4bb281ccf2994035e4f1: Deleting consensus metadata
I20240502 01:53:54.230998 21609 tablet_replica.cc:330] T 62887e332443457f8a5b41a982354dc4 P 2f306549407b4bb281ccf2994035e4f1: stopping tablet replica
I20240502 01:53:54.231225 21609 raft_consensus.cc:2229] T 62887e332443457f8a5b41a982354dc4 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:54.231487 21609 raft_consensus.cc:2258] T 62887e332443457f8a5b41a982354dc4 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:54.232362 21609 ts_tablet_manager.cc:1875] T 62887e332443457f8a5b41a982354dc4 P 2f306549407b4bb281ccf2994035e4f1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:54.232527 17322 catalog_manager.cc:4830] TS 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): tablet b7274d4a59be4949a42f218c3edf743d (table rebalancer_dynamic_table_new_name [id=5b70d9fa9fb44c838d736b49154a72f5]) successfully deleted
I20240502 01:53:54.238694 21609 ts_tablet_manager.cc:1888] T 62887e332443457f8a5b41a982354dc4 P 2f306549407b4bb281ccf2994035e4f1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:53:54.238891 21609 log.cc:1196] T 62887e332443457f8a5b41a982354dc4 P 2f306549407b4bb281ccf2994035e4f1: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal/wals/62887e332443457f8a5b41a982354dc4
I20240502 01:53:54.245867 17322 catalog_manager.cc:4830] TS 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): tablet 62887e332443457f8a5b41a982354dc4 (table rebalancer_extra_table_6 [id=3e245c8534e84c55922b7cd77670a4a3]) successfully deleted
I20240502 01:53:54.246543 21579 raft_consensus.cc:1050] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9: attempting to promote NON_VOTER bb31bceba76d466db24b9341c04900c3 to VOTER
I20240502 01:53:54.247397 21579 consensus_queue.cc:237] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } }
I20240502 01:53:54.250164 21430 raft_consensus.cc:1050] T 975f253ff03447ff96fe84169c05097e P f19cc6cb1e76460aa6711e49a5775091: attempting to promote NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 to VOTER
I20240502 01:53:54.251096 21430 consensus_queue.cc:237] T 975f253ff03447ff96fe84169c05097e P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } }
I20240502 01:53:54.209455 21363 consensus_queue.cc:1035] T 9642b9b37a784cf881bfc4971d8c9cb6 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:54.212674 17475 raft_consensus.cc:1261] T 52ac2775e02844d195677035796c3847 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:54.263909 21608 raft_consensus.cc:2229] T b7274d4a59be4949a42f218c3edf743d P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:54.264338 21608 raft_consensus.cc:2258] T b7274d4a59be4949a42f218c3edf743d P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:54.264752 17736 raft_consensus.cc:1261] T 9642b9b37a784cf881bfc4971d8c9cb6 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:54.265412 21608 ts_tablet_manager.cc:1875] T b7274d4a59be4949a42f218c3edf743d P 4da690c0fa13407a85c0689ce8de9ade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:54.266242 17605 raft_consensus.cc:1261] T 9642b9b37a784cf881bfc4971d8c9cb6 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:54.265888 21543 consensus_queue.cc:1035] T 52ac2775e02844d195677035796c3847 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240502 01:53:54.273489 17334 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:53:54.274739 20608 tablet_service.cc:1514] Processing DeleteTablet for tablet b7274d4a59be4949a42f218c3edf743d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:54 UTC) from {username='slave'} at 127.0.0.1:33698
I20240502 01:53:54.290853 21009 raft_consensus.cc:482] T 472d7827b3074236a81a3ede098f6c0a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:54.291059 21009 raft_consensus.cc:504] T 472d7827b3074236a81a3ede098f6c0a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:54.291795 21009 leader_election.cc:288] T 472d7827b3074236a81a3ede098f6c0a P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:54.292608 20233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472d7827b3074236a81a3ede098f6c0a" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:53:54.292889 20233 raft_consensus.cc:2454] T 472d7827b3074236a81a3ede098f6c0a P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 0.
I20240502 01:53:54.293421 20303 leader_election.cc:302] T 472d7827b3074236a81a3ede098f6c0a P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:53:54.294128 21009 raft_consensus.cc:2781] T 472d7827b3074236a81a3ede098f6c0a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:54.294225 21009 raft_consensus.cc:482] T 472d7827b3074236a81a3ede098f6c0a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:54.294317 21009 raft_consensus.cc:3037] T 472d7827b3074236a81a3ede098f6c0a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:54.294932 20610 tablet_service.cc:1467] Processing CreateTablet for tablet dc5ac6b3423248c19209032b774885ca (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4bc119cbe600465a9c4d50d5f21a0e9d]), partition=
I20240502 01:53:54.295049 20211 tablet_service.cc:1467] Processing CreateTablet for tablet dc5ac6b3423248c19209032b774885ca (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4bc119cbe600465a9c4d50d5f21a0e9d]), partition=
I20240502 01:53:54.295328 20610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc5ac6b3423248c19209032b774885ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:54.295405 20211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc5ac6b3423248c19209032b774885ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:54.296923 21009 raft_consensus.cc:504] T 472d7827b3074236a81a3ede098f6c0a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:54.298944 20233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472d7827b3074236a81a3ede098f6c0a" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:53:54.299167 20233 raft_consensus.cc:3037] T 472d7827b3074236a81a3ede098f6c0a P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:54.301611 21553 tablet_bootstrap.cc:492] T dc5ac6b3423248c19209032b774885ca P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:53:54.302376 20233 raft_consensus.cc:2454] T 472d7827b3074236a81a3ede098f6c0a P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 1.
I20240502 01:53:54.302836 20303 leader_election.cc:302] T 472d7827b3074236a81a3ede098f6c0a P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:53:54.303121 21553 tablet_bootstrap.cc:654] T dc5ac6b3423248c19209032b774885ca P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:54.303133 21582 raft_consensus.cc:2781] T 472d7827b3074236a81a3ede098f6c0a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:54.303252 21582 raft_consensus.cc:686] T 472d7827b3074236a81a3ede098f6c0a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Becoming Leader. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Running, Role: LEADER
I20240502 01:53:54.303540 21582 consensus_queue.cc:237] T 472d7827b3074236a81a3ede098f6c0a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:54.304006 21253 tablet_bootstrap.cc:492] T dc5ac6b3423248c19209032b774885ca P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:53:54.305357 21253 tablet_bootstrap.cc:654] T dc5ac6b3423248c19209032b774885ca P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:54.305433 21009 leader_election.cc:288] T 472d7827b3074236a81a3ede098f6c0a P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Requested vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:54.265370 21499 consensus_queue.cc:1035] T 9642b9b37a784cf881bfc4971d8c9cb6 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:54.306221 21499 raft_consensus.cc:482] T 472d7827b3074236a81a3ede098f6c0a P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:54.306398 21499 raft_consensus.cc:504] T 472d7827b3074236a81a3ede098f6c0a P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:54.307085 17474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472d7827b3074236a81a3ede098f6c0a" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:53:54.307329 17474 raft_consensus.cc:3037] T 472d7827b3074236a81a3ede098f6c0a P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:54.307427 21363 consensus_queue.cc:1035] T 9642b9b37a784cf881bfc4971d8c9cb6 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:54.297726 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472d7827b3074236a81a3ede098f6c0a" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:54.301954 17454 tablet_service.cc:1467] Processing CreateTablet for tablet dc5ac6b3423248c19209032b774885ca (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4bc119cbe600465a9c4d50d5f21a0e9d]), partition=
I20240502 01:53:54.310319 17474 raft_consensus.cc:2454] T 472d7827b3074236a81a3ede098f6c0a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 1.
I20240502 01:53:54.307340 21499 leader_election.cc:288] T 472d7827b3074236a81a3ede098f6c0a P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:53:54.315021 21553 tablet_bootstrap.cc:492] T dc5ac6b3423248c19209032b774885ca P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:53:54.315182 21553 ts_tablet_manager.cc:1367] T dc5ac6b3423248c19209032b774885ca P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.001s
I20240502 01:53:54.315176 17334 catalog_manager.cc:5484] T 472d7827b3074236a81a3ede098f6c0a P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70). New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:54.315925 21553 raft_consensus.cc:348] T dc5ac6b3423248c19209032b774885ca P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:54.316123 21553 raft_consensus.cc:374] T dc5ac6b3423248c19209032b774885ca P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:54.316195 21553 raft_consensus.cc:729] T dc5ac6b3423248c19209032b774885ca P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:53:54.316618 21553 consensus_queue.cc:260] T dc5ac6b3423248c19209032b774885ca P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:54.318527 21553 ts_tablet_manager.cc:1398] T dc5ac6b3423248c19209032b774885ca P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:53:54.320238 21253 tablet_bootstrap.cc:492] T dc5ac6b3423248c19209032b774885ca P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:53:54.320452 21253 ts_tablet_manager.cc:1367] T dc5ac6b3423248c19209032b774885ca P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.004s
I20240502 01:53:54.321273 21253 raft_consensus.cc:348] T dc5ac6b3423248c19209032b774885ca P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:54.321550 21253 raft_consensus.cc:374] T dc5ac6b3423248c19209032b774885ca P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:54.321681 21253 raft_consensus.cc:729] T dc5ac6b3423248c19209032b774885ca P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:53:54.321998 21253 consensus_queue.cc:260] T dc5ac6b3423248c19209032b774885ca P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:54.321851 20868 raft_consensus.cc:2932] T 52ac2775e02844d195677035796c3847 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:53:54.322726 21253 ts_tablet_manager.cc:1398] T dc5ac6b3423248c19209032b774885ca P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:54.327459 17474 raft_consensus.cc:2932] T 52ac2775e02844d195677035796c3847 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:53:54.337576 17334 catalog_manager.cc:5484] T 52ac2775e02844d195677035796c3847 P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: config changed from index 4 to 5, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
W20240502 01:53:54.339008 17411 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.42.72:42411, user_credentials={real_user=slave}} blocked reactor thread for 51283us
I20240502 01:53:54.343245 17866 raft_consensus.cc:1261] T e42126e16cda4d908b00e2e4a6807d6b P bb31bceba76d466db24b9341c04900c3 [term 1 LEARNER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:53:54.343462 17474 raft_consensus.cc:1261] T e42126e16cda4d908b00e2e4a6807d6b P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:53:54.343873 20233 raft_consensus.cc:1261] T e42126e16cda4d908b00e2e4a6807d6b P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:53:54.344014 20813 consensus_queue.cc:1035] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:53:54.345456 20813 consensus_queue.cc:1035] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:53:54.347679 20813 consensus_queue.cc:1035] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:53:54.348294 21329 raft_consensus.cc:2932] T 6aacf7c4f19546f581c9a54ad171916b P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:53:54.349557 21621 tablet_replica.cc:330] T b7274d4a59be4949a42f218c3edf743d P f19cc6cb1e76460aa6711e49a5775091: stopping tablet replica
I20240502 01:53:54.352356 17604 raft_consensus.cc:1261] T 975f253ff03447ff96fe84169c05097e P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEARNER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:53:54.352901 20367 raft_consensus.cc:2932] T 52ac2775e02844d195677035796c3847 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:53:54.354125 20610 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:53:54.354329 21429 consensus_queue.cc:1035] T 975f253ff03447ff96fe84169c05097e P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:53:54.356346 21621 raft_consensus.cc:2229] T b7274d4a59be4949a42f218c3edf743d P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:54.357030 21621 raft_consensus.cc:2258] T b7274d4a59be4949a42f218c3edf743d P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:54.369423 17454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc5ac6b3423248c19209032b774885ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:54.374151 17474 raft_consensus.cc:1261] T 975f253ff03447ff96fe84169c05097e P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:53:54.374783 21429 consensus_queue.cc:1035] T 975f253ff03447ff96fe84169c05097e P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:53:54.379144 21009 raft_consensus.cc:2932] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:53:54.379882 20233 raft_consensus.cc:2932] T e42126e16cda4d908b00e2e4a6807d6b P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:53:54.384197 17475 raft_consensus.cc:2932] T e42126e16cda4d908b00e2e4a6807d6b P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:53:54.387852 21637 tablet_bootstrap.cc:492] T dc5ac6b3423248c19209032b774885ca P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:54.392392 17334 catalog_manager.cc:5484] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: config changed from index 2 to 3, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:54.397140 21009 consensus_queue.cc:1035] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:54.406780 20233 raft_consensus.cc:1261] T 975f253ff03447ff96fe84169c05097e P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:53:54.408497 21429 consensus_queue.cc:1035] T 975f253ff03447ff96fe84169c05097e P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:53:54.407423 17865 raft_consensus.cc:2932] T e42126e16cda4d908b00e2e4a6807d6b P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:53:54.411342 21637 tablet_bootstrap.cc:654] T dc5ac6b3423248c19209032b774885ca P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:54.422956 20367 consensus_queue.cc:237] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } }
I20240502 01:53:54.423888 20232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472d7827b3074236a81a3ede098f6c0a" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:53:54.424149 20232 raft_consensus.cc:2379] T 472d7827b3074236a81a3ede098f6c0a P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4da690c0fa13407a85c0689ce8de9ade in current term 1: Already voted for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in this term.
I20240502 01:53:54.426401 17865 raft_consensus.cc:1261] T e42126e16cda4d908b00e2e4a6807d6b P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:54.427290 21608 ts_tablet_manager.cc:1888] T b7274d4a59be4949a42f218c3edf743d P 4da690c0fa13407a85c0689ce8de9ade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:53:54.427498 20366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472d7827b3074236a81a3ede098f6c0a" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:53:54.428403 21009 consensus_queue.cc:1035] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:54.428825 21621 ts_tablet_manager.cc:1875] T b7274d4a59be4949a42f218c3edf743d P f19cc6cb1e76460aa6711e49a5775091: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:54.429533 21608 log.cc:1196] T b7274d4a59be4949a42f218c3edf743d P 4da690c0fa13407a85c0689ce8de9ade: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/b7274d4a59be4949a42f218c3edf743d
I20240502 01:53:54.430619 20232 raft_consensus.cc:1261] T e42126e16cda4d908b00e2e4a6807d6b P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:54.431149 21579 consensus_queue.cc:1035] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:54.433185 21579 raft_consensus.cc:2932] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:53:54.433499 21608 ts_tablet_manager.cc:1909] T b7274d4a59be4949a42f218c3edf743d P 4da690c0fa13407a85c0689ce8de9ade: Deleting consensus metadata
I20240502 01:53:54.435770 17321 catalog_manager.cc:4830] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): tablet b7274d4a59be4949a42f218c3edf743d (table rebalancer_dynamic_table_new_name [id=5b70d9fa9fb44c838d736b49154a72f5]) successfully deleted
I20240502 01:53:54.437214 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e42126e16cda4d908b00e2e4a6807d6b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:54.438657 20232 raft_consensus.cc:2932] T e42126e16cda4d908b00e2e4a6807d6b P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:53:54.443156 21637 tablet_bootstrap.cc:492] T dc5ac6b3423248c19209032b774885ca P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:54.443878 17865 raft_consensus.cc:2932] T e42126e16cda4d908b00e2e4a6807d6b P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:53:54.443346 21637 ts_tablet_manager.cc:1367] T dc5ac6b3423248c19209032b774885ca P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.056s user 0.002s sys 0.003s
W20240502 01:53:54.447023 17543 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.42.68:44083, user_credentials={real_user=slave}} blocked reactor thread for 82065.2us
I20240502 01:53:54.447760 21612 raft_consensus.cc:2932] T 9642b9b37a784cf881bfc4971d8c9cb6 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:54.449483 21637 raft_consensus.cc:348] T dc5ac6b3423248c19209032b774885ca P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:54.450960 17326 catalog_manager.cc:5484] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: config changed from index 3 to 4, VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) evicted. New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:54.451944 21637 raft_consensus.cc:374] T dc5ac6b3423248c19209032b774885ca P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:54.454676 21637 raft_consensus.cc:729] T dc5ac6b3423248c19209032b774885ca P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:54.454404 17335 catalog_manager.cc:5484] T 9642b9b37a784cf881bfc4971d8c9cb6 P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: config changed from index 4 to 5, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:54.451009 17411 leader_election.cc:302] T 472d7827b3074236a81a3ede098f6c0a P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade; no voters: 6a36c952c51542c0bf7ea3ec2857a9f9, fdc2ac372492454c85387589a91f63af
I20240502 01:53:54.455651 21612 raft_consensus.cc:2726] T 472d7827b3074236a81a3ede098f6c0a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:53:54.456581 17454 tablet_service.cc:1514] Processing DeleteTablet for tablet e42126e16cda4d908b00e2e4a6807d6b with delete_type TABLET_DATA_TOMBSTONED (TS 4da690c0fa13407a85c0689ce8de9ade not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54520
I20240502 01:53:54.456760 21637 consensus_queue.cc:260] T dc5ac6b3423248c19209032b774885ca P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:54.457818 20629 raft_consensus.cc:2932] T 9642b9b37a784cf881bfc4971d8c9cb6 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:54.462922 17474 consensus_queue.cc:237] T 9642b9b37a784cf881bfc4971d8c9cb6 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } }
I20240502 01:53:54.468011 21608 tablet_replica.cc:330] T e42126e16cda4d908b00e2e4a6807d6b P 4da690c0fa13407a85c0689ce8de9ade: stopping tablet replica
I20240502 01:53:54.469461 20629 raft_consensus.cc:1261] T 9642b9b37a784cf881bfc4971d8c9cb6 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:53:54.470501 17735 raft_consensus.cc:2932] T 6aacf7c4f19546f581c9a54ad171916b P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:53:54.481551 21329 raft_consensus.cc:1050] T 1f01949cf70440b38e59a5020fd48c44 P 3fceccefba274b1dbbd8216af8b0bc29: attempting to promote NON_VOTER f19cc6cb1e76460aa6711e49a5775091 to VOTER
I20240502 01:53:54.481453 21431 raft_consensus.cc:2932] T 975f253ff03447ff96fe84169c05097e P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:53:54.482403 21329 consensus_queue.cc:237] T 1f01949cf70440b38e59a5020fd48c44 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } }
I20240502 01:53:54.483860 20211 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:54.483969 20232 raft_consensus.cc:2932] T 975f253ff03447ff96fe84169c05097e P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:53:54.491693 21582 consensus_queue.cc:1035] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:54.492844 17335 catalog_manager.cc:5484] T 975f253ff03447ff96fe84169c05097e P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:54.491979 20497 raft_consensus.cc:2932] T 6aacf7c4f19546f581c9a54ad171916b P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:53:54.505841 17334 catalog_manager.cc:5484] T 6aacf7c4f19546f581c9a54ad171916b P dfa1152987644bd781b46c7e42a7a685 reported cstate change: config changed from index 4 to 5, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:53:54.510392 17865 raft_consensus.cc:2932] T 6aacf7c4f19546f581c9a54ad171916b P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:53:54.516295 17604 raft_consensus.cc:2932] T 9642b9b37a784cf881bfc4971d8c9cb6 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:54.520344 20630 consensus_queue.cc:237] T 975f253ff03447ff96fe84169c05097e P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } }
I20240502 01:53:54.520529 17866 raft_consensus.cc:2932] T 52ac2775e02844d195677035796c3847 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:53:54.522433 21608 raft_consensus.cc:2229] T e42126e16cda4d908b00e2e4a6807d6b P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:54.526576 17605 raft_consensus.cc:2932] T 975f253ff03447ff96fe84169c05097e P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:53:54.527451 21216 consensus_queue.cc:1035] T 139304da92ab4f48b8a08fa24ed60c95 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:54.529268 21621 ts_tablet_manager.cc:1888] T b7274d4a59be4949a42f218c3edf743d P f19cc6cb1e76460aa6711e49a5775091: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:53:54.529631 17607 raft_consensus.cc:1261] T 975f253ff03447ff96fe84169c05097e P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:54.532495 17716 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:53:54.536806 17866 raft_consensus.cc:1261] T 1f01949cf70440b38e59a5020fd48c44 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:54.542181 17734 raft_consensus.cc:1261] T 1f01949cf70440b38e59a5020fd48c44 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:54.543063 17604 consensus_queue.cc:237] T 52ac2775e02844d195677035796c3847 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } }
I20240502 01:53:54.544819 21629 consensus_queue.cc:1035] T 1f01949cf70440b38e59a5020fd48c44 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:54.551379 21363 consensus_queue.cc:1035] T 9642b9b37a784cf881bfc4971d8c9cb6 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
I20240502 01:53:54.551947 21622 raft_consensus.cc:482] T dc5ac6b3423248c19209032b774885ca P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:54.552089 21622 raft_consensus.cc:504] T dc5ac6b3423248c19209032b774885ca P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:54.552839 21622 leader_election.cc:288] T dc5ac6b3423248c19209032b774885ca P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:53:54.553529 20231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc5ac6b3423248c19209032b774885ca" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:53:54.553781 20231 raft_consensus.cc:2454] T dc5ac6b3423248c19209032b774885ca P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 0.
I20240502 01:53:54.553694 17474 raft_consensus.cc:2932] T 975f253ff03447ff96fe84169c05097e P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:53:54.554173 21363 raft_consensus.cc:2932] T 9642b9b37a784cf881bfc4971d8c9cb6 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:54.555138 17412 leader_election.cc:302] T dc5ac6b3423248c19209032b774885ca P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:53:54.555441 21499 raft_consensus.cc:2781] T dc5ac6b3423248c19209032b774885ca P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:54.555546 21499 raft_consensus.cc:482] T dc5ac6b3423248c19209032b774885ca P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:54.555647 21499 raft_consensus.cc:3037] T dc5ac6b3423248c19209032b774885ca P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:54.555698 21608 raft_consensus.cc:2258] T e42126e16cda4d908b00e2e4a6807d6b P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:54.558492 21499 raft_consensus.cc:504] T dc5ac6b3423248c19209032b774885ca P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:54.534188 21431 consensus_queue.cc:1035] T 975f253ff03447ff96fe84169c05097e P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:54.541363 20630 raft_consensus.cc:1261] T 1f01949cf70440b38e59a5020fd48c44 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:54.560009 20367 raft_consensus.cc:1261] T 52ac2775e02844d195677035796c3847 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:53:54.553202 20628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc5ac6b3423248c19209032b774885ca" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091" is_pre_election: true
I20240502 01:53:54.560225 20628 raft_consensus.cc:2454] T dc5ac6b3423248c19209032b774885ca P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 0.
I20240502 01:53:54.554636 20629 raft_consensus.cc:2932] T 9642b9b37a784cf881bfc4971d8c9cb6 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:54.560490 21621 log.cc:1196] T b7274d4a59be4949a42f218c3edf743d P f19cc6cb1e76460aa6711e49a5775091: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal/wals/b7274d4a59be4949a42f218c3edf743d
I20240502 01:53:54.560761 20628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc5ac6b3423248c19209032b774885ca" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091"
I20240502 01:53:54.560963 20628 raft_consensus.cc:3037] T dc5ac6b3423248c19209032b774885ca P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:54.567535 21499 leader_election.cc:288] T dc5ac6b3423248c19209032b774885ca P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Requested vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:53:54.567195 20233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc5ac6b3423248c19209032b774885ca" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:53:54.568629 20233 raft_consensus.cc:3037] T dc5ac6b3423248c19209032b774885ca P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:54.571189 20233 raft_consensus.cc:2454] T dc5ac6b3423248c19209032b774885ca P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
I20240502 01:53:54.571569 17412 leader_election.cc:302] T dc5ac6b3423248c19209032b774885ca P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:53:54.572000 21622 raft_consensus.cc:2781] T dc5ac6b3423248c19209032b774885ca P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:54.572193 21622 raft_consensus.cc:686] T dc5ac6b3423248c19209032b774885ca P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Becoming Leader. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Running, Role: LEADER
I20240502 01:53:54.572506 21622 consensus_queue.cc:237] T dc5ac6b3423248c19209032b774885ca P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:54.572528 17604 consensus_queue.cc:237] T 6aacf7c4f19546f581c9a54ad171916b P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } }
I20240502 01:53:54.572954 17866 raft_consensus.cc:1261] T 52ac2775e02844d195677035796c3847 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:53:54.573902 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9642b9b37a784cf881bfc4971d8c9cb6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:54.574848 17606 raft_consensus.cc:1261] T 9642b9b37a784cf881bfc4971d8c9cb6 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:53:54.564713 20628 raft_consensus.cc:2454] T dc5ac6b3423248c19209032b774885ca P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
I20240502 01:53:54.561002 21621 ts_tablet_manager.cc:1909] T b7274d4a59be4949a42f218c3edf743d P f19cc6cb1e76460aa6711e49a5775091: Deleting consensus metadata
I20240502 01:53:54.575704 17335 catalog_manager.cc:5484] T 9642b9b37a784cf881bfc4971d8c9cb6 P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: config changed from index 5 to 6, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New cstate: current_term: 1 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:54.577167 17475 raft_consensus.cc:1261] T 975f253ff03447ff96fe84169c05097e P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:54.577997 17321 catalog_manager.cc:4830] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): tablet b7274d4a59be4949a42f218c3edf743d (table rebalancer_dynamic_table_new_name [id=5b70d9fa9fb44c838d736b49154a72f5]) successfully deleted
I20240502 01:53:54.578243 21431 consensus_queue.cc:1035] T 975f253ff03447ff96fe84169c05097e P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:54.578665 21634 consensus_queue.cc:1035] T 1f01949cf70440b38e59a5020fd48c44 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:54.578675 21629 consensus_queue.cc:1035] T 52ac2775e02844d195677035796c3847 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:53:54.580685 21431 raft_consensus.cc:2932] T 975f253ff03447ff96fe84169c05097e P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:53:54.582075 21608 ts_tablet_manager.cc:1875] T e42126e16cda4d908b00e2e4a6807d6b P 4da690c0fa13407a85c0689ce8de9ade: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:54.583642 21612 consensus_queue.cc:1035] T 9642b9b37a784cf881bfc4971d8c9cb6 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:53:54.583901 20497 raft_consensus.cc:1261] T 6aacf7c4f19546f581c9a54ad171916b P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:53:54.585736 17475 raft_consensus.cc:2932] T 975f253ff03447ff96fe84169c05097e P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:53:54.592978 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 975f253ff03447ff96fe84169c05097e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:54.592756 17605 raft_consensus.cc:2932] T 975f253ff03447ff96fe84169c05097e P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:53:54.551379 21637 ts_tablet_manager.cc:1398] T dc5ac6b3423248c19209032b774885ca P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240502 01:53:54.597239 21629 consensus_queue.cc:1035] T 6aacf7c4f19546f581c9a54ad171916b P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:53:54.598452 17333 catalog_manager.cc:5484] T dc5ac6b3423248c19209032b774885ca P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: term changed from 0 to 1, leader changed from <none> to 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65). New cstate: current_term: 1 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } }
I20240502 01:53:54.592684 17604 raft_consensus.cc:2932] T 9642b9b37a784cf881bfc4971d8c9cb6 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:54.601624 17716 tablet_service.cc:1514] Processing DeleteTablet for tablet 9642b9b37a784cf881bfc4971d8c9cb6 with delete_type TABLET_DATA_TOMBSTONED (TS dfa1152987644bd781b46c7e42a7a685 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52198
I20240502 01:53:54.601969 21591 tablet_replica.cc:330] T 9642b9b37a784cf881bfc4971d8c9cb6 P dfa1152987644bd781b46c7e42a7a685: stopping tablet replica
I20240502 01:53:54.602172 21591 raft_consensus.cc:2229] T 9642b9b37a784cf881bfc4971d8c9cb6 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:54.602399 21591 pending_rounds.cc:62] T 9642b9b37a784cf881bfc4971d8c9cb6 P dfa1152987644bd781b46c7e42a7a685: Trying to abort 1 pending ops.
I20240502 01:53:54.602509 21591 pending_rounds.cc:68] T 9642b9b37a784cf881bfc4971d8c9cb6 P dfa1152987644bd781b46c7e42a7a685: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023062360689860608 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "9642b9b37a784cf881bfc4971d8c9cb6" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } } }
I20240502 01:53:54.602991 21591 raft_consensus.cc:2894] T 9642b9b37a784cf881bfc4971d8c9cb6 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240502 01:53:54.603106 21591 raft_consensus.cc:2866] T 9642b9b37a784cf881bfc4971d8c9cb6 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 01:53:54.603224 21591 raft_consensus.cc:2258] T 9642b9b37a784cf881bfc4971d8c9cb6 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:54.604126 21635 raft_consensus.cc:2932] T 52ac2775e02844d195677035796c3847 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:53:54.604064 20868 raft_consensus.cc:2932] T 1f01949cf70440b38e59a5020fd48c44 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:54.608350 21591 ts_tablet_manager.cc:1875] T 9642b9b37a784cf881bfc4971d8c9cb6 P dfa1152987644bd781b46c7e42a7a685: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:54.609406 17736 raft_consensus.cc:1261] T 6aacf7c4f19546f581c9a54ad171916b P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:53:54.610165 21634 consensus_queue.cc:1035] T 1f01949cf70440b38e59a5020fd48c44 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240502 01:53:54.611922 21634 consensus_queue.cc:1035] T 52ac2775e02844d195677035796c3847 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:53:54.612138 17319 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 52ac2775e02844d195677035796c3847 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:54.612737 21634 consensus_queue.cc:1035] T 6aacf7c4f19546f581c9a54ad171916b P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:53:54.613976 20628 raft_consensus.cc:2932] T 1f01949cf70440b38e59a5020fd48c44 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:54.614531 21591 ts_tablet_manager.cc:1888] T 9642b9b37a784cf881bfc4971d8c9cb6 P dfa1152987644bd781b46c7e42a7a685: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:53:54.614657 21591 log.cc:1196] T 9642b9b37a784cf881bfc4971d8c9cb6 P dfa1152987644bd781b46c7e42a7a685: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/wals/9642b9b37a784cf881bfc4971d8c9cb6
I20240502 01:53:54.615450 17321 catalog_manager.cc:4830] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): tablet 9642b9b37a784cf881bfc4971d8c9cb6 (table rebalance_test_table_3 [id=b996bd60793c4d3c83bf34f4989da909]) successfully deleted
I20240502 01:53:54.618839 17334 catalog_manager.cc:5484] T 975f253ff03447ff96fe84169c05097e P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: config changed from index 3 to 4, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:53:54.621083 17865 raft_consensus.cc:2932] T 1f01949cf70440b38e59a5020fd48c44 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:54.621341 17866 raft_consensus.cc:2932] T 52ac2775e02844d195677035796c3847 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:53:54.623764 20211 tablet_service.cc:1514] Processing DeleteTablet for tablet 975f253ff03447ff96fe84169c05097e with delete_type TABLET_DATA_TOMBSTONED (TS fdc2ac372492454c85387589a91f63af not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44624
I20240502 01:53:54.626760 21216 consensus_queue.cc:1035] T 139304da92ab4f48b8a08fa24ed60c95 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:54.626889 17736 raft_consensus.cc:2932] T 1f01949cf70440b38e59a5020fd48c44 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:54.629786 21329 raft_consensus.cc:2932] T 6aacf7c4f19546f581c9a54ad171916b P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:53:54.630391 20367 raft_consensus.cc:2932] T 52ac2775e02844d195677035796c3847 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:53:54.633378 20497 raft_consensus.cc:2932] T 6aacf7c4f19546f581c9a54ad171916b P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:53:54.635648 17334 catalog_manager.cc:5484] T 1f01949cf70440b38e59a5020fd48c44 P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: config changed from index 4 to 5, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:54.639252 17326 catalog_manager.cc:5484] T 6aacf7c4f19546f581c9a54ad171916b P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: config changed from index 5 to 6, VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) evicted. New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:53:54.640523 17327 catalog_manager.cc:5484] T 52ac2775e02844d195677035796c3847 P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: config changed from index 5 to 6, VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) evicted. New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:53:54.645138 21653 tablet_replica.cc:330] T 975f253ff03447ff96fe84169c05097e P fdc2ac372492454c85387589a91f63af: stopping tablet replica
I20240502 01:53:54.645450 21653 raft_consensus.cc:2229] T 975f253ff03447ff96fe84169c05097e P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:54.645794 21653 raft_consensus.cc:2258] T 975f253ff03447ff96fe84169c05097e P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:54.647032 21608 ts_tablet_manager.cc:1888] T e42126e16cda4d908b00e2e4a6807d6b P 4da690c0fa13407a85c0689ce8de9ade: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:53:54.647033 17846 tablet_service.cc:1514] Processing DeleteTablet for tablet 6aacf7c4f19546f581c9a54ad171916b with delete_type TABLET_DATA_TOMBSTONED (TS bb31bceba76d466db24b9341c04900c3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47226
I20240502 01:53:54.647259 21608 log.cc:1196] T e42126e16cda4d908b00e2e4a6807d6b P 4da690c0fa13407a85c0689ce8de9ade: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/e42126e16cda4d908b00e2e4a6807d6b
I20240502 01:53:54.648000 17321 catalog_manager.cc:4830] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): tablet e42126e16cda4d908b00e2e4a6807d6b (table rebalancer_extra_table_4 [id=dca50b50b02e4a8c8b90e0de6f305bd8]) successfully deleted
I20240502 01:53:54.648669 21653 ts_tablet_manager.cc:1875] T 975f253ff03447ff96fe84169c05097e P fdc2ac372492454c85387589a91f63af: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:54.651213 21656 tablet_replica.cc:330] T 6aacf7c4f19546f581c9a54ad171916b P bb31bceba76d466db24b9341c04900c3: stopping tablet replica
I20240502 01:53:54.653795 17319 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6aacf7c4f19546f581c9a54ad171916b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:54.654099 21656 raft_consensus.cc:2229] T 6aacf7c4f19546f581c9a54ad171916b P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:54.654479 21656 raft_consensus.cc:2258] T 6aacf7c4f19546f581c9a54ad171916b P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:54.656388 21656 ts_tablet_manager.cc:1875] T 6aacf7c4f19546f581c9a54ad171916b P bb31bceba76d466db24b9341c04900c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:54.660008 17736 raft_consensus.cc:2932] T 6aacf7c4f19546f581c9a54ad171916b P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:53:54.662160 21653 ts_tablet_manager.cc:1888] T 975f253ff03447ff96fe84169c05097e P fdc2ac372492454c85387589a91f63af: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:53:54.662299 21653 log.cc:1196] T 975f253ff03447ff96fe84169c05097e P fdc2ac372492454c85387589a91f63af: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal/wals/975f253ff03447ff96fe84169c05097e
I20240502 01:53:54.662688 21656 ts_tablet_manager.cc:1888] T 6aacf7c4f19546f581c9a54ad171916b P bb31bceba76d466db24b9341c04900c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:53:54.662811 21656 log.cc:1196] T 6aacf7c4f19546f581c9a54ad171916b P bb31bceba76d466db24b9341c04900c3: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/wals/6aacf7c4f19546f581c9a54ad171916b
I20240502 01:53:54.663226 17322 catalog_manager.cc:4830] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): tablet 975f253ff03447ff96fe84169c05097e (table rebalancer_extra_table_0 [id=2da12dd1e8bc4159bc3e1040e3db33fe]) successfully deleted
I20240502 01:53:54.663677 17322 catalog_manager.cc:4830] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): tablet 6aacf7c4f19546f581c9a54ad171916b (table rebalance_test_table_15 [id=21fe1df2eeed4efd801d10edf592ecce]) successfully deleted
I20240502 01:53:54.665510 17454 tablet_service.cc:1514] Processing DeleteTablet for tablet 52ac2775e02844d195677035796c3847 with delete_type TABLET_DATA_TOMBSTONED (TS 4da690c0fa13407a85c0689ce8de9ade not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54520
I20240502 01:53:54.665916 21608 tablet_replica.cc:330] T 52ac2775e02844d195677035796c3847 P 4da690c0fa13407a85c0689ce8de9ade: stopping tablet replica
I20240502 01:53:54.666133 21608 raft_consensus.cc:2229] T 52ac2775e02844d195677035796c3847 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:54.666435 21608 raft_consensus.cc:2258] T 52ac2775e02844d195677035796c3847 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:54.670392 21608 ts_tablet_manager.cc:1875] T 52ac2775e02844d195677035796c3847 P 4da690c0fa13407a85c0689ce8de9ade: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:54.677129 21608 ts_tablet_manager.cc:1888] T 52ac2775e02844d195677035796c3847 P 4da690c0fa13407a85c0689ce8de9ade: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:53:54.677253 21608 log.cc:1196] T 52ac2775e02844d195677035796c3847 P 4da690c0fa13407a85c0689ce8de9ade: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/52ac2775e02844d195677035796c3847
I20240502 01:53:54.677927 17321 catalog_manager.cc:4830] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): tablet 52ac2775e02844d195677035796c3847 (table rebalance_test_table_7 [id=c6c3a53b24874f31aa38118254253dcb]) successfully deleted
I20240502 01:53:54.695032 21009 consensus_queue.cc:1035] T 472d7827b3074236a81a3ede098f6c0a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:54.699739 20813 consensus_queue.cc:1035] T 472d7827b3074236a81a3ede098f6c0a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:54.711917 21579 raft_consensus.cc:1050] T ebb9bca9f1bd41b0857939909b1110e4 P 6a36c952c51542c0bf7ea3ec2857a9f9: attempting to promote NON_VOTER 2f306549407b4bb281ccf2994035e4f1 to VOTER
I20240502 01:53:54.712762 21579 consensus_queue.cc:237] T ebb9bca9f1bd41b0857939909b1110e4 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } }
I20240502 01:53:54.715462 21259 raft_consensus.cc:1050] T 65bdab2c9fba48a6828a864278319cc0 P bb31bceba76d466db24b9341c04900c3: attempting to promote NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 to VOTER
I20240502 01:53:54.716228 21259 consensus_queue.cc:237] T 65bdab2c9fba48a6828a864278319cc0 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } }
I20240502 01:53:54.738162 17335 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:53:54.742599 17604 raft_consensus.cc:1261] T 65bdab2c9fba48a6828a864278319cc0 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:54.743639 21581 consensus_queue.cc:1035] T 65bdab2c9fba48a6828a864278319cc0 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:54.755199 20497 raft_consensus.cc:1261] T ebb9bca9f1bd41b0857939909b1110e4 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:53:54.755765 20813 consensus_queue.cc:1035] T ebb9bca9f1bd41b0857939909b1110e4 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:53:54.768549 17736 raft_consensus.cc:1261] T ebb9bca9f1bd41b0857939909b1110e4 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:53:54.768930 20233 raft_consensus.cc:1261] T ebb9bca9f1bd41b0857939909b1110e4 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:53:54.769691 21582 consensus_queue.cc:1035] T ebb9bca9f1bd41b0857939909b1110e4 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:53:54.771531 21582 consensus_queue.cc:1035] T ebb9bca9f1bd41b0857939909b1110e4 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:53:54.777877 20367 raft_consensus.cc:1261] T 65bdab2c9fba48a6828a864278319cc0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:54.779038 21581 consensus_queue.cc:1035] T 65bdab2c9fba48a6828a864278319cc0 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:54.784489 20813 raft_consensus.cc:2932] T ebb9bca9f1bd41b0857939909b1110e4 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:53:54.788108 20628 raft_consensus.cc:1261] T dc5ac6b3423248c19209032b774885ca P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:54.789048 20231 raft_consensus.cc:1261] T dc5ac6b3423248c19209032b774885ca P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:54.788998 21612 consensus_queue.cc:1035] T dc5ac6b3423248c19209032b774885ca P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:54.790098 20233 raft_consensus.cc:2932] T ebb9bca9f1bd41b0857939909b1110e4 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:53:54.796490 17736 raft_consensus.cc:1261] T 65bdab2c9fba48a6828a864278319cc0 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:54.799439 21612 consensus_queue.cc:1035] T dc5ac6b3423248c19209032b774885ca P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:54.800225 17332 catalog_manager.cc:5484] T ebb9bca9f1bd41b0857939909b1110e4 P fdc2ac372492454c85387589a91f63af reported cstate change: config changed from index 2 to 3, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:53:54.804314 20497 raft_consensus.cc:2932] T ebb9bca9f1bd41b0857939909b1110e4 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:53:54.805260 17736 raft_consensus.cc:2932] T ebb9bca9f1bd41b0857939909b1110e4 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:53:54.808111 20367 consensus_queue.cc:237] T ebb9bca9f1bd41b0857939909b1110e4 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } }
I20240502 01:53:54.809209 17327 catalog_manager.cc:5576] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411) does not have the latest schema for tablet dc5ac6b3423248c19209032b774885ca (table rebalancer_dynamic_table [id=4bc119cbe600465a9c4d50d5f21a0e9d]). Expected version 1 got 0
I20240502 01:53:54.809834 21259 consensus_queue.cc:1035] T 65bdab2c9fba48a6828a864278319cc0 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:54.817179 17736 raft_consensus.cc:1261] T ebb9bca9f1bd41b0857939909b1110e4 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:54.818935 21672 tablet.cc:1722] T dc5ac6b3423248c19209032b774885ca P 4da690c0fa13407a85c0689ce8de9ade: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:53:54.820196 21673 tablet.cc:1722] T dc5ac6b3423248c19209032b774885ca P f19cc6cb1e76460aa6711e49a5775091: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:53:54.820717 21673 tablet.cc:1620] T dc5ac6b3423248c19209032b774885ca P f19cc6cb1e76460aa6711e49a5775091: MemRowSet was empty: no flush needed.
I20240502 01:53:54.821547 21672 tablet.cc:1620] T dc5ac6b3423248c19209032b774885ca P 4da690c0fa13407a85c0689ce8de9ade: MemRowSet was empty: no flush needed.
I20240502 01:53:54.822544 20813 consensus_queue.cc:1035] T ebb9bca9f1bd41b0857939909b1110e4 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:54.825682 21582 raft_consensus.cc:2932] T ebb9bca9f1bd41b0857939909b1110e4 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:53:54.832616 17333 catalog_manager.cc:5576] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411) does not have the latest schema for tablet dc5ac6b3423248c19209032b774885ca (table rebalancer_dynamic_table [id=4bc119cbe600465a9c4d50d5f21a0e9d]). Expected version 1 got 0
I20240502 01:53:54.834519 17736 raft_consensus.cc:2932] T ebb9bca9f1bd41b0857939909b1110e4 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:53:54.837244 17326 catalog_manager.cc:5576] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339) does not have the latest schema for tablet dc5ac6b3423248c19209032b774885ca (table rebalancer_dynamic_table [id=4bc119cbe600465a9c4d50d5f21a0e9d]). Expected version 1 got 0
I20240502 01:53:54.838624 20497 raft_consensus.cc:1261] T ebb9bca9f1bd41b0857939909b1110e4 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:54.839282 21579 consensus_queue.cc:1035] T ebb9bca9f1bd41b0857939909b1110e4 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:54.840651 20497 raft_consensus.cc:2932] T ebb9bca9f1bd41b0857939909b1110e4 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:53:54.846171 17334 catalog_manager.cc:5484] T ebb9bca9f1bd41b0857939909b1110e4 P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: config changed from index 3 to 4, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:53:54.847342 17326 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=4bc119cbe600465a9c4d50d5f21a0e9d] alter complete (version 1)
I20240502 01:53:54.850720 20211 tablet_service.cc:1514] Processing DeleteTablet for tablet ebb9bca9f1bd41b0857939909b1110e4 with delete_type TABLET_DATA_TOMBSTONED (TS fdc2ac372492454c85387589a91f63af not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44624
I20240502 01:53:54.851099 21653 tablet_replica.cc:330] T ebb9bca9f1bd41b0857939909b1110e4 P fdc2ac372492454c85387589a91f63af: stopping tablet replica
I20240502 01:53:54.851313 21653 raft_consensus.cc:2229] T ebb9bca9f1bd41b0857939909b1110e4 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:54.851647 21653 raft_consensus.cc:2258] T ebb9bca9f1bd41b0857939909b1110e4 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:54.853269 21653 ts_tablet_manager.cc:1875] T ebb9bca9f1bd41b0857939909b1110e4 P fdc2ac372492454c85387589a91f63af: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:54.860587 17327 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_extra_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:54.869220 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ebb9bca9f1bd41b0857939909b1110e4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:54.870266 20347 tablet_service.cc:1467] Processing CreateTablet for tablet ee418fc4028a422eabac02ed63ea9047 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=54db4901eae145fbbc5c66c0719703f1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:54.870432 20478 tablet_service.cc:1467] Processing CreateTablet for tablet ee418fc4028a422eabac02ed63ea9047 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=54db4901eae145fbbc5c66c0719703f1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:54.870754 20347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee418fc4028a422eabac02ed63ea9047. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:54.870900 20478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee418fc4028a422eabac02ed63ea9047. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:54.875427 20476 tablet_service.cc:1467] Processing CreateTablet for tablet 18549019e31b47d899d011cef0637221 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=54db4901eae145fbbc5c66c0719703f1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:54.875818 20476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18549019e31b47d899d011cef0637221. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:54.876267 20345 tablet_service.cc:1467] Processing CreateTablet for tablet 18549019e31b47d899d011cef0637221 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=54db4901eae145fbbc5c66c0719703f1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:54.876654 20345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18549019e31b47d899d011cef0637221. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:54.881412 20346 tablet_service.cc:1467] Processing CreateTablet for tablet 29d8cb76dc514e7b9002bd2291d8ee8d (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=54db4901eae145fbbc5c66c0719703f1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:54.881484 17587 tablet_service.cc:1467] Processing CreateTablet for tablet 18549019e31b47d899d011cef0637221 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=54db4901eae145fbbc5c66c0719703f1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:54.881843 20346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29d8cb76dc514e7b9002bd2291d8ee8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:54.881999 17587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18549019e31b47d899d011cef0637221. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:54.885681 20477 tablet_service.cc:1467] Processing CreateTablet for tablet 29d8cb76dc514e7b9002bd2291d8ee8d (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=54db4901eae145fbbc5c66c0719703f1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:54.886106 20477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29d8cb76dc514e7b9002bd2291d8ee8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:54.886248 17716 tablet_service.cc:1467] Processing CreateTablet for tablet 29d8cb76dc514e7b9002bd2291d8ee8d (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=54db4901eae145fbbc5c66c0719703f1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:54.886656 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29d8cb76dc514e7b9002bd2291d8ee8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:54.887168 21680 tablet_bootstrap.cc:492] T ee418fc4028a422eabac02ed63ea9047 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:53:54.893332 21680 tablet_bootstrap.cc:654] T ee418fc4028a422eabac02ed63ea9047 P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:54.904603 21667 raft_consensus.cc:2932] T 65bdab2c9fba48a6828a864278319cc0 P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:53:54.906687 20367 raft_consensus.cc:2932] T 65bdab2c9fba48a6828a864278319cc0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:53:54.912462 17604 raft_consensus.cc:2932] T 65bdab2c9fba48a6828a864278319cc0 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:53:54.914826 21653 ts_tablet_manager.cc:1888] T ebb9bca9f1bd41b0857939909b1110e4 P fdc2ac372492454c85387589a91f63af: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:53:54.921241 17335 catalog_manager.cc:5484] T 65bdab2c9fba48a6828a864278319cc0 P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: config changed from index 4 to 5, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:53:54.923791 21653 log.cc:1196] T ebb9bca9f1bd41b0857939909b1110e4 P fdc2ac372492454c85387589a91f63af: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal/wals/ebb9bca9f1bd41b0857939909b1110e4
I20240502 01:53:54.925527 21685 tablet_bootstrap.cc:492] T 18549019e31b47d899d011cef0637221 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:54.927228 17322 catalog_manager.cc:4830] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): tablet ebb9bca9f1bd41b0857939909b1110e4 (table rebalancer_extra_table_0 [id=2da12dd1e8bc4159bc3e1040e3db33fe]) successfully deleted
I20240502 01:53:54.928850 21685 tablet_bootstrap.cc:654] T 18549019e31b47d899d011cef0637221 P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:54.932332 17736 raft_consensus.cc:2932] T 65bdab2c9fba48a6828a864278319cc0 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:53:54.934296 17454 tablet_service.cc:1467] Processing CreateTablet for tablet ee418fc4028a422eabac02ed63ea9047 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=54db4901eae145fbbc5c66c0719703f1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:54.934770 17454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee418fc4028a422eabac02ed63ea9047. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:54.942598 21637 tablet_bootstrap.cc:492] T ee418fc4028a422eabac02ed63ea9047 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:54.943032 21677 tablet.cc:1722] T dc5ac6b3423248c19209032b774885ca P fdc2ac372492454c85387589a91f63af: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:53:54.943522 21677 tablet.cc:1620] T dc5ac6b3423248c19209032b774885ca P fdc2ac372492454c85387589a91f63af: MemRowSet was empty: no flush needed.
I20240502 01:53:54.943455 17866 consensus_queue.cc:237] T 65bdab2c9fba48a6828a864278319cc0 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } }
I20240502 01:53:54.944314 21681 tablet_bootstrap.cc:492] T ee418fc4028a422eabac02ed63ea9047 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:53:54.944427 21637 tablet_bootstrap.cc:654] T ee418fc4028a422eabac02ed63ea9047 P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:54.945587 17736 raft_consensus.cc:1261] T 65bdab2c9fba48a6828a864278319cc0 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:53:54.946496 20367 raft_consensus.cc:1261] T 65bdab2c9fba48a6828a864278319cc0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:53:54.949643 21681 tablet_bootstrap.cc:654] T ee418fc4028a422eabac02ed63ea9047 P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:54.950464 21581 consensus_queue.cc:1035] T 65bdab2c9fba48a6828a864278319cc0 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:53:54.951473 17335 catalog_manager.cc:5576] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339) does not have the latest schema for tablet dc5ac6b3423248c19209032b774885ca (table rebalancer_dynamic_table [id=4bc119cbe600465a9c4d50d5f21a0e9d]). Expected version 1 got 0
I20240502 01:53:54.951550 21581 consensus_queue.cc:1035] T 65bdab2c9fba48a6828a864278319cc0 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:53:54.952807 21683 tablet_bootstrap.cc:492] T 29d8cb76dc514e7b9002bd2291d8ee8d P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:54.955686 21683 tablet_bootstrap.cc:654] T 29d8cb76dc514e7b9002bd2291d8ee8d P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:54.957237 21667 raft_consensus.cc:2932] T 65bdab2c9fba48a6828a864278319cc0 P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:53:54.958542 20367 raft_consensus.cc:2932] T 65bdab2c9fba48a6828a864278319cc0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:53:54.961885 17322 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 65bdab2c9fba48a6828a864278319cc0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:54.963201 17332 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:53:54.967098 21685 tablet_bootstrap.cc:492] T 18549019e31b47d899d011cef0637221 P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:54.967375 21685 ts_tablet_manager.cc:1367] T 18549019e31b47d899d011cef0637221 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.000s
I20240502 01:53:54.969239 21637 tablet_bootstrap.cc:492] T ee418fc4028a422eabac02ed63ea9047 P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:54.969328 21685 raft_consensus.cc:348] T 18549019e31b47d899d011cef0637221 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:54.969655 21637 ts_tablet_manager.cc:1367] T ee418fc4028a422eabac02ed63ea9047 P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.008s
I20240502 01:53:54.970443 21685 raft_consensus.cc:374] T 18549019e31b47d899d011cef0637221 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:54.970546 21637 raft_consensus.cc:348] T ee418fc4028a422eabac02ed63ea9047 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:54.970825 21685 raft_consensus.cc:729] T 18549019e31b47d899d011cef0637221 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:54.971347 21685 consensus_queue.cc:260] T 18549019e31b47d899d011cef0637221 P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:54.971493 21637 raft_consensus.cc:374] T ee418fc4028a422eabac02ed63ea9047 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:54.971570 21637 raft_consensus.cc:729] T ee418fc4028a422eabac02ed63ea9047 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:54.971859 21637 consensus_queue.cc:260] T ee418fc4028a422eabac02ed63ea9047 P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:54.972442 21685 ts_tablet_manager.cc:1398] T 18549019e31b47d899d011cef0637221 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:53:54.973847 21637 ts_tablet_manager.cc:1398] T ee418fc4028a422eabac02ed63ea9047 P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:53:54.979301 17334 catalog_manager.cc:5484] T 65bdab2c9fba48a6828a864278319cc0 P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: config changed from index 5 to 6, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:53:54.982995 17335 catalog_manager.cc:5576] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339) does not have the latest schema for tablet dc5ac6b3423248c19209032b774885ca (table rebalancer_dynamic_table [id=4bc119cbe600465a9c4d50d5f21a0e9d]). Expected version 2 got 1
I20240502 01:53:54.983142 17735 raft_consensus.cc:2932] T 65bdab2c9fba48a6828a864278319cc0 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:53:54.985539 21681 tablet_bootstrap.cc:492] T ee418fc4028a422eabac02ed63ea9047 P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:53:54.985688 21681 ts_tablet_manager.cc:1367] T ee418fc4028a422eabac02ed63ea9047 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.042s user 0.002s sys 0.002s
I20240502 01:53:54.985631 21672 tablet.cc:1722] T dc5ac6b3423248c19209032b774885ca P 4da690c0fa13407a85c0689ce8de9ade: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:53:54.986114 21672 tablet.cc:1620] T dc5ac6b3423248c19209032b774885ca P 4da690c0fa13407a85c0689ce8de9ade: MemRowSet was empty: no flush needed.
I20240502 01:53:54.986531 21681 raft_consensus.cc:348] T ee418fc4028a422eabac02ed63ea9047 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:54.987744 21681 raft_consensus.cc:374] T ee418fc4028a422eabac02ed63ea9047 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:54.987844 21681 raft_consensus.cc:729] T ee418fc4028a422eabac02ed63ea9047 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:53:54.988103 21681 consensus_queue.cc:260] T ee418fc4028a422eabac02ed63ea9047 P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:54.988664 21681 ts_tablet_manager.cc:1398] T ee418fc4028a422eabac02ed63ea9047 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 01:53:54.989075 21681 tablet_bootstrap.cc:492] T 29d8cb76dc514e7b9002bd2291d8ee8d P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:53:54.990274 21673 tablet.cc:1722] T dc5ac6b3423248c19209032b774885ca P f19cc6cb1e76460aa6711e49a5775091: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:53:54.990801 21673 tablet.cc:1620] T dc5ac6b3423248c19209032b774885ca P f19cc6cb1e76460aa6711e49a5775091: MemRowSet was empty: no flush needed.
I20240502 01:53:54.992940 21677 tablet.cc:1722] T dc5ac6b3423248c19209032b774885ca P fdc2ac372492454c85387589a91f63af: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:53:54.993435 21677 tablet.cc:1620] T dc5ac6b3423248c19209032b774885ca P fdc2ac372492454c85387589a91f63af: MemRowSet was empty: no flush needed.
I20240502 01:53:55.004230 17587 tablet_service.cc:1514] Processing DeleteTablet for tablet 65bdab2c9fba48a6828a864278319cc0 with delete_type TABLET_DATA_TOMBSTONED (TS 3fceccefba274b1dbbd8216af8b0bc29 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55232
I20240502 01:53:55.009598 17327 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=4bc119cbe600465a9c4d50d5f21a0e9d] alter complete (version 2)
I20240502 01:53:55.014742 21681 tablet_bootstrap.cc:654] T 29d8cb76dc514e7b9002bd2291d8ee8d P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:55.014745 17474 consensus_queue.cc:237] T 5e8e78bd417d46cdb166087dbfa86446 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } }
I20240502 01:53:55.025913 21683 tablet_bootstrap.cc:492] T 29d8cb76dc514e7b9002bd2291d8ee8d P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:55.026121 21683 ts_tablet_manager.cc:1367] T 29d8cb76dc514e7b9002bd2291d8ee8d P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.073s user 0.003s sys 0.000s
I20240502 01:53:55.027060 21680 tablet_bootstrap.cc:492] T ee418fc4028a422eabac02ed63ea9047 P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:53:55.027022 21683 raft_consensus.cc:348] T 29d8cb76dc514e7b9002bd2291d8ee8d P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:55.027227 21683 raft_consensus.cc:374] T 29d8cb76dc514e7b9002bd2291d8ee8d P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:55.027380 21680 ts_tablet_manager.cc:1367] T ee418fc4028a422eabac02ed63ea9047 P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.140s user 0.056s sys 0.056s
I20240502 01:53:55.027464 21683 raft_consensus.cc:729] T 29d8cb76dc514e7b9002bd2291d8ee8d P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:55.027732 21683 consensus_queue.cc:260] T 29d8cb76dc514e7b9002bd2291d8ee8d P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:55.028327 21680 raft_consensus.cc:348] T ee418fc4028a422eabac02ed63ea9047 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:55.028576 21680 raft_consensus.cc:374] T ee418fc4028a422eabac02ed63ea9047 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:55.028709 21680 raft_consensus.cc:729] T ee418fc4028a422eabac02ed63ea9047 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:53:55.028868 21690 tablet_replica.cc:330] T 65bdab2c9fba48a6828a864278319cc0 P 3fceccefba274b1dbbd8216af8b0bc29: stopping tablet replica
I20240502 01:53:55.029325 21683 ts_tablet_manager.cc:1398] T 29d8cb76dc514e7b9002bd2291d8ee8d P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:55.029314 21680 consensus_queue.cc:260] T ee418fc4028a422eabac02ed63ea9047 P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:55.030689 21680 ts_tablet_manager.cc:1398] T ee418fc4028a422eabac02ed63ea9047 P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:53:55.031001 21680 tablet_bootstrap.cc:492] T 29d8cb76dc514e7b9002bd2291d8ee8d P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
W20240502 01:53:55.031934 17409 consensus_peers.cc:485] T 5e8e78bd417d46cdb166087dbfa86446 P 4da690c0fa13407a85c0689ce8de9ade -> Peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): Couldn't send request to peer 3fceccefba274b1dbbd8216af8b0bc29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5e8e78bd417d46cdb166087dbfa86446. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:55.032693 21680 tablet_bootstrap.cc:654] T 29d8cb76dc514e7b9002bd2291d8ee8d P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:55.033625 20367 raft_consensus.cc:1261] T 5e8e78bd417d46cdb166087dbfa86446 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:55.034286 21622 consensus_queue.cc:1035] T 5e8e78bd417d46cdb166087dbfa86446 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:53:55.036854 21594 raft_consensus.cc:482] T ee418fc4028a422eabac02ed63ea9047 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:55.036993 21594 raft_consensus.cc:504] T ee418fc4028a422eabac02ed63ea9047 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:55.037364 21681 tablet_bootstrap.cc:492] T 29d8cb76dc514e7b9002bd2291d8ee8d P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:53:55.037515 21681 ts_tablet_manager.cc:1367] T 29d8cb76dc514e7b9002bd2291d8ee8d P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.049s user 0.001s sys 0.002s
I20240502 01:53:55.037864 21594 leader_election.cc:288] T ee418fc4028a422eabac02ed63ea9047 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:55.038359 21681 raft_consensus.cc:348] T 29d8cb76dc514e7b9002bd2291d8ee8d P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:55.038574 21681 raft_consensus.cc:374] T 29d8cb76dc514e7b9002bd2291d8ee8d P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:55.038651 21681 raft_consensus.cc:729] T 29d8cb76dc514e7b9002bd2291d8ee8d P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:53:55.038543 20365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee418fc4028a422eabac02ed63ea9047" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:53:55.038815 20365 raft_consensus.cc:2454] T ee418fc4028a422eabac02ed63ea9047 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 0.
I20240502 01:53:55.038901 21681 consensus_queue.cc:260] T 29d8cb76dc514e7b9002bd2291d8ee8d P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:55.039404 21681 ts_tablet_manager.cc:1398] T 29d8cb76dc514e7b9002bd2291d8ee8d P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:55.039438 20433 leader_election.cc:302] T ee418fc4028a422eabac02ed63ea9047 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 6a36c952c51542c0bf7ea3ec2857a9f9; no voters:
I20240502 01:53:55.039801 21681 tablet_bootstrap.cc:492] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:53:55.040020 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee418fc4028a422eabac02ed63ea9047" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:53:55.040227 17475 raft_consensus.cc:2454] T ee418fc4028a422eabac02ed63ea9047 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 0.
I20240502 01:53:55.040478 21594 raft_consensus.cc:2781] T ee418fc4028a422eabac02ed63ea9047 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:55.040588 21594 raft_consensus.cc:482] T ee418fc4028a422eabac02ed63ea9047 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:55.040670 21594 raft_consensus.cc:3037] T ee418fc4028a422eabac02ed63ea9047 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:55.041131 20497 raft_consensus.cc:1261] T 5e8e78bd417d46cdb166087dbfa86446 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:55.041447 21681 tablet_bootstrap.cc:654] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:55.041700 21622 consensus_queue.cc:1035] T 5e8e78bd417d46cdb166087dbfa86446 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:53:55.043288 21594 raft_consensus.cc:504] T ee418fc4028a422eabac02ed63ea9047 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:55.044348 20365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee418fc4028a422eabac02ed63ea9047" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
I20240502 01:53:55.044526 20365 raft_consensus.cc:3037] T ee418fc4028a422eabac02ed63ea9047 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:55.047713 20365 raft_consensus.cc:2454] T ee418fc4028a422eabac02ed63ea9047 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 1.
I20240502 01:53:55.048226 20433 leader_election.cc:302] T ee418fc4028a422eabac02ed63ea9047 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 6a36c952c51542c0bf7ea3ec2857a9f9; no voters:
I20240502 01:53:55.048501 21216 raft_consensus.cc:2781] T ee418fc4028a422eabac02ed63ea9047 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:55.050426 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee418fc4028a422eabac02ed63ea9047" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:53:55.050621 17475 raft_consensus.cc:3037] T ee418fc4028a422eabac02ed63ea9047 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:55.053745 17475 raft_consensus.cc:2454] T ee418fc4028a422eabac02ed63ea9047 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 1.
I20240502 01:53:55.054385 21594 leader_election.cc:288] T ee418fc4028a422eabac02ed63ea9047 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 election: Requested vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:55.057193 21681 tablet_bootstrap.cc:492] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:53:55.057384 21681 ts_tablet_manager.cc:1367] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.007s
I20240502 01:53:55.058108 21681 raft_consensus.cc:348] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:55.058396 21681 raft_consensus.cc:374] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:55.058526 21681 raft_consensus.cc:729] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:53:55.058804 21681 consensus_queue.cc:260] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:55.060226 21681 ts_tablet_manager.cc:1398] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:53:55.060695 21216 raft_consensus.cc:686] T ee418fc4028a422eabac02ed63ea9047 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Running, Role: LEADER
I20240502 01:53:55.060989 21216 consensus_queue.cc:237] T ee418fc4028a422eabac02ed63ea9047 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:53:55.061997 21690 raft_consensus.cc:2229] T 65bdab2c9fba48a6828a864278319cc0 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:55.064605 21690 raft_consensus.cc:2258] T 65bdab2c9fba48a6828a864278319cc0 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:55.065910 21690 ts_tablet_manager.cc:1875] T 65bdab2c9fba48a6828a864278319cc0 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:55.068439 21544 raft_consensus.cc:2932] T 5e8e78bd417d46cdb166087dbfa86446 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } } }
I20240502 01:53:55.068639 17334 catalog_manager.cc:5484] T ee418fc4028a422eabac02ed63ea9047 P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71). New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:55.068938 20497 raft_consensus.cc:2932] T 5e8e78bd417d46cdb166087dbfa86446 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } } }
I20240502 01:53:55.070609 20365 raft_consensus.cc:2932] T 5e8e78bd417d46cdb166087dbfa86446 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } } }
I20240502 01:53:55.073627 21680 tablet_bootstrap.cc:492] T 29d8cb76dc514e7b9002bd2291d8ee8d P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:53:55.073875 21680 ts_tablet_manager.cc:1367] T 29d8cb76dc514e7b9002bd2291d8ee8d P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.043s user 0.000s sys 0.007s
I20240502 01:53:55.074864 21680 raft_consensus.cc:348] T 29d8cb76dc514e7b9002bd2291d8ee8d P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:55.075091 21680 raft_consensus.cc:374] T 29d8cb76dc514e7b9002bd2291d8ee8d P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:55.075194 21680 raft_consensus.cc:729] T 29d8cb76dc514e7b9002bd2291d8ee8d P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:53:55.075469 21680 consensus_queue.cc:260] T 29d8cb76dc514e7b9002bd2291d8ee8d P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:55.076159 21680 ts_tablet_manager.cc:1398] T 29d8cb76dc514e7b9002bd2291d8ee8d P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:53:55.075992 17327 catalog_manager.cc:5484] T 5e8e78bd417d46cdb166087dbfa86446 P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: config changed from index -1 to 2, NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) added. New cstate: current_term: 1 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:55.076514 21680 tablet_bootstrap.cc:492] T 18549019e31b47d899d011cef0637221 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:53:55.077873 21582 raft_consensus.cc:482] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:55.078038 21582 raft_consensus.cc:504] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:55.078219 21680 tablet_bootstrap.cc:654] T 18549019e31b47d899d011cef0637221 P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:55.078755 21582 leader_election.cc:288] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:55.079365 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18549019e31b47d899d011cef0637221" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:53:55.079643 17604 raft_consensus.cc:2454] T 18549019e31b47d899d011cef0637221 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 0.
I20240502 01:53:55.080121 20300 leader_election.cc:302] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, 6a36c952c51542c0bf7ea3ec2857a9f9; no voters:
I20240502 01:53:55.080899 20497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18549019e31b47d899d011cef0637221" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
W20240502 01:53:55.081518 20303 leader_election.cc:341] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:53:55.082512 21582 raft_consensus.cc:2781] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:55.082633 21582 raft_consensus.cc:482] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:55.082726 21582 raft_consensus.cc:3037] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:55.086097 21582 raft_consensus.cc:504] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:55.086982 21582 leader_election.cc:288] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Requested vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:55.086982 20497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18549019e31b47d899d011cef0637221" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
I20240502 01:53:55.087435 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18549019e31b47d899d011cef0637221" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
W20240502 01:53:55.087604 20303 leader_election.cc:341] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:53:55.087664 17604 raft_consensus.cc:3037] T 18549019e31b47d899d011cef0637221 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:55.089000 21690 ts_tablet_manager.cc:1888] T 65bdab2c9fba48a6828a864278319cc0 P 3fceccefba274b1dbbd8216af8b0bc29: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:53:55.089202 21690 log.cc:1196] T 65bdab2c9fba48a6828a864278319cc0 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/wals/65bdab2c9fba48a6828a864278319cc0
I20240502 01:53:55.090296 17319 catalog_manager.cc:4830] TS 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): tablet 65bdab2c9fba48a6828a864278319cc0 (table rebalance_test_table_17 [id=c6474d0c5256497eb6fa5844f3530b62]) successfully deleted
I20240502 01:53:55.091028 17604 raft_consensus.cc:2454] T 18549019e31b47d899d011cef0637221 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 1.
I20240502 01:53:55.091536 20300 leader_election.cc:302] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, 6a36c952c51542c0bf7ea3ec2857a9f9; no voters: 2f306549407b4bb281ccf2994035e4f1
I20240502 01:53:55.091849 21582 raft_consensus.cc:2781] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:55.092005 21582 raft_consensus.cc:686] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Becoming Leader. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Running, Role: LEADER
I20240502 01:53:55.092232 21582 consensus_queue.cc:237] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:55.094058 21680 tablet_bootstrap.cc:492] T 18549019e31b47d899d011cef0637221 P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:53:55.094195 21680 ts_tablet_manager.cc:1367] T 18549019e31b47d899d011cef0637221 P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.009s
I20240502 01:53:55.094889 21680 raft_consensus.cc:348] T 18549019e31b47d899d011cef0637221 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:55.095099 21680 raft_consensus.cc:374] T 18549019e31b47d899d011cef0637221 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:55.095216 21680 raft_consensus.cc:729] T 18549019e31b47d899d011cef0637221 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:53:55.095487 21680 consensus_queue.cc:260] T 18549019e31b47d899d011cef0637221 P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:55.096127 21680 ts_tablet_manager.cc:1398] T 18549019e31b47d899d011cef0637221 P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:53:55.100740 17334 catalog_manager.cc:5484] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70). New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } }
I20240502 01:53:55.104425 17474 consensus_queue.cc:237] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } }
I20240502 01:53:55.107143 17736 raft_consensus.cc:1261] T 0618761d30f34cfe9650d13c9275cdcb P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:55.107163 17866 raft_consensus.cc:1261] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:55.107659 21612 consensus_queue.cc:1035] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:55.107976 21544 consensus_queue.cc:1035] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:53:55.108393 17412 consensus_peers.cc:485] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0618761d30f34cfe9650d13c9275cdcb. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:55.110687 17334 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:53:55.110771 21544 raft_consensus.cc:2932] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:53:55.111407 17866 raft_consensus.cc:2932] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:53:55.112246 17736 raft_consensus.cc:2932] T 0618761d30f34cfe9650d13c9275cdcb P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:53:55.118737 17327 catalog_manager.cc:5484] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: config changed from index -1 to 4, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New cstate: current_term: 1 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:55.121687 21672 tablet.cc:1722] T dc5ac6b3423248c19209032b774885ca P 4da690c0fa13407a85c0689ce8de9ade: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:53:55.122599 21673 tablet.cc:1722] T dc5ac6b3423248c19209032b774885ca P f19cc6cb1e76460aa6711e49a5775091: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:53:55.128360 21677 tablet.cc:1722] T dc5ac6b3423248c19209032b774885ca P fdc2ac372492454c85387589a91f63af: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:53:55.134780 17335 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table_new_name [id=4bc119cbe600465a9c4d50d5f21a0e9d] alter complete (version 3)
I20240502 01:53:55.154256 17736 consensus_queue.cc:237] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:53:55.157649 17866 raft_consensus.cc:1261] T 5df83f83d2674155afba93c2d992af0a P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:55.158627 21530 consensus_queue.cc:1035] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:55.187661 21530 raft_consensus.cc:2932] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:53:55.188158 17474 raft_consensus.cc:1261] T 5df83f83d2674155afba93c2d992af0a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:55.188861 21674 consensus_queue.cc:1035] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:55.199225 17474 raft_consensus.cc:2932] T 5df83f83d2674155afba93c2d992af0a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:53:55.201308 17604 consensus_queue.cc:237] T 1f01949cf70440b38e59a5020fd48c44 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } }
I20240502 01:53:55.203547 20628 raft_consensus.cc:1261] T 1f01949cf70440b38e59a5020fd48c44 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:53:55.203696 17866 raft_consensus.cc:1261] T 1f01949cf70440b38e59a5020fd48c44 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:53:55.204073 21629 consensus_queue.cc:1035] T 1f01949cf70440b38e59a5020fd48c44 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:53:55.204716 21629 consensus_queue.cc:1035] T 1f01949cf70440b38e59a5020fd48c44 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:53:55.205215 17335 catalog_manager.cc:5484] T 5df83f83d2674155afba93c2d992af0a P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: config changed from index -1 to 4, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:53:55.210999 17865 raft_consensus.cc:2932] T 5df83f83d2674155afba93c2d992af0a P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:53:55.212044 21629 raft_consensus.cc:2932] T 1f01949cf70440b38e59a5020fd48c44 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:55.215891 17319 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1f01949cf70440b38e59a5020fd48c44 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:55.216820 20628 raft_consensus.cc:2932] T 1f01949cf70440b38e59a5020fd48c44 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:55.217648 17866 raft_consensus.cc:2932] T 1f01949cf70440b38e59a5020fd48c44 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:55.220018 17334 catalog_manager.cc:5484] T 1f01949cf70440b38e59a5020fd48c44 P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: config changed from index 5 to 6, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:55.224774 17716 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f01949cf70440b38e59a5020fd48c44 with delete_type TABLET_DATA_TOMBSTONED (TS dfa1152987644bd781b46c7e42a7a685 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52198
W20240502 01:53:55.225109 17672 consensus_peers.cc:485] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5df83f83d2674155afba93c2d992af0a. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:55.238512 17334 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:53:55.238761 17334 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:53:55.241894 17334 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet dc5ac6b3423248c19209032b774885ca
I20240502 01:53:55.243103 17454 tablet_service.cc:1514] Processing DeleteTablet for tablet dc5ac6b3423248c19209032b774885ca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:55 UTC) from {username='slave'} at 127.0.0.1:54520
I20240502 01:53:55.243799 20608 tablet_service.cc:1514] Processing DeleteTablet for tablet dc5ac6b3423248c19209032b774885ca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:55 UTC) from {username='slave'} at 127.0.0.1:33698
I20240502 01:53:55.244169 20211 tablet_service.cc:1514] Processing DeleteTablet for tablet dc5ac6b3423248c19209032b774885ca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:55 UTC) from {username='slave'} at 127.0.0.1:44624
I20240502 01:53:55.244467 21653 tablet_replica.cc:330] T dc5ac6b3423248c19209032b774885ca P fdc2ac372492454c85387589a91f63af: stopping tablet replica
I20240502 01:53:55.244673 21653 raft_consensus.cc:2229] T dc5ac6b3423248c19209032b774885ca P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:55.244894 21653 raft_consensus.cc:2258] T dc5ac6b3423248c19209032b774885ca P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:55.243808 21729 tablet_replica.cc:330] T dc5ac6b3423248c19209032b774885ca P 4da690c0fa13407a85c0689ce8de9ade: stopping tablet replica
I20240502 01:53:55.245795 21729 raft_consensus.cc:2229] T dc5ac6b3423248c19209032b774885ca P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:55.246125 21653 ts_tablet_manager.cc:1875] T dc5ac6b3423248c19209032b774885ca P fdc2ac372492454c85387589a91f63af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:55.246323 21724 tablet_replica.cc:330] T 1f01949cf70440b38e59a5020fd48c44 P dfa1152987644bd781b46c7e42a7a685: stopping tablet replica
I20240502 01:53:55.246569 21724 raft_consensus.cc:2229] T 1f01949cf70440b38e59a5020fd48c44 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:55.246677 21729 raft_consensus.cc:2258] T dc5ac6b3423248c19209032b774885ca P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:55.247033 21724 raft_consensus.cc:2258] T 1f01949cf70440b38e59a5020fd48c44 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:55.247925 21729 ts_tablet_manager.cc:1875] T dc5ac6b3423248c19209032b774885ca P 4da690c0fa13407a85c0689ce8de9ade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:55.247952 21724 ts_tablet_manager.cc:1875] T 1f01949cf70440b38e59a5020fd48c44 P dfa1152987644bd781b46c7e42a7a685: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:55.254405 21653 ts_tablet_manager.cc:1888] T dc5ac6b3423248c19209032b774885ca P fdc2ac372492454c85387589a91f63af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:53:55.255298 21724 ts_tablet_manager.cc:1888] T 1f01949cf70440b38e59a5020fd48c44 P dfa1152987644bd781b46c7e42a7a685: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:53:55.255450 21724 log.cc:1196] T 1f01949cf70440b38e59a5020fd48c44 P dfa1152987644bd781b46c7e42a7a685: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/wals/1f01949cf70440b38e59a5020fd48c44
I20240502 01:53:55.256186 17321 catalog_manager.cc:4830] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): tablet 1f01949cf70440b38e59a5020fd48c44 (table rebalance_test_table_11 [id=5e2315325f6c4e7f8834471d07555cf5]) successfully deleted
I20240502 01:53:55.257164 21729 ts_tablet_manager.cc:1888] T dc5ac6b3423248c19209032b774885ca P 4da690c0fa13407a85c0689ce8de9ade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:53:55.257344 21729 log.cc:1196] T dc5ac6b3423248c19209032b774885ca P 4da690c0fa13407a85c0689ce8de9ade: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/dc5ac6b3423248c19209032b774885ca
I20240502 01:53:55.257809 21729 ts_tablet_manager.cc:1909] T dc5ac6b3423248c19209032b774885ca P 4da690c0fa13407a85c0689ce8de9ade: Deleting consensus metadata
I20240502 01:53:55.258821 17321 catalog_manager.cc:4830] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): tablet dc5ac6b3423248c19209032b774885ca (table rebalancer_dynamic_table_new_name [id=4bc119cbe600465a9c4d50d5f21a0e9d]) successfully deleted
I20240502 01:53:55.262048 21730 tablet_replica.cc:330] T dc5ac6b3423248c19209032b774885ca P f19cc6cb1e76460aa6711e49a5775091: stopping tablet replica
I20240502 01:53:55.262665 21730 raft_consensus.cc:2229] T dc5ac6b3423248c19209032b774885ca P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:55.262965 21730 raft_consensus.cc:2258] T dc5ac6b3423248c19209032b774885ca P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:55.263705 21674 raft_consensus.cc:482] T 29d8cb76dc514e7b9002bd2291d8ee8d P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:55.263862 21674 raft_consensus.cc:504] T 29d8cb76dc514e7b9002bd2291d8ee8d P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:55.264621 21674 leader_election.cc:288] T 29d8cb76dc514e7b9002bd2291d8ee8d P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:53:55.265801 21653 log.cc:1196] T dc5ac6b3423248c19209032b774885ca P fdc2ac372492454c85387589a91f63af: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal/wals/dc5ac6b3423248c19209032b774885ca
I20240502 01:53:55.266283 21653 ts_tablet_manager.cc:1909] T dc5ac6b3423248c19209032b774885ca P fdc2ac372492454c85387589a91f63af: Deleting consensus metadata
I20240502 01:53:55.266564 20367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29d8cb76dc514e7b9002bd2291d8ee8d" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:53:55.266817 20367 raft_consensus.cc:2454] T 29d8cb76dc514e7b9002bd2291d8ee8d P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:53:55.267350 17672 leader_election.cc:302] T 29d8cb76dc514e7b9002bd2291d8ee8d P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:55.267887 21717 raft_consensus.cc:2781] T 29d8cb76dc514e7b9002bd2291d8ee8d P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:55.267974 21717 raft_consensus.cc:482] T 29d8cb76dc514e7b9002bd2291d8ee8d P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:55.267944 17322 catalog_manager.cc:4830] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): tablet dc5ac6b3423248c19209032b774885ca (table rebalancer_dynamic_table_new_name [id=4bc119cbe600465a9c4d50d5f21a0e9d]) successfully deleted
I20240502 01:53:55.268065 21717 raft_consensus.cc:3037] T 29d8cb76dc514e7b9002bd2291d8ee8d P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:55.270830 21717 raft_consensus.cc:504] T 29d8cb76dc514e7b9002bd2291d8ee8d P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:55.273698 20367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29d8cb76dc514e7b9002bd2291d8ee8d" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
I20240502 01:53:55.273926 20367 raft_consensus.cc:3037] T 29d8cb76dc514e7b9002bd2291d8ee8d P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:55.276676 20367 raft_consensus.cc:2454] T 29d8cb76dc514e7b9002bd2291d8ee8d P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:53:55.277231 21216 raft_consensus.cc:482] T 18549019e31b47d899d011cef0637221 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:55.277398 21216 raft_consensus.cc:504] T 18549019e31b47d899d011cef0637221 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:55.278112 21216 leader_election.cc:288] T 18549019e31b47d899d011cef0637221 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:53:55.278460 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18549019e31b47d899d011cef0637221" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:53:55.278730 17604 raft_consensus.cc:2379] T 18549019e31b47d899d011cef0637221 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2f306549407b4bb281ccf2994035e4f1 in current term 1: Already voted for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in this term.
I20240502 01:53:55.278698 20365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18549019e31b47d899d011cef0637221" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:53:55.279290 20433 leader_election.cc:302] T 18549019e31b47d899d011cef0637221 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1; no voters: 3fceccefba274b1dbbd8216af8b0bc29, 6a36c952c51542c0bf7ea3ec2857a9f9
I20240502 01:53:55.279620 21216 raft_consensus.cc:3037] T 18549019e31b47d899d011cef0637221 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:55.279944 17672 leader_election.cc:302] T 29d8cb76dc514e7b9002bd2291d8ee8d P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:55.280570 20497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29d8cb76dc514e7b9002bd2291d8ee8d" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
I20240502 01:53:55.280779 20497 raft_consensus.cc:2454] T 29d8cb76dc514e7b9002bd2291d8ee8d P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:53:55.281364 21674 raft_consensus.cc:2781] T 29d8cb76dc514e7b9002bd2291d8ee8d P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:55.281488 21674 raft_consensus.cc:686] T 29d8cb76dc514e7b9002bd2291d8ee8d P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Becoming Leader. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Running, Role: LEADER
I20240502 01:53:55.281736 21674 consensus_queue.cc:237] T 29d8cb76dc514e7b9002bd2291d8ee8d P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:55.282737 21216 raft_consensus.cc:2726] T 18549019e31b47d899d011cef0637221 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:53:55.283497 21717 leader_election.cc:288] T 29d8cb76dc514e7b9002bd2291d8ee8d P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Requested vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:53:55.283689 20497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29d8cb76dc514e7b9002bd2291d8ee8d" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
I20240502 01:53:55.283883 20497 raft_consensus.cc:3037] T 29d8cb76dc514e7b9002bd2291d8ee8d P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:55.286633 20497 raft_consensus.cc:2454] T 29d8cb76dc514e7b9002bd2291d8ee8d P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:53:55.288895 21730 ts_tablet_manager.cc:1875] T dc5ac6b3423248c19209032b774885ca P f19cc6cb1e76460aa6711e49a5775091: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:55.290866 20498 consensus_queue.cc:237] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:53:55.296080 17474 raft_consensus.cc:1261] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:55.296976 21216 consensus_queue.cc:1035] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:53:55.297452 20367 raft_consensus.cc:1261] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:55.299379 21216 consensus_queue.cc:1035] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:53:55.303025 17334 catalog_manager.cc:5484] T 29d8cb76dc514e7b9002bd2291d8ee8d P dfa1152987644bd781b46c7e42a7a685 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfa1152987644bd781b46c7e42a7a685 (127.11.42.67). New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:55.304065 21730 ts_tablet_manager.cc:1888] T dc5ac6b3423248c19209032b774885ca P f19cc6cb1e76460aa6711e49a5775091: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:53:55.304247 21730 log.cc:1196] T dc5ac6b3423248c19209032b774885ca P f19cc6cb1e76460aa6711e49a5775091: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal/wals/dc5ac6b3423248c19209032b774885ca
I20240502 01:53:55.304736 21730 ts_tablet_manager.cc:1909] T dc5ac6b3423248c19209032b774885ca P f19cc6cb1e76460aa6711e49a5775091: Deleting consensus metadata
W20240502 01:53:55.305138 20433 consensus_peers.cc:485] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 648b1cc4d8b54c5eb0948ca8c1984a1f. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:55.306246 17321 catalog_manager.cc:4830] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): tablet dc5ac6b3423248c19209032b774885ca (table rebalancer_dynamic_table_new_name [id=4bc119cbe600465a9c4d50d5f21a0e9d]) successfully deleted
I20240502 01:53:55.307860 21594 raft_consensus.cc:2932] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:53:55.312693 17334 catalog_manager.cc:5484] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:55.314936 20367 raft_consensus.cc:2932] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:53:55.326881 17474 raft_consensus.cc:2932] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:53:55.342344 17604 consensus_queue.cc:237] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } }
I20240502 01:53:55.344442 17736 raft_consensus.cc:1261] T f0b3862e504d4b43a60477e5efef29b1 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:55.345618 17866 raft_consensus.cc:1261] T f0b3862e504d4b43a60477e5efef29b1 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:55.345602 21629 consensus_queue.cc:1035] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:55.345703 17334 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:53:55.346665 21629 consensus_queue.cc:1035] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:55.353909 21543 raft_consensus.cc:2932] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:53:55.354300 20211 tablet_service.cc:1467] Processing CreateTablet for tablet 3435cabff9bb4bdfbb7b5fdda725f500 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a816e480a60a400a872452ea686fb049]), partition=
I20240502 01:53:55.354583 20347 tablet_service.cc:1467] Processing CreateTablet for tablet 3435cabff9bb4bdfbb7b5fdda725f500 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a816e480a60a400a872452ea686fb049]), partition=
I20240502 01:53:55.354677 20211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3435cabff9bb4bdfbb7b5fdda725f500. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:55.354918 20347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3435cabff9bb4bdfbb7b5fdda725f500. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:55.358079 17454 tablet_service.cc:1467] Processing CreateTablet for tablet 3435cabff9bb4bdfbb7b5fdda725f500 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a816e480a60a400a872452ea686fb049]), partition=
I20240502 01:53:55.358491 17454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3435cabff9bb4bdfbb7b5fdda725f500. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:55.359558 21750 tablet_bootstrap.cc:492] T 3435cabff9bb4bdfbb7b5fdda725f500 P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
W20240502 01:53:55.360312 17543 consensus_peers.cc:485] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f0b3862e504d4b43a60477e5efef29b1. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:55.361163 21681 tablet_bootstrap.cc:492] T 3435cabff9bb4bdfbb7b5fdda725f500 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:53:55.362493 21750 tablet_bootstrap.cc:654] T 3435cabff9bb4bdfbb7b5fdda725f500 P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:55.362613 21681 tablet_bootstrap.cc:654] T 3435cabff9bb4bdfbb7b5fdda725f500 P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:55.362437 17334 catalog_manager.cc:5484] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: config changed from index -1 to 4, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:55.363968 17866 raft_consensus.cc:2932] T f0b3862e504d4b43a60477e5efef29b1 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:53:55.364270 17736 raft_consensus.cc:2932] T f0b3862e504d4b43a60477e5efef29b1 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:53:55.370188 21750 tablet_bootstrap.cc:492] T 3435cabff9bb4bdfbb7b5fdda725f500 P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:53:55.370386 21750 ts_tablet_manager.cc:1367] T 3435cabff9bb4bdfbb7b5fdda725f500 P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20240502 01:53:55.371431 21750 raft_consensus.cc:348] T 3435cabff9bb4bdfbb7b5fdda725f500 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:55.371783 21750 raft_consensus.cc:374] T 3435cabff9bb4bdfbb7b5fdda725f500 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:55.371930 21750 raft_consensus.cc:729] T 3435cabff9bb4bdfbb7b5fdda725f500 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:53:55.372193 21750 consensus_queue.cc:260] T 3435cabff9bb4bdfbb7b5fdda725f500 P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:55.374296 21681 tablet_bootstrap.cc:492] T 3435cabff9bb4bdfbb7b5fdda725f500 P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:53:55.374512 21681 ts_tablet_manager.cc:1367] T 3435cabff9bb4bdfbb7b5fdda725f500 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240502 01:53:55.375077 21681 raft_consensus.cc:348] T 3435cabff9bb4bdfbb7b5fdda725f500 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:55.375245 21681 raft_consensus.cc:374] T 3435cabff9bb4bdfbb7b5fdda725f500 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:55.375327 21681 raft_consensus.cc:729] T 3435cabff9bb4bdfbb7b5fdda725f500 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:53:55.375638 21750 ts_tablet_manager.cc:1398] T 3435cabff9bb4bdfbb7b5fdda725f500 P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:53:55.375635 21681 consensus_queue.cc:260] T 3435cabff9bb4bdfbb7b5fdda725f500 P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:55.376289 21681 ts_tablet_manager.cc:1398] T 3435cabff9bb4bdfbb7b5fdda725f500 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:55.379523 21637 tablet_bootstrap.cc:492] T 3435cabff9bb4bdfbb7b5fdda725f500 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:55.381151 21637 tablet_bootstrap.cc:654] T 3435cabff9bb4bdfbb7b5fdda725f500 P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:55.386263 21637 tablet_bootstrap.cc:492] T 3435cabff9bb4bdfbb7b5fdda725f500 P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:53:55.386406 21637 ts_tablet_manager.cc:1367] T 3435cabff9bb4bdfbb7b5fdda725f500 P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20240502 01:53:55.386992 21637 raft_consensus.cc:348] T 3435cabff9bb4bdfbb7b5fdda725f500 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:55.387163 21637 raft_consensus.cc:374] T 3435cabff9bb4bdfbb7b5fdda725f500 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:55.387284 21637 raft_consensus.cc:729] T 3435cabff9bb4bdfbb7b5fdda725f500 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:53:55.387939 21637 consensus_queue.cc:260] T 3435cabff9bb4bdfbb7b5fdda725f500 P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:55.388638 21637 ts_tablet_manager.cc:1398] T 3435cabff9bb4bdfbb7b5fdda725f500 P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:53:55.468286 21801 ts_tablet_manager.cc:902] T 5e8e78bd417d46cdb166087dbfa86446 P 3fceccefba274b1dbbd8216af8b0bc29: Initiating tablet copy from peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:55.471652 21801 tablet_copy_client.cc:308] T 5e8e78bd417d46cdb166087dbfa86446 P 3fceccefba274b1dbbd8216af8b0bc29: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.65:38321
I20240502 01:53:55.472365 17496 tablet_copy_service.cc:140] P 4da690c0fa13407a85c0689ce8de9ade: Received BeginTabletCopySession request for tablet 5e8e78bd417d46cdb166087dbfa86446 from peer 3fceccefba274b1dbbd8216af8b0bc29 ({username='slave'} at 127.11.42.66:43759)
I20240502 01:53:55.472544 17496 tablet_copy_service.cc:161] P 4da690c0fa13407a85c0689ce8de9ade: Beginning new tablet copy session on tablet 5e8e78bd417d46cdb166087dbfa86446 from peer 3fceccefba274b1dbbd8216af8b0bc29 at {username='slave'} at 127.11.42.66:43759: session id = 3fceccefba274b1dbbd8216af8b0bc29-5e8e78bd417d46cdb166087dbfa86446
I20240502 01:53:55.475234 17496 tablet_copy_source_session.cc:205] T 5e8e78bd417d46cdb166087dbfa86446 P 4da690c0fa13407a85c0689ce8de9ade: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:55.476537 21801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e8e78bd417d46cdb166087dbfa86446. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:55.482856 21801 tablet_copy_client.cc:783] T 5e8e78bd417d46cdb166087dbfa86446 P 3fceccefba274b1dbbd8216af8b0bc29: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:55.483119 21801 tablet_copy_client.cc:647] T 5e8e78bd417d46cdb166087dbfa86446 P 3fceccefba274b1dbbd8216af8b0bc29: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:55.488797 21801 tablet_copy_client.cc:523] T 5e8e78bd417d46cdb166087dbfa86446 P 3fceccefba274b1dbbd8216af8b0bc29: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:55.491889 21801 tablet_bootstrap.cc:492] T 5e8e78bd417d46cdb166087dbfa86446 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:55.499629 21801 tablet_bootstrap.cc:492] T 5e8e78bd417d46cdb166087dbfa86446 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:55.499996 21801 tablet_bootstrap.cc:492] T 5e8e78bd417d46cdb166087dbfa86446 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap complete.
I20240502 01:53:55.500226 21801 ts_tablet_manager.cc:1367] T 5e8e78bd417d46cdb166087dbfa86446 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.006s
I20240502 01:53:55.500996 21801 raft_consensus.cc:348] T 5e8e78bd417d46cdb166087dbfa86446 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } }
I20240502 01:53:55.501226 21801 raft_consensus.cc:729] T 5e8e78bd417d46cdb166087dbfa86446 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: LEARNER
I20240502 01:53:55.501441 21801 consensus_queue.cc:260] T 5e8e78bd417d46cdb166087dbfa86446 P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } }
I20240502 01:53:55.501987 21801 ts_tablet_manager.cc:1398] T 5e8e78bd417d46cdb166087dbfa86446 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:53:55.502912 17496 tablet_copy_service.cc:342] P 4da690c0fa13407a85c0689ce8de9ade: Request end of tablet copy session 3fceccefba274b1dbbd8216af8b0bc29-5e8e78bd417d46cdb166087dbfa86446 received from {username='slave'} at 127.11.42.66:43759
I20240502 01:53:55.503046 17496 tablet_copy_service.cc:433] P 4da690c0fa13407a85c0689ce8de9ade: ending tablet copy session 3fceccefba274b1dbbd8216af8b0bc29-5e8e78bd417d46cdb166087dbfa86446 on tablet 5e8e78bd417d46cdb166087dbfa86446 with peer 3fceccefba274b1dbbd8216af8b0bc29
I20240502 01:53:55.561619 21845 ts_tablet_manager.cc:902] T 0618761d30f34cfe9650d13c9275cdcb P fdc2ac372492454c85387589a91f63af: Initiating tablet copy from peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:53:55.565964 21594 consensus_queue.cc:1035] T ee418fc4028a422eabac02ed63ea9047 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:55.600167 21216 consensus_queue.cc:1035] T ee418fc4028a422eabac02ed63ea9047 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:55.619524 21845 tablet_copy_client.cc:308] T 0618761d30f34cfe9650d13c9275cdcb P fdc2ac372492454c85387589a91f63af: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.65:38321
I20240502 01:53:55.620230 17496 tablet_copy_service.cc:140] P 4da690c0fa13407a85c0689ce8de9ade: Received BeginTabletCopySession request for tablet 0618761d30f34cfe9650d13c9275cdcb from peer fdc2ac372492454c85387589a91f63af ({username='slave'} at 127.11.42.69:58249)
I20240502 01:53:55.620404 17496 tablet_copy_service.cc:161] P 4da690c0fa13407a85c0689ce8de9ade: Beginning new tablet copy session on tablet 0618761d30f34cfe9650d13c9275cdcb from peer fdc2ac372492454c85387589a91f63af at {username='slave'} at 127.11.42.69:58249: session id = fdc2ac372492454c85387589a91f63af-0618761d30f34cfe9650d13c9275cdcb
I20240502 01:53:55.623502 17496 tablet_copy_source_session.cc:205] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:55.624912 21845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0618761d30f34cfe9650d13c9275cdcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:55.625502 21861 ts_tablet_manager.cc:902] T 5df83f83d2674155afba93c2d992af0a P f19cc6cb1e76460aa6711e49a5775091: Initiating tablet copy from peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:53:55.629200 21009 consensus_queue.cc:1035] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:55.642346 21622 raft_consensus.cc:482] T 3435cabff9bb4bdfbb7b5fdda725f500 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:55.642504 21622 raft_consensus.cc:504] T 3435cabff9bb4bdfbb7b5fdda725f500 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:55.643163 21622 leader_election.cc:288] T 3435cabff9bb4bdfbb7b5fdda725f500 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:53:55.643571 20231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3435cabff9bb4bdfbb7b5fdda725f500" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:53:55.643791 20231 raft_consensus.cc:2454] T 3435cabff9bb4bdfbb7b5fdda725f500 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 0.
I20240502 01:53:55.644412 17412 leader_election.cc:302] T 3435cabff9bb4bdfbb7b5fdda725f500 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:53:55.644475 20365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3435cabff9bb4bdfbb7b5fdda725f500" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:53:55.644704 20365 raft_consensus.cc:2454] T 3435cabff9bb4bdfbb7b5fdda725f500 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 0.
I20240502 01:53:55.644816 21622 raft_consensus.cc:2781] T 3435cabff9bb4bdfbb7b5fdda725f500 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:55.644927 21622 raft_consensus.cc:482] T 3435cabff9bb4bdfbb7b5fdda725f500 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:55.645053 21622 raft_consensus.cc:3037] T 3435cabff9bb4bdfbb7b5fdda725f500 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:55.645823 21579 consensus_queue.cc:1035] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:55.647619 21622 raft_consensus.cc:504] T 3435cabff9bb4bdfbb7b5fdda725f500 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:55.650218 20231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3435cabff9bb4bdfbb7b5fdda725f500" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:53:55.650420 20231 raft_consensus.cc:3037] T 3435cabff9bb4bdfbb7b5fdda725f500 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:55.653319 21622 leader_election.cc:288] T 3435cabff9bb4bdfbb7b5fdda725f500 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Requested vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:53:55.653357 20231 raft_consensus.cc:2454] T 3435cabff9bb4bdfbb7b5fdda725f500 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
I20240502 01:53:55.653769 20365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3435cabff9bb4bdfbb7b5fdda725f500" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
I20240502 01:53:55.653829 17412 leader_election.cc:302] T 3435cabff9bb4bdfbb7b5fdda725f500 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:53:55.654026 20365 raft_consensus.cc:3037] T 3435cabff9bb4bdfbb7b5fdda725f500 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:55.654263 21622 raft_consensus.cc:2781] T 3435cabff9bb4bdfbb7b5fdda725f500 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:55.656893 20365 raft_consensus.cc:2454] T 3435cabff9bb4bdfbb7b5fdda725f500 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
I20240502 01:53:55.665408 17604 consensus_queue.cc:237] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } }
I20240502 01:53:55.666486 21622 raft_consensus.cc:686] T 3435cabff9bb4bdfbb7b5fdda725f500 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Becoming Leader. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Running, Role: LEADER
I20240502 01:53:55.666898 21622 consensus_queue.cc:237] T 3435cabff9bb4bdfbb7b5fdda725f500 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:55.674796 17327 catalog_manager.cc:5484] T 3435cabff9bb4bdfbb7b5fdda725f500 P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: term changed from 0 to 1, leader changed from <none> to 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65). New cstate: current_term: 1 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:55.675926 21845 tablet_copy_client.cc:783] T 0618761d30f34cfe9650d13c9275cdcb P fdc2ac372492454c85387589a91f63af: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:55.676190 21845 tablet_copy_client.cc:647] T 0618761d30f34cfe9650d13c9275cdcb P fdc2ac372492454c85387589a91f63af: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:55.678030 21845 tablet_copy_client.cc:523] T 0618761d30f34cfe9650d13c9275cdcb P fdc2ac372492454c85387589a91f63af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:55.678197 21861 tablet_copy_client.cc:308] T 5df83f83d2674155afba93c2d992af0a P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.67:36115
I20240502 01:53:55.678974 17757 tablet_copy_service.cc:140] P dfa1152987644bd781b46c7e42a7a685: Received BeginTabletCopySession request for tablet 5df83f83d2674155afba93c2d992af0a from peer f19cc6cb1e76460aa6711e49a5775091 ({username='slave'} at 127.11.42.72:53665)
I20240502 01:53:55.679173 17757 tablet_copy_service.cc:161] P dfa1152987644bd781b46c7e42a7a685: Beginning new tablet copy session on tablet 5df83f83d2674155afba93c2d992af0a from peer f19cc6cb1e76460aa6711e49a5775091 at {username='slave'} at 127.11.42.72:53665: session id = f19cc6cb1e76460aa6711e49a5775091-5df83f83d2674155afba93c2d992af0a
I20240502 01:53:55.682490 17757 tablet_copy_source_session.cc:205] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:55.684537 21674 consensus_queue.cc:1035] T 29d8cb76dc514e7b9002bd2291d8ee8d P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:55.686791 21861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5df83f83d2674155afba93c2d992af0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:55.688217 21845 tablet_bootstrap.cc:492] T 0618761d30f34cfe9650d13c9275cdcb P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:53:55.691458 17865 raft_consensus.cc:1261] T 1fdd104f1a28446cbc25d4ca16d03d7d P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:55.692009 17736 raft_consensus.cc:1261] T 1fdd104f1a28446cbc25d4ca16d03d7d P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:55.692414 21634 consensus_queue.cc:1035] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:55.692714 21629 consensus_queue.cc:1035] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:53:55.693037 17542 consensus_peers.cc:485] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): Couldn't send request to peer 6a36c952c51542c0bf7ea3ec2857a9f9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1fdd104f1a28446cbc25d4ca16d03d7d. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:55.695420 21717 consensus_queue.cc:1035] T 29d8cb76dc514e7b9002bd2291d8ee8d P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:55.695993 21861 tablet_copy_client.cc:783] T 5df83f83d2674155afba93c2d992af0a P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:55.696218 21861 tablet_copy_client.cc:647] T 5df83f83d2674155afba93c2d992af0a P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:55.700590 21861 tablet_copy_client.cc:523] T 5df83f83d2674155afba93c2d992af0a P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:55.703910 21861 tablet_bootstrap.cc:492] T 5df83f83d2674155afba93c2d992af0a P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:53:55.716516 21634 raft_consensus.cc:2932] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } } }
I20240502 01:53:55.721818 17736 raft_consensus.cc:2932] T 1fdd104f1a28446cbc25d4ca16d03d7d P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } } }
I20240502 01:53:55.737864 17327 catalog_manager.cc:5484] T 1fdd104f1a28446cbc25d4ca16d03d7d P dfa1152987644bd781b46c7e42a7a685 reported cstate change: config changed from index -1 to 4, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } } }
I20240502 01:53:55.757710 17866 raft_consensus.cc:2932] T 1fdd104f1a28446cbc25d4ca16d03d7d P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } } }
I20240502 01:53:55.788420 21861 tablet_bootstrap.cc:492] T 5df83f83d2674155afba93c2d992af0a P f19cc6cb1e76460aa6711e49a5775091: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:55.788897 21861 tablet_bootstrap.cc:492] T 5df83f83d2674155afba93c2d992af0a P f19cc6cb1e76460aa6711e49a5775091: Bootstrap complete.
I20240502 01:53:55.789206 21861 ts_tablet_manager.cc:1367] T 5df83f83d2674155afba93c2d992af0a P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.085s user 0.038s sys 0.019s
I20240502 01:53:55.790122 21861 raft_consensus.cc:348] T 5df83f83d2674155afba93c2d992af0a P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:53:55.790374 21861 raft_consensus.cc:729] T 5df83f83d2674155afba93c2d992af0a P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: LEARNER
I20240502 01:53:55.790557 21861 consensus_queue.cc:260] T 5df83f83d2674155afba93c2d992af0a P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:53:55.791232 21861 ts_tablet_manager.cc:1398] T 5df83f83d2674155afba93c2d992af0a P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:53:55.792022 17757 tablet_copy_service.cc:342] P dfa1152987644bd781b46c7e42a7a685: Request end of tablet copy session f19cc6cb1e76460aa6711e49a5775091-5df83f83d2674155afba93c2d992af0a received from {username='slave'} at 127.11.42.72:53665
I20240502 01:53:55.792143 17757 tablet_copy_service.cc:433] P dfa1152987644bd781b46c7e42a7a685: ending tablet copy session f19cc6cb1e76460aa6711e49a5775091-5df83f83d2674155afba93c2d992af0a on tablet 5df83f83d2674155afba93c2d992af0a with peer f19cc6cb1e76460aa6711e49a5775091
I20240502 01:53:55.794389 21892 ts_tablet_manager.cc:902] T 648b1cc4d8b54c5eb0948ca8c1984a1f P dfa1152987644bd781b46c7e42a7a685: Initiating tablet copy from peer 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:53:55.803655 17604 consensus_queue.cc:237] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
W20240502 01:53:55.814092 17542 consensus_peers.cc:485] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4aefc61cb95f4e949e7a23a1c677749e. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:55.815893 17736 raft_consensus.cc:1261] T 4aefc61cb95f4e949e7a23a1c677749e P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:55.816524 21629 consensus_queue.cc:1035] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:55.816779 17865 raft_consensus.cc:1261] T 4aefc61cb95f4e949e7a23a1c677749e P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:55.817327 21543 consensus_queue.cc:1035] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:55.820147 21543 raft_consensus.cc:2932] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:53:55.823431 21845 tablet_bootstrap.cc:492] T 0618761d30f34cfe9650d13c9275cdcb P fdc2ac372492454c85387589a91f63af: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:55.823228 17736 raft_consensus.cc:2932] T 4aefc61cb95f4e949e7a23a1c677749e P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:53:55.828123 17335 catalog_manager.cc:5484] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: config changed from index -1 to 4, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:55.828987 21845 tablet_bootstrap.cc:492] T 0618761d30f34cfe9650d13c9275cdcb P fdc2ac372492454c85387589a91f63af: Bootstrap complete.
I20240502 01:53:55.829370 21845 ts_tablet_manager.cc:1367] T 0618761d30f34cfe9650d13c9275cdcb P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.141s user 0.040s sys 0.013s
I20240502 01:53:55.830170 21845 raft_consensus.cc:348] T 0618761d30f34cfe9650d13c9275cdcb P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } }
I20240502 01:53:55.830502 21845 raft_consensus.cc:729] T 0618761d30f34cfe9650d13c9275cdcb P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: LEARNER
I20240502 01:53:55.830775 21845 consensus_queue.cc:260] T 0618761d30f34cfe9650d13c9275cdcb P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } }
I20240502 01:53:55.832208 21845 ts_tablet_manager.cc:1398] T 0618761d30f34cfe9650d13c9275cdcb P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:53:55.832301 17334 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:53:55.833055 17496 tablet_copy_service.cc:342] P 4da690c0fa13407a85c0689ce8de9ade: Request end of tablet copy session fdc2ac372492454c85387589a91f63af-0618761d30f34cfe9650d13c9275cdcb received from {username='slave'} at 127.11.42.69:58249
I20240502 01:53:55.833184 17496 tablet_copy_service.cc:433] P 4da690c0fa13407a85c0689ce8de9ade: ending tablet copy session fdc2ac372492454c85387589a91f63af-0618761d30f34cfe9650d13c9275cdcb on tablet 0618761d30f34cfe9650d13c9275cdcb with peer fdc2ac372492454c85387589a91f63af
I20240502 01:53:55.843425 20365 raft_consensus.cc:1261] T 3435cabff9bb4bdfbb7b5fdda725f500 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:55.843729 17866 raft_consensus.cc:2932] T 4aefc61cb95f4e949e7a23a1c677749e P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:53:55.844103 21622 consensus_queue.cc:1035] T 3435cabff9bb4bdfbb7b5fdda725f500 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:55.845757 21892 tablet_copy_client.cc:308] T 648b1cc4d8b54c5eb0948ca8c1984a1f P dfa1152987644bd781b46c7e42a7a685: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.71:38623
I20240502 01:53:55.846908 20518 tablet_copy_service.cc:140] P 2f306549407b4bb281ccf2994035e4f1: Received BeginTabletCopySession request for tablet 648b1cc4d8b54c5eb0948ca8c1984a1f from peer dfa1152987644bd781b46c7e42a7a685 ({username='slave'} at 127.11.42.67:59949)
I20240502 01:53:55.847097 20518 tablet_copy_service.cc:161] P 2f306549407b4bb281ccf2994035e4f1: Beginning new tablet copy session on tablet 648b1cc4d8b54c5eb0948ca8c1984a1f from peer dfa1152987644bd781b46c7e42a7a685 at {username='slave'} at 127.11.42.67:59949: session id = dfa1152987644bd781b46c7e42a7a685-648b1cc4d8b54c5eb0948ca8c1984a1f
I20240502 01:53:55.850054 20231 raft_consensus.cc:1261] T 3435cabff9bb4bdfbb7b5fdda725f500 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:55.850695 21622 consensus_queue.cc:1035] T 3435cabff9bb4bdfbb7b5fdda725f500 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:55.854562 20518 tablet_copy_source_session.cc:205] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:55.857704 21892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 648b1cc4d8b54c5eb0948ca8c1984a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:55.860975 17327 catalog_manager.cc:5576] TS 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155) does not have the latest schema for tablet 3435cabff9bb4bdfbb7b5fdda725f500 (table rebalancer_dynamic_table [id=a816e480a60a400a872452ea686fb049]). Expected version 1 got 0
I20240502 01:53:55.864827 17327 catalog_manager.cc:5576] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339) does not have the latest schema for tablet 3435cabff9bb4bdfbb7b5fdda725f500 (table rebalancer_dynamic_table [id=a816e480a60a400a872452ea686fb049]). Expected version 1 got 0
I20240502 01:53:55.864868 21892 tablet_copy_client.cc:783] T 648b1cc4d8b54c5eb0948ca8c1984a1f P dfa1152987644bd781b46c7e42a7a685: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:55.865134 21892 tablet_copy_client.cc:647] T 648b1cc4d8b54c5eb0948ca8c1984a1f P dfa1152987644bd781b46c7e42a7a685: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:55.866868 17332 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_extra_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:55.869652 21910 tablet.cc:1722] T 3435cabff9bb4bdfbb7b5fdda725f500 P fdc2ac372492454c85387589a91f63af: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:53:55.870887 21910 tablet.cc:1620] T 3435cabff9bb4bdfbb7b5fdda725f500 P fdc2ac372492454c85387589a91f63af: MemRowSet was empty: no flush needed.
I20240502 01:53:55.871276 17327 catalog_manager.cc:5576] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339) does not have the latest schema for tablet 3435cabff9bb4bdfbb7b5fdda725f500 (table rebalancer_dynamic_table [id=a816e480a60a400a872452ea686fb049]). Expected version 1 got 0
I20240502 01:53:55.872730 21892 tablet_copy_client.cc:523] T 648b1cc4d8b54c5eb0948ca8c1984a1f P dfa1152987644bd781b46c7e42a7a685: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:55.877738 20211 tablet_service.cc:1467] Processing CreateTablet for tablet 3ff84ce60ac1438d92b2c944bb27aabb (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=61898cfacac64133a52ca6484799789a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:55.878149 20211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ff84ce60ac1438d92b2c944bb27aabb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:55.878273 20347 tablet_service.cc:1467] Processing CreateTablet for tablet fac0ba26fabd4643bc38b7559faa471c (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=61898cfacac64133a52ca6484799789a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:55.878684 20347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fac0ba26fabd4643bc38b7559faa471c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:55.880239 21892 tablet_bootstrap.cc:492] T 648b1cc4d8b54c5eb0948ca8c1984a1f P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:55.881598 20610 tablet_service.cc:1467] Processing CreateTablet for tablet fac0ba26fabd4643bc38b7559faa471c (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=61898cfacac64133a52ca6484799789a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:55.882011 20610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fac0ba26fabd4643bc38b7559faa471c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:55.883239 20609 tablet_service.cc:1467] Processing CreateTablet for tablet fbd94103cae04473ba81709da39d7ed8 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=61898cfacac64133a52ca6484799789a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:55.883585 20609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbd94103cae04473ba81709da39d7ed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:55.885210 20608 tablet_service.cc:1467] Processing CreateTablet for tablet 3ff84ce60ac1438d92b2c944bb27aabb (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=61898cfacac64133a52ca6484799789a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:55.885597 20608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ff84ce60ac1438d92b2c944bb27aabb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:55.890085 20477 tablet_service.cc:1467] Processing CreateTablet for tablet fac0ba26fabd4643bc38b7559faa471c (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=61898cfacac64133a52ca6484799789a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:55.890116 20476 tablet_service.cc:1467] Processing CreateTablet for tablet 3ff84ce60ac1438d92b2c944bb27aabb (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=61898cfacac64133a52ca6484799789a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:55.890522 20477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fac0ba26fabd4643bc38b7559faa471c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:55.890688 20212 tablet_service.cc:1467] Processing CreateTablet for tablet fbd94103cae04473ba81709da39d7ed8 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=61898cfacac64133a52ca6484799789a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:55.891080 20212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbd94103cae04473ba81709da39d7ed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:55.893299 20478 tablet_service.cc:1467] Processing CreateTablet for tablet fbd94103cae04473ba81709da39d7ed8 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=61898cfacac64133a52ca6484799789a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:55.893682 20478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbd94103cae04473ba81709da39d7ed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:55.894992 20476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ff84ce60ac1438d92b2c944bb27aabb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:55.921133 17327 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=a816e480a60a400a872452ea686fb049] alter complete (version 1)
I20240502 01:53:55.922269 21845 ts_tablet_manager.cc:902] T f0b3862e504d4b43a60477e5efef29b1 P fdc2ac372492454c85387589a91f63af: Initiating tablet copy from peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:55.925125 21915 tablet_bootstrap.cc:492] T fac0ba26fabd4643bc38b7559faa471c P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:53:55.928879 21915 tablet_bootstrap.cc:654] T fac0ba26fabd4643bc38b7559faa471c P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:55.930059 21912 tablet_bootstrap.cc:492] T 3ff84ce60ac1438d92b2c944bb27aabb P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:53:55.931469 21914 tablet_bootstrap.cc:492] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:53:55.932933 21912 tablet_bootstrap.cc:654] T 3ff84ce60ac1438d92b2c944bb27aabb P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:55.935622 21911 tablet_bootstrap.cc:492] T fac0ba26fabd4643bc38b7559faa471c P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:53:55.936707 21914 tablet_bootstrap.cc:654] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:55.939000 21911 tablet_bootstrap.cc:654] T fac0ba26fabd4643bc38b7559faa471c P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:55.942917 21914 tablet_bootstrap.cc:492] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:53:55.943401 21914 ts_tablet_manager.cc:1367] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240502 01:53:55.944130 21915 tablet_bootstrap.cc:492] T fac0ba26fabd4643bc38b7559faa471c P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:53:55.944451 21914 raft_consensus.cc:348] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:55.944607 21915 ts_tablet_manager.cc:1367] T fac0ba26fabd4643bc38b7559faa471c P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.002s
I20240502 01:53:55.945044 21907 tablet.cc:1722] T 3435cabff9bb4bdfbb7b5fdda725f500 P 4da690c0fa13407a85c0689ce8de9ade: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:53:55.945550 21907 tablet.cc:1620] T 3435cabff9bb4bdfbb7b5fdda725f500 P 4da690c0fa13407a85c0689ce8de9ade: MemRowSet was empty: no flush needed.
I20240502 01:53:55.946070 21912 tablet_bootstrap.cc:492] T 3ff84ce60ac1438d92b2c944bb27aabb P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:53:55.946273 21912 ts_tablet_manager.cc:1367] T 3ff84ce60ac1438d92b2c944bb27aabb P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.007s
I20240502 01:53:55.948145 17332 catalog_manager.cc:5576] TS 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155) does not have the latest schema for tablet 3435cabff9bb4bdfbb7b5fdda725f500 (table rebalancer_dynamic_table [id=a816e480a60a400a872452ea686fb049]). Expected version 1 got 0
I20240502 01:53:55.943136 21909 tablet.cc:1722] T 3435cabff9bb4bdfbb7b5fdda725f500 P 6a36c952c51542c0bf7ea3ec2857a9f9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:53:55.949677 21912 raft_consensus.cc:348] T 3ff84ce60ac1438d92b2c944bb27aabb P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:55.949967 21912 raft_consensus.cc:374] T 3ff84ce60ac1438d92b2c944bb27aabb P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:55.950145 21912 raft_consensus.cc:729] T 3ff84ce60ac1438d92b2c944bb27aabb P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:53:55.950217 21909 tablet.cc:1620] T 3435cabff9bb4bdfbb7b5fdda725f500 P 6a36c952c51542c0bf7ea3ec2857a9f9: MemRowSet was empty: no flush needed.
I20240502 01:53:55.950512 21912 consensus_queue.cc:260] T 3ff84ce60ac1438d92b2c944bb27aabb P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:55.951102 21912 ts_tablet_manager.cc:1398] T 3ff84ce60ac1438d92b2c944bb27aabb P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240502 01:53:55.951401 21912 tablet_bootstrap.cc:492] T fbd94103cae04473ba81709da39d7ed8 P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:53:55.953248 21892 tablet_bootstrap.cc:492] T 648b1cc4d8b54c5eb0948ca8c1984a1f P dfa1152987644bd781b46c7e42a7a685: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:55.953763 21892 tablet_bootstrap.cc:492] T 648b1cc4d8b54c5eb0948ca8c1984a1f P dfa1152987644bd781b46c7e42a7a685: Bootstrap complete.
I20240502 01:53:55.954041 21892 ts_tablet_manager.cc:1367] T 648b1cc4d8b54c5eb0948ca8c1984a1f P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.074s user 0.007s sys 0.006s
I20240502 01:53:55.954855 21892 raft_consensus.cc:348] T 648b1cc4d8b54c5eb0948ca8c1984a1f P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:53:55.955114 21892 raft_consensus.cc:729] T 648b1cc4d8b54c5eb0948ca8c1984a1f P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: LEARNER
I20240502 01:53:55.955382 21892 consensus_queue.cc:260] T 648b1cc4d8b54c5eb0948ca8c1984a1f P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:53:55.955775 21915 raft_consensus.cc:348] T fac0ba26fabd4643bc38b7559faa471c P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:55.956005 21915 raft_consensus.cc:374] T fac0ba26fabd4643bc38b7559faa471c P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:55.956138 21915 raft_consensus.cc:729] T fac0ba26fabd4643bc38b7559faa471c P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:53:55.956411 21915 consensus_queue.cc:260] T fac0ba26fabd4643bc38b7559faa471c P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:55.953699 20498 consensus_queue.cc:237] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } }
I20240502 01:53:55.957496 21915 ts_tablet_manager.cc:1398] T fac0ba26fabd4643bc38b7559faa471c P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20240502 01:53:55.957799 21915 tablet_bootstrap.cc:492] T 3ff84ce60ac1438d92b2c944bb27aabb P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:53:55.958606 21914 raft_consensus.cc:374] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:55.958786 21914 raft_consensus.cc:729] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:53:55.959411 21915 tablet_bootstrap.cc:654] T 3ff84ce60ac1438d92b2c944bb27aabb P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:55.959635 21914 consensus_queue.cc:260] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:55.960253 21912 tablet_bootstrap.cc:654] T fbd94103cae04473ba81709da39d7ed8 P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:55.961618 21892 ts_tablet_manager.cc:1398] T 648b1cc4d8b54c5eb0948ca8c1984a1f P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240502 01:53:55.961999 20628 raft_consensus.cc:1261] T 617443200e2c47b69d63ef056294f40c P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:55.962420 20518 tablet_copy_service.cc:342] P 2f306549407b4bb281ccf2994035e4f1: Request end of tablet copy session dfa1152987644bd781b46c7e42a7a685-648b1cc4d8b54c5eb0948ca8c1984a1f received from {username='slave'} at 127.11.42.67:59949
I20240502 01:53:55.962599 20518 tablet_copy_service.cc:433] P 2f306549407b4bb281ccf2994035e4f1: ending tablet copy session dfa1152987644bd781b46c7e42a7a685-648b1cc4d8b54c5eb0948ca8c1984a1f on tablet 648b1cc4d8b54c5eb0948ca8c1984a1f with peer dfa1152987644bd781b46c7e42a7a685
W20240502 01:53:55.965581 20433 consensus_peers.cc:485] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1 -> Peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Couldn't send request to peer 4da690c0fa13407a85c0689ce8de9ade. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 617443200e2c47b69d63ef056294f40c. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:55.968775 21914 ts_tablet_manager.cc:1398] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.025s user 0.003s sys 0.000s
I20240502 01:53:55.969135 21914 tablet_bootstrap.cc:492] T fbd94103cae04473ba81709da39d7ed8 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:53:55.970758 20231 raft_consensus.cc:1261] T 617443200e2c47b69d63ef056294f40c P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:55.971155 21914 tablet_bootstrap.cc:654] T fbd94103cae04473ba81709da39d7ed8 P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:55.970311 21924 consensus_queue.cc:1035] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 01:53:55.971617 21594 consensus_queue.cc:1035] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:53:55.973088 21914 tablet_bootstrap.cc:492] T fbd94103cae04473ba81709da39d7ed8 P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:53:55.973347 21914 ts_tablet_manager.cc:1367] T fbd94103cae04473ba81709da39d7ed8 P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:53:55.974157 21914 raft_consensus.cc:348] T fbd94103cae04473ba81709da39d7ed8 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:55.975013 21914 raft_consensus.cc:374] T fbd94103cae04473ba81709da39d7ed8 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:55.975143 21914 raft_consensus.cc:729] T fbd94103cae04473ba81709da39d7ed8 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:53:55.975451 21914 consensus_queue.cc:260] T fbd94103cae04473ba81709da39d7ed8 P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:55.976284 21914 ts_tablet_manager.cc:1398] T fbd94103cae04473ba81709da39d7ed8 P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:55.976874 21914 tablet_bootstrap.cc:492] T 3ff84ce60ac1438d92b2c944bb27aabb P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:53:55.978617 21914 tablet_bootstrap.cc:654] T 3ff84ce60ac1438d92b2c944bb27aabb P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:55.982329 21915 tablet_bootstrap.cc:492] T 3ff84ce60ac1438d92b2c944bb27aabb P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:53:55.982942 21915 ts_tablet_manager.cc:1367] T 3ff84ce60ac1438d92b2c944bb27aabb P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.004s
I20240502 01:53:55.984015 21915 raft_consensus.cc:348] T 3ff84ce60ac1438d92b2c944bb27aabb P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:55.985355 21915 raft_consensus.cc:374] T 3ff84ce60ac1438d92b2c944bb27aabb P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:55.985617 21915 raft_consensus.cc:729] T 3ff84ce60ac1438d92b2c944bb27aabb P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:53:55.986119 21915 consensus_queue.cc:260] T 3ff84ce60ac1438d92b2c944bb27aabb P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:55.987469 21915 ts_tablet_manager.cc:1398] T 3ff84ce60ac1438d92b2c944bb27aabb P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 01:53:55.988207 21915 tablet_bootstrap.cc:492] T fbd94103cae04473ba81709da39d7ed8 P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:53:55.991185 17604 raft_consensus.cc:1203] T 5e8e78bd417d46cdb166087dbfa86446 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:53:55.992163 21429 raft_consensus.cc:482] T 3ff84ce60ac1438d92b2c944bb27aabb P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:55.992426 21429 raft_consensus.cc:504] T 3ff84ce60ac1438d92b2c944bb27aabb P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:55.993139 21429 leader_election.cc:288] T 3ff84ce60ac1438d92b2c944bb27aabb P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:53:55.994098 20498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff84ce60ac1438d92b2c944bb27aabb" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
I20240502 01:53:55.994746 20233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff84ce60ac1438d92b2c944bb27aabb" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:53:55.994978 20233 raft_consensus.cc:2454] T 3ff84ce60ac1438d92b2c944bb27aabb P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 0.
I20240502 01:53:55.995507 20565 leader_election.cc:302] T 3ff84ce60ac1438d92b2c944bb27aabb P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f19cc6cb1e76460aa6711e49a5775091, fdc2ac372492454c85387589a91f63af; no voters:
W20240502 01:53:55.995796 20565 leader_election.cc:341] T 3ff84ce60ac1438d92b2c944bb27aabb P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:53:55.996208 21429 raft_consensus.cc:2781] T 3ff84ce60ac1438d92b2c944bb27aabb P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:55.996299 21429 raft_consensus.cc:482] T 3ff84ce60ac1438d92b2c944bb27aabb P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:55.992041 21915 tablet_bootstrap.cc:654] T fbd94103cae04473ba81709da39d7ed8 P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:55.996417 21429 raft_consensus.cc:3037] T 3ff84ce60ac1438d92b2c944bb27aabb P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:55.999354 21429 raft_consensus.cc:504] T 3ff84ce60ac1438d92b2c944bb27aabb P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:56.000437 20233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff84ce60ac1438d92b2c944bb27aabb" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:53:56.000634 20233 raft_consensus.cc:3037] T 3ff84ce60ac1438d92b2c944bb27aabb P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:56.003535 20233 raft_consensus.cc:2454] T 3ff84ce60ac1438d92b2c944bb27aabb P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 1.
I20240502 01:53:56.004068 20565 leader_election.cc:302] T 3ff84ce60ac1438d92b2c944bb27aabb P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f19cc6cb1e76460aa6711e49a5775091, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:53:56.020488 21429 leader_election.cc:288] T 3ff84ce60ac1438d92b2c944bb27aabb P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 election: Requested vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:53:56.020776 21429 raft_consensus.cc:2781] T 3ff84ce60ac1438d92b2c944bb27aabb P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:56.020980 21429 raft_consensus.cc:686] T 3ff84ce60ac1438d92b2c944bb27aabb P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Becoming Leader. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Running, Role: LEADER
I20240502 01:53:56.021322 21429 consensus_queue.cc:237] T 3ff84ce60ac1438d92b2c944bb27aabb P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:56.024183 17332 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:53:56.029450 21924 raft_consensus.cc:482] T fbd94103cae04473ba81709da39d7ed8 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:56.029639 21924 raft_consensus.cc:504] T fbd94103cae04473ba81709da39d7ed8 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:56.030324 21924 leader_election.cc:288] T fbd94103cae04473ba81709da39d7ed8 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:53:56.030999 20629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbd94103cae04473ba81709da39d7ed8" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091" is_pre_election: true
I20240502 01:53:56.031239 20233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbd94103cae04473ba81709da39d7ed8" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
W20240502 01:53:56.031662 20433 leader_election.cc:341] T fbd94103cae04473ba81709da39d7ed8 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Illegal state: must be running to vote when last-logged opid is not known
W20240502 01:53:56.033563 20434 leader_election.cc:341] T fbd94103cae04473ba81709da39d7ed8 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:53:56.034025 20434 leader_election.cc:302] T fbd94103cae04473ba81709da39d7ed8 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1; no voters: f19cc6cb1e76460aa6711e49a5775091, fdc2ac372492454c85387589a91f63af
I20240502 01:53:56.034358 21924 raft_consensus.cc:2726] T fbd94103cae04473ba81709da39d7ed8 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 01:53:56.090216 20565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.42.69:34339, user_credentials={real_user=slave}} blocked reactor thread for 86155.9us
I20240502 01:53:56.091066 20498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff84ce60ac1438d92b2c944bb27aabb" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
I20240502 01:53:56.093727 21924 raft_consensus.cc:482] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:56.093907 21924 raft_consensus.cc:504] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:56.094543 21924 leader_election.cc:288] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:53:56.095464 20629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fac0ba26fabd4643bc38b7559faa471c" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091" is_pre_election: true
I20240502 01:53:56.095728 20629 raft_consensus.cc:2454] T fac0ba26fabd4643bc38b7559faa471c P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 0.
I20240502 01:53:56.096292 20433 leader_election.cc:302] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:53:56.096661 21924 raft_consensus.cc:2781] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:56.096753 21924 raft_consensus.cc:482] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:56.096845 21924 raft_consensus.cc:3037] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:56.099861 21924 raft_consensus.cc:504] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:56.100937 20629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fac0ba26fabd4643bc38b7559faa471c" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091"
I20240502 01:53:56.101183 20629 raft_consensus.cc:3037] T fac0ba26fabd4643bc38b7559faa471c P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:56.104101 20629 raft_consensus.cc:2454] T fac0ba26fabd4643bc38b7559faa471c P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 1.
I20240502 01:53:56.104715 20433 leader_election.cc:302] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:53:56.104997 21697 raft_consensus.cc:2781] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:56.105150 21697 raft_consensus.cc:686] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Running, Role: LEADER
I20240502 01:53:56.105433 21697 consensus_queue.cc:237] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:56.107679 20367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fac0ba26fabd4643bc38b7559faa471c" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
I20240502 01:53:56.107982 21924 leader_election.cc:288] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 election: Requested vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
W20240502 01:53:56.108281 20433 leader_election.cc:341] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:53:56.114764 20365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fac0ba26fabd4643bc38b7559faa471c" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:53:56.126688 21594 raft_consensus.cc:2932] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } } }
I20240502 01:53:56.127605 20233 raft_consensus.cc:2932] T 617443200e2c47b69d63ef056294f40c P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } } }
W20240502 01:53:56.129997 20433 leader_election.cc:341] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:53:56.137609 17334 catalog_manager.cc:5484] T 617443200e2c47b69d63ef056294f40c P fdc2ac372492454c85387589a91f63af reported cstate change: config changed from index -1 to 2, NON_VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) added. New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } } }
I20240502 01:53:56.142030 21845 tablet_copy_client.cc:308] T f0b3862e504d4b43a60477e5efef29b1 P fdc2ac372492454c85387589a91f63af: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.66:38531
W20240502 01:53:56.144435 20565 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.11.42.126:39453, user_credentials={real_user=slave}} blocked reactor thread for 50791.7us
I20240502 01:53:56.148042 17334 catalog_manager.cc:5484] T 3ff84ce60ac1438d92b2c944bb27aabb P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: term changed from 0 to 1, leader changed from <none> to f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72). New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: HEALTHY } } }
I20240502 01:53:56.154770 17627 tablet_copy_service.cc:140] P 3fceccefba274b1dbbd8216af8b0bc29: Received BeginTabletCopySession request for tablet f0b3862e504d4b43a60477e5efef29b1 from peer fdc2ac372492454c85387589a91f63af ({username='slave'} at 127.11.42.69:59447)
I20240502 01:53:56.154974 17627 tablet_copy_service.cc:161] P 3fceccefba274b1dbbd8216af8b0bc29: Beginning new tablet copy session on tablet f0b3862e504d4b43a60477e5efef29b1 from peer fdc2ac372492454c85387589a91f63af at {username='slave'} at 127.11.42.69:59447: session id = fdc2ac372492454c85387589a91f63af-f0b3862e504d4b43a60477e5efef29b1
I20240502 01:53:56.161494 17627 tablet_copy_source_session.cc:205] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:56.163053 21845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0b3862e504d4b43a60477e5efef29b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:56.166803 20628 raft_consensus.cc:2932] T 617443200e2c47b69d63ef056294f40c P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } } }
I20240502 01:53:56.169381 21911 tablet_bootstrap.cc:492] T fac0ba26fabd4643bc38b7559faa471c P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:53:56.169523 21911 ts_tablet_manager.cc:1367] T fac0ba26fabd4643bc38b7559faa471c P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.234s user 0.004s sys 0.000s
W20240502 01:53:56.170675 20565 leader_election.cc:341] T 3ff84ce60ac1438d92b2c944bb27aabb P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:53:56.175369 17334 catalog_manager.cc:5484] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71). New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:56.176573 20233 raft_consensus.cc:1203] T 0618761d30f34cfe9650d13c9275cdcb P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:53:56.195578 20628 raft_consensus.cc:1203] T 5df83f83d2674155afba93c2d992af0a P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:53:56.197196 17736 raft_consensus.cc:1203] T 648b1cc4d8b54c5eb0948ca8c1984a1f P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:53:56.197759 21912 tablet_bootstrap.cc:492] T fbd94103cae04473ba81709da39d7ed8 P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:53:56.197940 21912 ts_tablet_manager.cc:1367] T fbd94103cae04473ba81709da39d7ed8 P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.247s user 0.075s sys 0.096s
I20240502 01:53:56.198741 21912 raft_consensus.cc:348] T fbd94103cae04473ba81709da39d7ed8 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:56.198954 21912 raft_consensus.cc:374] T fbd94103cae04473ba81709da39d7ed8 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:56.199018 21912 raft_consensus.cc:729] T fbd94103cae04473ba81709da39d7ed8 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:53:56.199262 21912 consensus_queue.cc:260] T fbd94103cae04473ba81709da39d7ed8 P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:56.200904 21912 ts_tablet_manager.cc:1398] T fbd94103cae04473ba81709da39d7ed8 P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:53:56.202430 21911 raft_consensus.cc:348] T fac0ba26fabd4643bc38b7559faa471c P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:56.202754 21911 raft_consensus.cc:374] T fac0ba26fabd4643bc38b7559faa471c P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:56.202950 21911 raft_consensus.cc:729] T fac0ba26fabd4643bc38b7559faa471c P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:53:56.203339 21911 consensus_queue.cc:260] T fac0ba26fabd4643bc38b7559faa471c P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:56.204226 21911 ts_tablet_manager.cc:1398] T fac0ba26fabd4643bc38b7559faa471c P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.035s user 0.002s sys 0.000s
I20240502 01:53:56.213701 17333 catalog_manager.cc:5576] TS 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155) does not have the latest schema for tablet 3435cabff9bb4bdfbb7b5fdda725f500 (table rebalancer_dynamic_table [id=a816e480a60a400a872452ea686fb049]). Expected version 2 got 1
I20240502 01:53:56.214821 17332 catalog_manager.cc:5576] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321) does not have the latest schema for tablet 3435cabff9bb4bdfbb7b5fdda725f500 (table rebalancer_dynamic_table [id=a816e480a60a400a872452ea686fb049]). Expected version 2 got 1
I20240502 01:53:56.217362 17864 consensus_queue.cc:237] T 5124abbb09db4be0955c6bc831af08fa P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
I20240502 01:53:56.221544 21910 tablet.cc:1722] T 3435cabff9bb4bdfbb7b5fdda725f500 P fdc2ac372492454c85387589a91f63af: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:53:56.222044 21910 tablet.cc:1620] T 3435cabff9bb4bdfbb7b5fdda725f500 P fdc2ac372492454c85387589a91f63af: MemRowSet was empty: no flush needed.
I20240502 01:53:56.227325 21909 tablet.cc:1722] T 3435cabff9bb4bdfbb7b5fdda725f500 P 6a36c952c51542c0bf7ea3ec2857a9f9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:53:56.228107 21909 tablet.cc:1620] T 3435cabff9bb4bdfbb7b5fdda725f500 P 6a36c952c51542c0bf7ea3ec2857a9f9: MemRowSet was empty: no flush needed.
I20240502 01:53:56.229084 21845 tablet_copy_client.cc:783] T f0b3862e504d4b43a60477e5efef29b1 P fdc2ac372492454c85387589a91f63af: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:56.229326 21845 tablet_copy_client.cc:647] T f0b3862e504d4b43a60477e5efef29b1 P fdc2ac372492454c85387589a91f63af: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:56.229550 21907 tablet.cc:1722] T 3435cabff9bb4bdfbb7b5fdda725f500 P 4da690c0fa13407a85c0689ce8de9ade: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:53:56.229976 21907 tablet.cc:1620] T 3435cabff9bb4bdfbb7b5fdda725f500 P 4da690c0fa13407a85c0689ce8de9ade: MemRowSet was empty: no flush needed.
I20240502 01:53:56.231209 21914 tablet_bootstrap.cc:492] T 3ff84ce60ac1438d92b2c944bb27aabb P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:53:56.231456 17327 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=a816e480a60a400a872452ea686fb049] alter complete (version 2)
I20240502 01:53:56.232056 21914 ts_tablet_manager.cc:1367] T 3ff84ce60ac1438d92b2c944bb27aabb P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.256s user 0.052s sys 0.103s
I20240502 01:53:56.232897 21914 raft_consensus.cc:348] T 3ff84ce60ac1438d92b2c944bb27aabb P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:56.233187 21914 raft_consensus.cc:374] T 3ff84ce60ac1438d92b2c944bb27aabb P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:56.233263 21845 tablet_copy_client.cc:523] T f0b3862e504d4b43a60477e5efef29b1 P fdc2ac372492454c85387589a91f63af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:56.233302 21914 raft_consensus.cc:729] T 3ff84ce60ac1438d92b2c944bb27aabb P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:53:56.233575 21914 consensus_queue.cc:260] T 3ff84ce60ac1438d92b2c944bb27aabb P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:56.234627 21915 tablet_bootstrap.cc:492] T fbd94103cae04473ba81709da39d7ed8 P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:53:56.234776 21915 ts_tablet_manager.cc:1367] T fbd94103cae04473ba81709da39d7ed8 P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.247s user 0.031s sys 0.047s
I20240502 01:53:56.235481 21915 raft_consensus.cc:348] T fbd94103cae04473ba81709da39d7ed8 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:56.235683 21915 raft_consensus.cc:374] T fbd94103cae04473ba81709da39d7ed8 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:56.235774 21915 raft_consensus.cc:729] T fbd94103cae04473ba81709da39d7ed8 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:53:56.236006 21915 consensus_queue.cc:260] T fbd94103cae04473ba81709da39d7ed8 P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:56.236662 21915 ts_tablet_manager.cc:1398] T fbd94103cae04473ba81709da39d7ed8 P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:53:56.237187 21914 ts_tablet_manager.cc:1398] T 3ff84ce60ac1438d92b2c944bb27aabb P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:53:56.240638 21845 tablet_bootstrap.cc:492] T f0b3862e504d4b43a60477e5efef29b1 P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:53:56.249094 21861 ts_tablet_manager.cc:902] T 4aefc61cb95f4e949e7a23a1c677749e P f19cc6cb1e76460aa6711e49a5775091: Initiating tablet copy from peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
W20240502 01:53:56.256875 17803 consensus_peers.cc:485] T 5124abbb09db4be0955c6bc831af08fa P bb31bceba76d466db24b9341c04900c3 -> Peer 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): Couldn't send request to peer 2f306549407b4bb281ccf2994035e4f1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5124abbb09db4be0955c6bc831af08fa. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:56.257004 17604 raft_consensus.cc:1261] T 5124abbb09db4be0955c6bc831af08fa P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:56.257844 21581 consensus_queue.cc:1035] T 5124abbb09db4be0955c6bc831af08fa P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:56.270576 17736 raft_consensus.cc:1261] T 5124abbb09db4be0955c6bc831af08fa P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:56.271144 21581 consensus_queue.cc:1035] T 5124abbb09db4be0955c6bc831af08fa P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:56.277987 21949 ts_tablet_manager.cc:902] T 1fdd104f1a28446cbc25d4ca16d03d7d P 6a36c952c51542c0bf7ea3ec2857a9f9: Initiating tablet copy from peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:56.282533 21581 raft_consensus.cc:2932] T 5124abbb09db4be0955c6bc831af08fa P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:53:56.290228 17604 raft_consensus.cc:2932] T 5124abbb09db4be0955c6bc831af08fa P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:53:56.290690 17332 catalog_manager.cc:5484] T 5124abbb09db4be0955c6bc831af08fa P bb31bceba76d466db24b9341c04900c3 reported cstate change: config changed from index -1 to 4, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:56.321684 17736 raft_consensus.cc:2932] T 5124abbb09db4be0955c6bc831af08fa P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:53:56.330638 21845 tablet_bootstrap.cc:492] T f0b3862e504d4b43a60477e5efef29b1 P fdc2ac372492454c85387589a91f63af: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:56.331086 21845 tablet_bootstrap.cc:492] T f0b3862e504d4b43a60477e5efef29b1 P fdc2ac372492454c85387589a91f63af: Bootstrap complete.
I20240502 01:53:56.331346 21845 ts_tablet_manager.cc:1367] T f0b3862e504d4b43a60477e5efef29b1 P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.091s user 0.045s sys 0.005s
I20240502 01:53:56.332289 21845 raft_consensus.cc:348] T f0b3862e504d4b43a60477e5efef29b1 P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } }
I20240502 01:53:56.332644 21845 raft_consensus.cc:729] T f0b3862e504d4b43a60477e5efef29b1 P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: LEARNER
I20240502 01:53:56.333894 21845 consensus_queue.cc:260] T f0b3862e504d4b43a60477e5efef29b1 P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } }
I20240502 01:53:56.335458 21845 ts_tablet_manager.cc:1398] T f0b3862e504d4b43a60477e5efef29b1 P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20240502 01:53:56.336145 17627 tablet_copy_service.cc:342] P 3fceccefba274b1dbbd8216af8b0bc29: Request end of tablet copy session fdc2ac372492454c85387589a91f63af-f0b3862e504d4b43a60477e5efef29b1 received from {username='slave'} at 127.11.42.69:59447
I20240502 01:53:56.336285 17627 tablet_copy_service.cc:433] P 3fceccefba274b1dbbd8216af8b0bc29: ending tablet copy session fdc2ac372492454c85387589a91f63af-f0b3862e504d4b43a60477e5efef29b1 on tablet f0b3862e504d4b43a60477e5efef29b1 with peer fdc2ac372492454c85387589a91f63af
I20240502 01:53:56.339525 21861 tablet_copy_client.cc:308] T 4aefc61cb95f4e949e7a23a1c677749e P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.66:38531
I20240502 01:53:56.340211 17627 tablet_copy_service.cc:140] P 3fceccefba274b1dbbd8216af8b0bc29: Received BeginTabletCopySession request for tablet 4aefc61cb95f4e949e7a23a1c677749e from peer f19cc6cb1e76460aa6711e49a5775091 ({username='slave'} at 127.11.42.72:53885)
I20240502 01:53:56.340369 17627 tablet_copy_service.cc:161] P 3fceccefba274b1dbbd8216af8b0bc29: Beginning new tablet copy session on tablet 4aefc61cb95f4e949e7a23a1c677749e from peer f19cc6cb1e76460aa6711e49a5775091 at {username='slave'} at 127.11.42.72:53885: session id = f19cc6cb1e76460aa6711e49a5775091-4aefc61cb95f4e949e7a23a1c677749e
I20240502 01:53:56.342788 17627 tablet_copy_source_session.cc:205] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:56.344035 21949 tablet_copy_client.cc:308] T 1fdd104f1a28446cbc25d4ca16d03d7d P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.66:38531
I20240502 01:53:56.344120 21861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aefc61cb95f4e949e7a23a1c677749e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:56.344659 17627 tablet_copy_service.cc:140] P 3fceccefba274b1dbbd8216af8b0bc29: Received BeginTabletCopySession request for tablet 1fdd104f1a28446cbc25d4ca16d03d7d from peer 6a36c952c51542c0bf7ea3ec2857a9f9 ({username='slave'} at 127.11.42.70:53247)
I20240502 01:53:56.344825 17627 tablet_copy_service.cc:161] P 3fceccefba274b1dbbd8216af8b0bc29: Beginning new tablet copy session on tablet 1fdd104f1a28446cbc25d4ca16d03d7d from peer 6a36c952c51542c0bf7ea3ec2857a9f9 at {username='slave'} at 127.11.42.70:53247: session id = 6a36c952c51542c0bf7ea3ec2857a9f9-1fdd104f1a28446cbc25d4ca16d03d7d
I20240502 01:53:56.346913 17627 tablet_copy_source_session.cc:205] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:56.347923 21949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fdd104f1a28446cbc25d4ca16d03d7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:56.351306 21861 tablet_copy_client.cc:783] T 4aefc61cb95f4e949e7a23a1c677749e P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:56.351574 21861 tablet_copy_client.cc:647] T 4aefc61cb95f4e949e7a23a1c677749e P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:56.351976 17736 consensus_queue.cc:237] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } }
I20240502 01:53:56.353220 21861 tablet_copy_client.cc:523] T 4aefc61cb95f4e949e7a23a1c677749e P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:56.353572 21949 tablet_copy_client.cc:783] T 1fdd104f1a28446cbc25d4ca16d03d7d P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:56.353847 21949 tablet_copy_client.cc:647] T 1fdd104f1a28446cbc25d4ca16d03d7d P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:56.356232 17474 raft_consensus.cc:1261] T 71cbc012efe14fdb937302796a9f84e5 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:56.356451 17864 raft_consensus.cc:1261] T 71cbc012efe14fdb937302796a9f84e5 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:56.356696 17327 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:53:56.356981 21530 consensus_queue.cc:1035] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:56.357442 21674 consensus_queue.cc:1035] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:53:56.359488 17672 consensus_peers.cc:485] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 -> Peer 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): Couldn't send request to peer 6a36c952c51542c0bf7ea3ec2857a9f9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 71cbc012efe14fdb937302796a9f84e5. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:56.362552 21861 tablet_bootstrap.cc:492] T 4aefc61cb95f4e949e7a23a1c677749e P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:53:56.367882 21907 tablet.cc:1722] T 3435cabff9bb4bdfbb7b5fdda725f500 P 4da690c0fa13407a85c0689ce8de9ade: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:53:56.374771 21910 tablet.cc:1722] T 3435cabff9bb4bdfbb7b5fdda725f500 P fdc2ac372492454c85387589a91f63af: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:53:56.378154 17327 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table_new_name [id=a816e480a60a400a872452ea686fb049] alter complete (version 3)
I20240502 01:53:56.382706 21909 tablet.cc:1722] T 3435cabff9bb4bdfbb7b5fdda725f500 P 6a36c952c51542c0bf7ea3ec2857a9f9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:53:56.393479 21949 tablet_copy_client.cc:523] T 1fdd104f1a28446cbc25d4ca16d03d7d P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:56.401120 21949 tablet_bootstrap.cc:492] T 1fdd104f1a28446cbc25d4ca16d03d7d P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:53:56.406767 21674 raft_consensus.cc:2932] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } } }
I20240502 01:53:56.413780 17866 raft_consensus.cc:2932] T 71cbc012efe14fdb937302796a9f84e5 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } } }
I20240502 01:53:56.417515 21920 raft_consensus.cc:1050] T 5e8e78bd417d46cdb166087dbfa86446 P 4da690c0fa13407a85c0689ce8de9ade: attempting to promote NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 to VOTER
I20240502 01:53:56.418373 21920 consensus_queue.cc:237] T 5e8e78bd417d46cdb166087dbfa86446 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } }
I20240502 01:53:56.420328 17327 catalog_manager.cc:5484] T 71cbc012efe14fdb937302796a9f84e5 P bb31bceba76d466db24b9341c04900c3 reported cstate change: config changed from index -1 to 4, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } } }
I20240502 01:53:56.427227 17604 raft_consensus.cc:1261] T 5e8e78bd417d46cdb166087dbfa86446 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEARNER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:53:56.428148 21876 consensus_queue.cc:1035] T 5e8e78bd417d46cdb166087dbfa86446 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:53:56.428591 20498 raft_consensus.cc:1261] T 5e8e78bd417d46cdb166087dbfa86446 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:53:56.429220 20365 raft_consensus.cc:1261] T 5e8e78bd417d46cdb166087dbfa86446 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:53:56.429821 21876 consensus_queue.cc:1035] T 5e8e78bd417d46cdb166087dbfa86446 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:53:56.430388 21876 consensus_queue.cc:1035] T 5e8e78bd417d46cdb166087dbfa86446 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:53:56.441613 17473 raft_consensus.cc:2932] T 71cbc012efe14fdb937302796a9f84e5 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } } }
I20240502 01:53:56.452543 20233 raft_consensus.cc:1203] T f0b3862e504d4b43a60477e5efef29b1 P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:53:56.455000 21934 raft_consensus.cc:482] T fbd94103cae04473ba81709da39d7ed8 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:56.455157 21934 raft_consensus.cc:504] T fbd94103cae04473ba81709da39d7ed8 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:56.456002 21934 leader_election.cc:288] T fbd94103cae04473ba81709da39d7ed8 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:53:56.456506 20497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbd94103cae04473ba81709da39d7ed8" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
I20240502 01:53:56.456645 20233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbd94103cae04473ba81709da39d7ed8" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:53:56.456763 20497 raft_consensus.cc:2454] T fbd94103cae04473ba81709da39d7ed8 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 0.
I20240502 01:53:56.456845 20233 raft_consensus.cc:2454] T fbd94103cae04473ba81709da39d7ed8 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 0.
I20240502 01:53:56.457274 20565 leader_election.cc:302] T fbd94103cae04473ba81709da39d7ed8 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f19cc6cb1e76460aa6711e49a5775091, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:53:56.457577 21934 raft_consensus.cc:2781] T fbd94103cae04473ba81709da39d7ed8 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:56.457713 21934 raft_consensus.cc:482] T fbd94103cae04473ba81709da39d7ed8 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:56.457815 21934 raft_consensus.cc:3037] T fbd94103cae04473ba81709da39d7ed8 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:56.460692 21934 raft_consensus.cc:504] T fbd94103cae04473ba81709da39d7ed8 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:56.462060 20497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbd94103cae04473ba81709da39d7ed8" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
I20240502 01:53:56.462244 21934 leader_election.cc:288] T fbd94103cae04473ba81709da39d7ed8 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 election: Requested vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:53:56.462262 20497 raft_consensus.cc:3037] T fbd94103cae04473ba81709da39d7ed8 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:56.462335 20233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbd94103cae04473ba81709da39d7ed8" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:53:56.462513 20233 raft_consensus.cc:3037] T fbd94103cae04473ba81709da39d7ed8 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:56.465040 20233 raft_consensus.cc:2454] T fbd94103cae04473ba81709da39d7ed8 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 1.
I20240502 01:53:56.465005 20497 raft_consensus.cc:2454] T fbd94103cae04473ba81709da39d7ed8 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 1.
I20240502 01:53:56.465557 20565 leader_election.cc:302] T fbd94103cae04473ba81709da39d7ed8 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:53:56.465886 21934 raft_consensus.cc:2781] T fbd94103cae04473ba81709da39d7ed8 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:56.466043 21934 raft_consensus.cc:686] T fbd94103cae04473ba81709da39d7ed8 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Becoming Leader. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Running, Role: LEADER
I20240502 01:53:56.466336 21934 consensus_queue.cc:237] T fbd94103cae04473ba81709da39d7ed8 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:56.487334 17332 catalog_manager.cc:5484] T fbd94103cae04473ba81709da39d7ed8 P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: term changed from 0 to 1, leader changed from <none> to f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72). New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:56.490370 17327 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:53:56.490631 17327 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:53:56.495290 21876 raft_consensus.cc:2932] T 5e8e78bd417d46cdb166087dbfa86446 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:53:56.495973 20367 raft_consensus.cc:2932] T 5e8e78bd417d46cdb166087dbfa86446 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:53:56.502229 17333 catalog_manager.cc:5484] T 5e8e78bd417d46cdb166087dbfa86446 P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:53:56.506302 20366 raft_consensus.cc:3037] T fac0ba26fabd4643bc38b7559faa471c P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:53:56.507498 17543 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.42.69:34339, user_credentials={real_user=slave}} blocked reactor thread for 54220.4us
I20240502 01:53:56.508008 17327 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet 3435cabff9bb4bdfbb7b5fdda725f500
I20240502 01:53:56.509375 20347 tablet_service.cc:1514] Processing DeleteTablet for tablet 3435cabff9bb4bdfbb7b5fdda725f500 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:56 UTC) from {username='slave'} at 127.0.0.1:40196
I20240502 01:53:56.509377 17454 tablet_service.cc:1514] Processing DeleteTablet for tablet 3435cabff9bb4bdfbb7b5fdda725f500 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:56 UTC) from {username='slave'} at 127.0.0.1:54520
I20240502 01:53:56.510071 21697 consensus_queue.cc:1035] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:56.511229 20211 tablet_service.cc:1514] Processing DeleteTablet for tablet 3435cabff9bb4bdfbb7b5fdda725f500 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:56 UTC) from {username='slave'} at 127.0.0.1:44624
I20240502 01:53:56.513388 17604 raft_consensus.cc:2932] T 5e8e78bd417d46cdb166087dbfa86446 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:53:56.520702 21991 tablet_replica.cc:330] T 3435cabff9bb4bdfbb7b5fdda725f500 P fdc2ac372492454c85387589a91f63af: stopping tablet replica
I20240502 01:53:56.521677 21989 tablet_replica.cc:330] T 3435cabff9bb4bdfbb7b5fdda725f500 P 4da690c0fa13407a85c0689ce8de9ade: stopping tablet replica
I20240502 01:53:56.522131 21989 raft_consensus.cc:2229] T 3435cabff9bb4bdfbb7b5fdda725f500 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:56.522464 21991 raft_consensus.cc:2229] T 3435cabff9bb4bdfbb7b5fdda725f500 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:56.523417 21989 raft_consensus.cc:2258] T 3435cabff9bb4bdfbb7b5fdda725f500 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:56.523728 21991 raft_consensus.cc:2258] T 3435cabff9bb4bdfbb7b5fdda725f500 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:56.526407 21991 ts_tablet_manager.cc:1875] T 3435cabff9bb4bdfbb7b5fdda725f500 P fdc2ac372492454c85387589a91f63af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:56.527158 21974 ts_tablet_manager.cc:902] T 617443200e2c47b69d63ef056294f40c P 4da690c0fa13407a85c0689ce8de9ade: Initiating tablet copy from peer 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:53:56.528542 21989 ts_tablet_manager.cc:1875] T 3435cabff9bb4bdfbb7b5fdda725f500 P 4da690c0fa13407a85c0689ce8de9ade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:56.531890 21988 tablet_replica.cc:330] T 3435cabff9bb4bdfbb7b5fdda725f500 P 6a36c952c51542c0bf7ea3ec2857a9f9: stopping tablet replica
I20240502 01:53:56.532107 21988 raft_consensus.cc:2229] T 3435cabff9bb4bdfbb7b5fdda725f500 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:56.532348 21988 raft_consensus.cc:2258] T 3435cabff9bb4bdfbb7b5fdda725f500 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:56.533319 21988 ts_tablet_manager.cc:1875] T 3435cabff9bb4bdfbb7b5fdda725f500 P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:56.539022 21991 ts_tablet_manager.cc:1888] T 3435cabff9bb4bdfbb7b5fdda725f500 P fdc2ac372492454c85387589a91f63af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:53:56.539184 21991 log.cc:1196] T 3435cabff9bb4bdfbb7b5fdda725f500 P fdc2ac372492454c85387589a91f63af: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal/wals/3435cabff9bb4bdfbb7b5fdda725f500
I20240502 01:53:56.538976 20498 raft_consensus.cc:2932] T 5e8e78bd417d46cdb166087dbfa86446 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:53:56.539700 21988 ts_tablet_manager.cc:1888] T 3435cabff9bb4bdfbb7b5fdda725f500 P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:53:56.539808 21988 log.cc:1196] T 3435cabff9bb4bdfbb7b5fdda725f500 P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal/wals/3435cabff9bb4bdfbb7b5fdda725f500
I20240502 01:53:56.540251 21988 ts_tablet_manager.cc:1909] T 3435cabff9bb4bdfbb7b5fdda725f500 P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting consensus metadata
I20240502 01:53:56.540756 17475 consensus_queue.cc:237] T 5e8e78bd417d46cdb166087dbfa86446 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } }
I20240502 01:53:56.541389 17321 catalog_manager.cc:4830] TS 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): tablet 3435cabff9bb4bdfbb7b5fdda725f500 (table rebalancer_dynamic_table_new_name [id=a816e480a60a400a872452ea686fb049]) successfully deleted
I20240502 01:53:56.543388 20497 raft_consensus.cc:1261] T 5e8e78bd417d46cdb166087dbfa86446 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:56.543918 21876 consensus_queue.cc:1035] T 5e8e78bd417d46cdb166087dbfa86446 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:56.544018 17604 raft_consensus.cc:1261] T 5e8e78bd417d46cdb166087dbfa86446 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:56.545209 21216 consensus_queue.cc:1035] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:56.545332 21876 consensus_queue.cc:1035] T 5e8e78bd417d46cdb166087dbfa86446 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:56.547842 21622 raft_consensus.cc:2932] T 5e8e78bd417d46cdb166087dbfa86446 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:53:56.551084 21989 ts_tablet_manager.cc:1888] T 3435cabff9bb4bdfbb7b5fdda725f500 P 4da690c0fa13407a85c0689ce8de9ade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:53:56.551225 21989 log.cc:1196] T 3435cabff9bb4bdfbb7b5fdda725f500 P 4da690c0fa13407a85c0689ce8de9ade: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/3435cabff9bb4bdfbb7b5fdda725f500
I20240502 01:53:56.552274 17604 raft_consensus.cc:2932] T 5e8e78bd417d46cdb166087dbfa86446 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:53:56.554667 21991 ts_tablet_manager.cc:1909] T 3435cabff9bb4bdfbb7b5fdda725f500 P fdc2ac372492454c85387589a91f63af: Deleting consensus metadata
I20240502 01:53:56.554468 20497 raft_consensus.cc:2932] T 5e8e78bd417d46cdb166087dbfa86446 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:53:56.556325 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5e8e78bd417d46cdb166087dbfa86446 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:56.561204 17322 catalog_manager.cc:4830] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): tablet 3435cabff9bb4bdfbb7b5fdda725f500 (table rebalancer_dynamic_table_new_name [id=a816e480a60a400a872452ea686fb049]) successfully deleted
I20240502 01:53:56.560825 17333 catalog_manager.cc:5484] T 5e8e78bd417d46cdb166087dbfa86446 P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: config changed from index 3 to 4, VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) evicted. New cstate: current_term: 1 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:56.562309 21989 ts_tablet_manager.cc:1909] T 3435cabff9bb4bdfbb7b5fdda725f500 P 4da690c0fa13407a85c0689ce8de9ade: Deleting consensus metadata
I20240502 01:53:56.564287 17321 catalog_manager.cc:4830] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): tablet 3435cabff9bb4bdfbb7b5fdda725f500 (table rebalancer_dynamic_table_new_name [id=a816e480a60a400a872452ea686fb049]) successfully deleted
I20240502 01:53:56.569120 20347 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e8e78bd417d46cdb166087dbfa86446 with delete_type TABLET_DATA_TOMBSTONED (TS 6a36c952c51542c0bf7ea3ec2857a9f9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40196
I20240502 01:53:56.573432 21988 tablet_replica.cc:330] T 5e8e78bd417d46cdb166087dbfa86446 P 6a36c952c51542c0bf7ea3ec2857a9f9: stopping tablet replica
I20240502 01:53:56.573755 21988 raft_consensus.cc:2229] T 5e8e78bd417d46cdb166087dbfa86446 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:56.574040 21988 raft_consensus.cc:2258] T 5e8e78bd417d46cdb166087dbfa86446 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:56.574016 17605 consensus_queue.cc:237] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:53:56.575351 21988 ts_tablet_manager.cc:1875] T 5e8e78bd417d46cdb166087dbfa86446 P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:56.581611 21974 tablet_copy_client.cc:308] T 617443200e2c47b69d63ef056294f40c P 4da690c0fa13407a85c0689ce8de9ade: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.71:38623
I20240502 01:53:56.581908 21988 ts_tablet_manager.cc:1888] T 5e8e78bd417d46cdb166087dbfa86446 P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:53:56.582055 21988 log.cc:1196] T 5e8e78bd417d46cdb166087dbfa86446 P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal/wals/5e8e78bd417d46cdb166087dbfa86446
I20240502 01:53:56.582486 20518 tablet_copy_service.cc:140] P 2f306549407b4bb281ccf2994035e4f1: Received BeginTabletCopySession request for tablet 617443200e2c47b69d63ef056294f40c from peer 4da690c0fa13407a85c0689ce8de9ade ({username='slave'} at 127.11.42.65:53097)
I20240502 01:53:56.582681 20518 tablet_copy_service.cc:161] P 2f306549407b4bb281ccf2994035e4f1: Beginning new tablet copy session on tablet 617443200e2c47b69d63ef056294f40c from peer 4da690c0fa13407a85c0689ce8de9ade at {username='slave'} at 127.11.42.65:53097: session id = 4da690c0fa13407a85c0689ce8de9ade-617443200e2c47b69d63ef056294f40c
I20240502 01:53:56.584756 17475 raft_consensus.cc:1261] T 10ec78a08ab04678897a6892988e515f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:56.585255 17735 raft_consensus.cc:1261] T 10ec78a08ab04678897a6892988e515f P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:56.585335 20518 tablet_copy_source_session.cc:205] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:56.585572 21982 consensus_queue.cc:1035] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:56.585906 20497 raft_consensus.cc:3037] T 3ff84ce60ac1438d92b2c944bb27aabb P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:56.588124 21974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 617443200e2c47b69d63ef056294f40c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:56.590016 21934 consensus_queue.cc:1035] T 3ff84ce60ac1438d92b2c944bb27aabb P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:56.591630 17542 consensus_peers.cc:485] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 10ec78a08ab04678897a6892988e515f. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:56.592079 21972 consensus_queue.cc:1035] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:56.594409 17321 catalog_manager.cc:4830] TS 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): tablet 5e8e78bd417d46cdb166087dbfa86446 (table rebalancer_extra_table_4 [id=dca50b50b02e4a8c8b90e0de6f305bd8]) successfully deleted
I20240502 01:53:56.603940 21974 tablet_copy_client.cc:783] T 617443200e2c47b69d63ef056294f40c P 4da690c0fa13407a85c0689ce8de9ade: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:56.604267 21974 tablet_copy_client.cc:647] T 617443200e2c47b69d63ef056294f40c P 4da690c0fa13407a85c0689ce8de9ade: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:56.606061 21969 raft_consensus.cc:2932] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:53:56.606721 17475 raft_consensus.cc:2932] T 10ec78a08ab04678897a6892988e515f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:53:56.608064 21934 consensus_queue.cc:1035] T 3ff84ce60ac1438d92b2c944bb27aabb P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:56.611778 21216 raft_consensus.cc:1050] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1: attempting to promote NON_VOTER dfa1152987644bd781b46c7e42a7a685 to VOTER
I20240502 01:53:56.580981 21861 tablet_bootstrap.cc:492] T 4aefc61cb95f4e949e7a23a1c677749e P f19cc6cb1e76460aa6711e49a5775091: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:56.615002 17332 catalog_manager.cc:5484] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: config changed from index -1 to 4, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:56.615738 21861 tablet_bootstrap.cc:492] T 4aefc61cb95f4e949e7a23a1c677749e P f19cc6cb1e76460aa6711e49a5775091: Bootstrap complete.
I20240502 01:53:56.616022 21861 ts_tablet_manager.cc:1367] T 4aefc61cb95f4e949e7a23a1c677749e P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.254s user 0.048s sys 0.003s
I20240502 01:53:56.615017 17327 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:53:56.616858 21861 raft_consensus.cc:348] T 4aefc61cb95f4e949e7a23a1c677749e P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:53:56.617120 21861 raft_consensus.cc:729] T 4aefc61cb95f4e949e7a23a1c677749e P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: LEARNER
I20240502 01:53:56.617333 21861 consensus_queue.cc:260] T 4aefc61cb95f4e949e7a23a1c677749e P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:53:56.617894 21861 ts_tablet_manager.cc:1398] T 4aefc61cb95f4e949e7a23a1c677749e P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:56.619722 17627 tablet_copy_service.cc:342] P 3fceccefba274b1dbbd8216af8b0bc29: Request end of tablet copy session f19cc6cb1e76460aa6711e49a5775091-4aefc61cb95f4e949e7a23a1c677749e received from {username='slave'} at 127.11.42.72:53885
I20240502 01:53:56.619836 17627 tablet_copy_service.cc:433] P 3fceccefba274b1dbbd8216af8b0bc29: ending tablet copy session f19cc6cb1e76460aa6711e49a5775091-4aefc61cb95f4e949e7a23a1c677749e on tablet 4aefc61cb95f4e949e7a23a1c677749e with peer f19cc6cb1e76460aa6711e49a5775091
I20240502 01:53:56.612795 21216 consensus_queue.cc:237] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:53:56.624487 17735 raft_consensus.cc:2932] T 10ec78a08ab04678897a6892988e515f P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:53:56.629510 20347 tablet_service.cc:1467] Processing CreateTablet for tablet acee3683811241d293a7565b02f2e274 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=492ccb5ecf704a8a878f6ef6acb0d927]), partition=
I20240502 01:53:56.629575 17846 tablet_service.cc:1467] Processing CreateTablet for tablet acee3683811241d293a7565b02f2e274 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=492ccb5ecf704a8a878f6ef6acb0d927]), partition=
I20240502 01:53:56.629655 20477 tablet_service.cc:1467] Processing CreateTablet for tablet acee3683811241d293a7565b02f2e274 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=492ccb5ecf704a8a878f6ef6acb0d927]), partition=
I20240502 01:53:56.629878 17846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acee3683811241d293a7565b02f2e274. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:56.629878 20347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acee3683811241d293a7565b02f2e274. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:56.629973 20477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acee3683811241d293a7565b02f2e274. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:56.635814 21911 tablet_bootstrap.cc:492] T acee3683811241d293a7565b02f2e274 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:53:56.637351 21911 tablet_bootstrap.cc:654] T acee3683811241d293a7565b02f2e274 P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:56.638381 20628 raft_consensus.cc:1203] T 4aefc61cb95f4e949e7a23a1c677749e P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:53:56.645215 17475 raft_consensus.cc:1261] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:53:56.646209 21924 consensus_queue.cc:1035] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:53:56.648056 21911 tablet_bootstrap.cc:492] T acee3683811241d293a7565b02f2e274 P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:53:56.648272 21911 ts_tablet_manager.cc:1367] T acee3683811241d293a7565b02f2e274 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.005s
I20240502 01:53:56.649107 21911 raft_consensus.cc:348] T acee3683811241d293a7565b02f2e274 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:56.649399 21911 raft_consensus.cc:374] T acee3683811241d293a7565b02f2e274 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:56.649531 21911 raft_consensus.cc:729] T acee3683811241d293a7565b02f2e274 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:53:56.649878 21911 consensus_queue.cc:260] T acee3683811241d293a7565b02f2e274 P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:56.652036 17735 raft_consensus.cc:1261] T 648b1cc4d8b54c5eb0948ca8c1984a1f P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:53:56.653323 21924 consensus_queue.cc:1035] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:53:56.653970 21914 tablet_bootstrap.cc:492] T acee3683811241d293a7565b02f2e274 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:53:56.654577 21876 raft_consensus.cc:1050] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade: attempting to promote NON_VOTER fdc2ac372492454c85387589a91f63af to VOTER
I20240502 01:53:56.655055 21911 ts_tablet_manager.cc:1398] T acee3683811241d293a7565b02f2e274 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 01:53:56.655447 21914 tablet_bootstrap.cc:654] T acee3683811241d293a7565b02f2e274 P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:56.655390 21876 consensus_queue.cc:237] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } }
I20240502 01:53:56.667946 22009 tablet_bootstrap.cc:492] T acee3683811241d293a7565b02f2e274 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:56.672128 22009 tablet_bootstrap.cc:654] T acee3683811241d293a7565b02f2e274 P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:56.673532 20366 raft_consensus.cc:1261] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:53:56.674074 21216 consensus_queue.cc:1035] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:53:56.676242 21974 tablet_copy_client.cc:523] T 617443200e2c47b69d63ef056294f40c P 4da690c0fa13407a85c0689ce8de9ade: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:56.680513 21974 tablet_bootstrap.cc:492] T 617443200e2c47b69d63ef056294f40c P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:56.692278 21949 tablet_bootstrap.cc:492] T 1fdd104f1a28446cbc25d4ca16d03d7d P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:56.692835 21949 tablet_bootstrap.cc:492] T 1fdd104f1a28446cbc25d4ca16d03d7d P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap complete.
I20240502 01:53:56.694078 20233 raft_consensus.cc:1261] T 0618761d30f34cfe9650d13c9275cdcb P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:56.694568 17736 raft_consensus.cc:1261] T 0618761d30f34cfe9650d13c9275cdcb P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:56.695027 17864 raft_consensus.cc:1261] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:56.695281 22009 tablet_bootstrap.cc:492] T acee3683811241d293a7565b02f2e274 P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:56.695266 21920 consensus_queue.cc:1035] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:56.695451 22009 ts_tablet_manager.cc:1367] T acee3683811241d293a7565b02f2e274 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.000s
I20240502 01:53:56.695837 21622 consensus_queue.cc:1035] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:56.697072 21622 consensus_queue.cc:1035] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:56.701565 21949 ts_tablet_manager.cc:1367] T 1fdd104f1a28446cbc25d4ca16d03d7d P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.300s user 0.040s sys 0.014s
I20240502 01:53:56.702361 21949 raft_consensus.cc:348] T 1fdd104f1a28446cbc25d4ca16d03d7d P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } }
I20240502 01:53:56.702610 21949 raft_consensus.cc:729] T 1fdd104f1a28446cbc25d4ca16d03d7d P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: LEARNER
I20240502 01:53:56.705713 21949 consensus_queue.cc:260] T 1fdd104f1a28446cbc25d4ca16d03d7d P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } }
I20240502 01:53:56.707713 21949 ts_tablet_manager.cc:1398] T 1fdd104f1a28446cbc25d4ca16d03d7d P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 01:53:56.709786 17627 tablet_copy_service.cc:342] P 3fceccefba274b1dbbd8216af8b0bc29: Request end of tablet copy session 6a36c952c51542c0bf7ea3ec2857a9f9-1fdd104f1a28446cbc25d4ca16d03d7d received from {username='slave'} at 127.11.42.70:53247
I20240502 01:53:56.709969 17627 tablet_copy_service.cc:433] P 3fceccefba274b1dbbd8216af8b0bc29: ending tablet copy session 6a36c952c51542c0bf7ea3ec2857a9f9-1fdd104f1a28446cbc25d4ca16d03d7d on tablet 1fdd104f1a28446cbc25d4ca16d03d7d with peer 6a36c952c51542c0bf7ea3ec2857a9f9
I20240502 01:53:56.717104 21216 raft_consensus.cc:2932] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:53:56.722154 22009 raft_consensus.cc:348] T acee3683811241d293a7565b02f2e274 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:56.722446 22009 raft_consensus.cc:374] T acee3683811241d293a7565b02f2e274 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:56.722638 22009 raft_consensus.cc:729] T acee3683811241d293a7565b02f2e274 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:56.723481 22009 consensus_queue.cc:260] T acee3683811241d293a7565b02f2e274 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:56.724891 21530 raft_consensus.cc:1050] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685: attempting to promote NON_VOTER f19cc6cb1e76460aa6711e49a5775091 to VOTER
I20240502 01:53:56.724745 17475 raft_consensus.cc:2932] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:53:56.725749 21530 consensus_queue.cc:237] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } }
I20240502 01:53:56.728276 17327 catalog_manager.cc:5484] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:56.729034 20366 raft_consensus.cc:2932] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:53:56.737090 21622 raft_consensus.cc:2932] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:53:56.738840 21914 tablet_bootstrap.cc:492] T acee3683811241d293a7565b02f2e274 P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:53:56.738996 21914 ts_tablet_manager.cc:1367] T acee3683811241d293a7565b02f2e274 P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.085s user 0.011s sys 0.005s
I20240502 01:53:56.741463 21914 raft_consensus.cc:348] T acee3683811241d293a7565b02f2e274 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:56.741686 21914 raft_consensus.cc:374] T acee3683811241d293a7565b02f2e274 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:56.741772 21914 raft_consensus.cc:729] T acee3683811241d293a7565b02f2e274 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:53:56.741887 17736 raft_consensus.cc:2932] T 0618761d30f34cfe9650d13c9275cdcb P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:53:56.743072 22024 ts_tablet_manager.cc:902] T 5124abbb09db4be0955c6bc831af08fa P 2f306549407b4bb281ccf2994035e4f1: Initiating tablet copy from peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:53:56.745791 22009 ts_tablet_manager.cc:1398] T acee3683811241d293a7565b02f2e274 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.050s user 0.003s sys 0.000s
I20240502 01:53:56.747040 21914 consensus_queue.cc:260] T acee3683811241d293a7565b02f2e274 P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:56.747800 21914 ts_tablet_manager.cc:1398] T acee3683811241d293a7565b02f2e274 P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 01:53:56.748091 17866 raft_consensus.cc:2932] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:53:56.750638 21974 tablet_bootstrap.cc:492] T 617443200e2c47b69d63ef056294f40c P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:56.751118 21974 tablet_bootstrap.cc:492] T 617443200e2c47b69d63ef056294f40c P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap complete.
I20240502 01:53:56.751438 21974 ts_tablet_manager.cc:1367] T 617443200e2c47b69d63ef056294f40c P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.071s user 0.007s sys 0.006s
I20240502 01:53:56.753952 21974 raft_consensus.cc:348] T 617443200e2c47b69d63ef056294f40c P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } }
I20240502 01:53:56.754271 21974 raft_consensus.cc:729] T 617443200e2c47b69d63ef056294f40c P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: LEARNER
I20240502 01:53:56.754547 21974 consensus_queue.cc:260] T 617443200e2c47b69d63ef056294f40c P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } }
I20240502 01:53:56.756027 21974 ts_tablet_manager.cc:1398] T 617443200e2c47b69d63ef056294f40c P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:53:56.756788 20518 tablet_copy_service.cc:342] P 2f306549407b4bb281ccf2994035e4f1: Request end of tablet copy session 4da690c0fa13407a85c0689ce8de9ade-617443200e2c47b69d63ef056294f40c received from {username='slave'} at 127.11.42.65:53097
I20240502 01:53:56.756897 20518 tablet_copy_service.cc:433] P 2f306549407b4bb281ccf2994035e4f1: ending tablet copy session 4da690c0fa13407a85c0689ce8de9ade-617443200e2c47b69d63ef056294f40c on tablet 617443200e2c47b69d63ef056294f40c with peer 4da690c0fa13407a85c0689ce8de9ade
I20240502 01:53:56.758297 20628 raft_consensus.cc:1261] T 5df83f83d2674155afba93c2d992af0a P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:56.758919 21674 consensus_queue.cc:1035] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:56.760151 17866 raft_consensus.cc:1261] T 5df83f83d2674155afba93c2d992af0a P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:56.760643 21674 consensus_queue.cc:1035] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:56.765415 17333 catalog_manager.cc:5484] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 reported cstate change: config changed from index 4 to 5, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:53:56.773492 21582 raft_consensus.cc:482] T acee3683811241d293a7565b02f2e274 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:56.773638 21582 raft_consensus.cc:504] T acee3683811241d293a7565b02f2e274 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:56.774341 21582 leader_election.cc:288] T acee3683811241d293a7565b02f2e274 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:53:56.774449 20366 raft_consensus.cc:1203] T 1fdd104f1a28446cbc25d4ca16d03d7d P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:53:56.774890 17866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acee3683811241d293a7565b02f2e274" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:53:56.774919 20497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acee3683811241d293a7565b02f2e274" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
I20240502 01:53:56.775102 20497 raft_consensus.cc:2454] T acee3683811241d293a7565b02f2e274 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 0.
I20240502 01:53:56.775099 17866 raft_consensus.cc:2454] T acee3683811241d293a7565b02f2e274 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 0.
I20240502 01:53:56.775885 20303 leader_election.cc:302] T acee3683811241d293a7565b02f2e274 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:53:56.776379 21582 raft_consensus.cc:2781] T acee3683811241d293a7565b02f2e274 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:56.776490 21582 raft_consensus.cc:482] T acee3683811241d293a7565b02f2e274 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:56.776575 21582 raft_consensus.cc:3037] T acee3683811241d293a7565b02f2e274 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:56.779098 21582 raft_consensus.cc:504] T acee3683811241d293a7565b02f2e274 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:56.779714 21582 leader_election.cc:288] T acee3683811241d293a7565b02f2e274 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Requested vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:53:56.780900 17866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acee3683811241d293a7565b02f2e274" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:53:56.781136 17866 raft_consensus.cc:3037] T acee3683811241d293a7565b02f2e274 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:56.781791 22024 tablet_copy_client.cc:308] T 5124abbb09db4be0955c6bc831af08fa P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.68:44083
I20240502 01:53:56.782477 17887 tablet_copy_service.cc:140] P bb31bceba76d466db24b9341c04900c3: Received BeginTabletCopySession request for tablet 5124abbb09db4be0955c6bc831af08fa from peer 2f306549407b4bb281ccf2994035e4f1 ({username='slave'} at 127.11.42.71:48285)
I20240502 01:53:56.782567 20497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acee3683811241d293a7565b02f2e274" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
I20240502 01:53:56.782660 17887 tablet_copy_service.cc:161] P bb31bceba76d466db24b9341c04900c3: Beginning new tablet copy session on tablet 5124abbb09db4be0955c6bc831af08fa from peer 2f306549407b4bb281ccf2994035e4f1 at {username='slave'} at 127.11.42.71:48285: session id = 2f306549407b4bb281ccf2994035e4f1-5124abbb09db4be0955c6bc831af08fa
I20240502 01:53:56.782737 20497 raft_consensus.cc:3037] T acee3683811241d293a7565b02f2e274 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:56.783994 17866 raft_consensus.cc:2454] T acee3683811241d293a7565b02f2e274 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 1.
I20240502 01:53:56.784675 20303 leader_election.cc:302] T acee3683811241d293a7565b02f2e274 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:53:56.785005 21582 raft_consensus.cc:2781] T acee3683811241d293a7565b02f2e274 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:56.785192 21582 raft_consensus.cc:686] T acee3683811241d293a7565b02f2e274 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Becoming Leader. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Running, Role: LEADER
I20240502 01:53:56.785583 20497 raft_consensus.cc:2454] T acee3683811241d293a7565b02f2e274 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 1.
I20240502 01:53:56.785697 17887 tablet_copy_source_session.cc:205] T 5124abbb09db4be0955c6bc831af08fa P bb31bceba76d466db24b9341c04900c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:56.786353 21582 consensus_queue.cc:237] T acee3683811241d293a7565b02f2e274 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:53:56.787873 22024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5124abbb09db4be0955c6bc831af08fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:56.792680 21717 raft_consensus.cc:2932] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:56.793948 17864 raft_consensus.cc:2932] T 5df83f83d2674155afba93c2d992af0a P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:56.795316 22024 tablet_copy_client.cc:783] T 5124abbb09db4be0955c6bc831af08fa P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:56.795545 22024 tablet_copy_client.cc:647] T 5124abbb09db4be0955c6bc831af08fa P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:56.797353 22024 tablet_copy_client.cc:523] T 5124abbb09db4be0955c6bc831af08fa P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:56.800628 20628 raft_consensus.cc:2932] T 5df83f83d2674155afba93c2d992af0a P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:56.802649 17736 consensus_queue.cc:237] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } }
I20240502 01:53:56.803654 22024 tablet_bootstrap.cc:492] T 5124abbb09db4be0955c6bc831af08fa P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:53:56.804780 17332 catalog_manager.cc:5484] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 reported cstate change: config changed from index 4 to 5, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 01:53:56.811795 17673 consensus_peers.cc:485] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8e7f88b6c509477fb0900024e7de8f00. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:56.812235 17605 raft_consensus.cc:1261] T 8e7f88b6c509477fb0900024e7de8f00 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:56.812919 22022 consensus_queue.cc:1035] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:56.813490 17866 raft_consensus.cc:1261] T 8e7f88b6c509477fb0900024e7de8f00 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:56.814034 22022 consensus_queue.cc:1035] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:56.815950 17475 raft_consensus.cc:1261] T 5df83f83d2674155afba93c2d992af0a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:56.816468 17605 raft_consensus.cc:2932] T 8e7f88b6c509477fb0900024e7de8f00 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:53:56.817483 21674 raft_consensus.cc:2932] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:53:56.820962 22022 consensus_queue.cc:1035] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:56.828896 17332 catalog_manager.cc:5484] T 8e7f88b6c509477fb0900024e7de8f00 P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: config changed from index -1 to 4, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:53:56.840420 17736 raft_consensus.cc:2932] T 648b1cc4d8b54c5eb0948ca8c1984a1f P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:53:56.847523 21934 consensus_queue.cc:1035] T fbd94103cae04473ba81709da39d7ed8 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:56.847656 17866 raft_consensus.cc:2932] T 8e7f88b6c509477fb0900024e7de8f00 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:53:56.849438 17475 raft_consensus.cc:2932] T 5df83f83d2674155afba93c2d992af0a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:56.860231 21932 consensus_queue.cc:1035] T fbd94103cae04473ba81709da39d7ed8 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:56.864171 17670 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.42.66:38531, user_credentials={real_user=slave}} blocked reactor thread for 43590.1us
I20240502 01:53:56.880609 17334 catalog_manager.cc:5484] T acee3683811241d293a7565b02f2e274 P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70). New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:56.883000 21949 ts_tablet_manager.cc:902] T 71cbc012efe14fdb937302796a9f84e5 P 6a36c952c51542c0bf7ea3ec2857a9f9: Initiating tablet copy from peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:53:56.883359 20233 raft_consensus.cc:2932] T 0618761d30f34cfe9650d13c9275cdcb P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:53:56.886595 21949 tablet_copy_client.cc:308] T 71cbc012efe14fdb937302796a9f84e5 P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.67:36115
I20240502 01:53:56.887301 17757 tablet_copy_service.cc:140] P dfa1152987644bd781b46c7e42a7a685: Received BeginTabletCopySession request for tablet 71cbc012efe14fdb937302796a9f84e5 from peer 6a36c952c51542c0bf7ea3ec2857a9f9 ({username='slave'} at 127.11.42.70:55001)
I20240502 01:53:56.887482 17757 tablet_copy_service.cc:161] P dfa1152987644bd781b46c7e42a7a685: Beginning new tablet copy session on tablet 71cbc012efe14fdb937302796a9f84e5 from peer 6a36c952c51542c0bf7ea3ec2857a9f9 at {username='slave'} at 127.11.42.70:55001: session id = 6a36c952c51542c0bf7ea3ec2857a9f9-71cbc012efe14fdb937302796a9f84e5
I20240502 01:53:56.890478 17757 tablet_copy_source_session.cc:205] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:56.895071 21949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71cbc012efe14fdb937302796a9f84e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:56.901630 17866 consensus_queue.cc:237] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
I20240502 01:53:56.903110 21949 tablet_copy_client.cc:783] T 71cbc012efe14fdb937302796a9f84e5 P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:56.903348 21949 tablet_copy_client.cc:647] T 71cbc012efe14fdb937302796a9f84e5 P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:56.903983 17736 raft_consensus.cc:1261] T 03bd47783e884a70bfea1dfab4f2e563 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:53:56.904474 17803 consensus_peers.cc:485] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 -> Peer 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): Couldn't send request to peer 2f306549407b4bb281ccf2994035e4f1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 03bd47783e884a70bfea1dfab4f2e563. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:56.904614 17475 raft_consensus.cc:1261] T 03bd47783e884a70bfea1dfab4f2e563 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:56.904716 21954 consensus_queue.cc:1035] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:56.905686 21954 consensus_queue.cc:1035] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:56.905897 21949 tablet_copy_client.cc:523] T 71cbc012efe14fdb937302796a9f84e5 P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:56.910387 21949 tablet_bootstrap.cc:492] T 71cbc012efe14fdb937302796a9f84e5 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:53:56.931598 21954 raft_consensus.cc:2932] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:53:56.936278 17475 raft_consensus.cc:2932] T 03bd47783e884a70bfea1dfab4f2e563 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:53:56.939960 21982 raft_consensus.cc:1050] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29: attempting to promote NON_VOTER fdc2ac372492454c85387589a91f63af to VOTER
I20240502 01:53:56.940820 21982 consensus_queue.cc:237] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } }
I20240502 01:53:56.942024 17335 catalog_manager.cc:5484] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 reported cstate change: config changed from index -1 to 4, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:56.955513 17736 raft_consensus.cc:2932] T 03bd47783e884a70bfea1dfab4f2e563 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:53:56.958804 20233 raft_consensus.cc:1261] T f0b3862e504d4b43a60477e5efef29b1 P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:56.959348 21982 consensus_queue.cc:1035] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:56.960894 17735 raft_consensus.cc:1261] T f0b3862e504d4b43a60477e5efef29b1 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:56.961756 17866 raft_consensus.cc:1261] T f0b3862e504d4b43a60477e5efef29b1 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:56.962479 21973 consensus_queue.cc:1035] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:56.963624 21973 consensus_queue.cc:1035] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:56.974838 17475 raft_consensus.cc:1203] T 617443200e2c47b69d63ef056294f40c P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:53:57.005964 22024 tablet_bootstrap.cc:492] T 5124abbb09db4be0955c6bc831af08fa P 2f306549407b4bb281ccf2994035e4f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:57.006510 22024 tablet_bootstrap.cc:492] T 5124abbb09db4be0955c6bc831af08fa P 2f306549407b4bb281ccf2994035e4f1: Bootstrap complete.
I20240502 01:53:57.006870 22024 ts_tablet_manager.cc:1367] T 5124abbb09db4be0955c6bc831af08fa P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.203s user 0.047s sys 0.013s
I20240502 01:53:57.008134 22024 raft_consensus.cc:348] T 5124abbb09db4be0955c6bc831af08fa P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
I20240502 01:53:57.008407 22024 raft_consensus.cc:729] T 5124abbb09db4be0955c6bc831af08fa P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: LEARNER
I20240502 01:53:57.008618 22024 consensus_queue.cc:260] T 5124abbb09db4be0955c6bc831af08fa P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
I20240502 01:53:57.010380 22024 ts_tablet_manager.cc:1398] T 5124abbb09db4be0955c6bc831af08fa P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:57.011113 17887 tablet_copy_service.cc:342] P bb31bceba76d466db24b9341c04900c3: Request end of tablet copy session 2f306549407b4bb281ccf2994035e4f1-5124abbb09db4be0955c6bc831af08fa received from {username='slave'} at 127.11.42.71:48285
I20240502 01:53:57.011230 17887 tablet_copy_service.cc:433] P bb31bceba76d466db24b9341c04900c3: ending tablet copy session 2f306549407b4bb281ccf2994035e4f1-5124abbb09db4be0955c6bc831af08fa on tablet 5124abbb09db4be0955c6bc831af08fa with peer 2f306549407b4bb281ccf2994035e4f1
I20240502 01:53:57.014235 17335 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:53:57.019099 21949 tablet_bootstrap.cc:492] T 71cbc012efe14fdb937302796a9f84e5 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:57.019520 21949 tablet_bootstrap.cc:492] T 71cbc012efe14fdb937302796a9f84e5 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap complete.
I20240502 01:53:57.019768 21949 ts_tablet_manager.cc:1367] T 71cbc012efe14fdb937302796a9f84e5 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.110s user 0.036s sys 0.020s
I20240502 01:53:57.020483 21949 raft_consensus.cc:348] T 71cbc012efe14fdb937302796a9f84e5 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } }
I20240502 01:53:57.021078 21949 raft_consensus.cc:729] T 71cbc012efe14fdb937302796a9f84e5 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: LEARNER
I20240502 01:53:57.021310 21949 consensus_queue.cc:260] T 71cbc012efe14fdb937302796a9f84e5 P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } }
I20240502 01:53:57.023695 20497 raft_consensus.cc:1261] T acee3683811241d293a7565b02f2e274 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:57.023743 17864 raft_consensus.cc:1261] T acee3683811241d293a7565b02f2e274 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:57.024509 21949 ts_tablet_manager.cc:1398] T 71cbc012efe14fdb937302796a9f84e5 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20240502 01:53:57.024487 21582 consensus_queue.cc:1035] T acee3683811241d293a7565b02f2e274 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:57.024917 21582 consensus_queue.cc:1035] T acee3683811241d293a7565b02f2e274 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:57.025046 20366 consensus_queue.cc:237] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } }
I20240502 01:53:57.025233 17757 tablet_copy_service.cc:342] P dfa1152987644bd781b46c7e42a7a685: Request end of tablet copy session 6a36c952c51542c0bf7ea3ec2857a9f9-71cbc012efe14fdb937302796a9f84e5 received from {username='slave'} at 127.11.42.70:55001
I20240502 01:53:57.025398 17757 tablet_copy_service.cc:433] P dfa1152987644bd781b46c7e42a7a685: ending tablet copy session 6a36c952c51542c0bf7ea3ec2857a9f9-71cbc012efe14fdb937302796a9f84e5 on tablet 71cbc012efe14fdb937302796a9f84e5 with peer 6a36c952c51542c0bf7ea3ec2857a9f9
I20240502 01:53:57.031343 21969 raft_consensus.cc:2932] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:53:57.034117 17736 raft_consensus.cc:2932] T f0b3862e504d4b43a60477e5efef29b1 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:53:57.036024 21861 ts_tablet_manager.cc:902] T 10ec78a08ab04678897a6892988e515f P f19cc6cb1e76460aa6711e49a5775091: Initiating tablet copy from peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:57.045552 20233 raft_consensus.cc:2932] T f0b3862e504d4b43a60477e5efef29b1 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:53:57.044912 17866 raft_consensus.cc:2932] T f0b3862e504d4b43a60477e5efef29b1 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:53:57.050283 17335 catalog_manager.cc:5484] T f0b3862e504d4b43a60477e5efef29b1 P dfa1152987644bd781b46c7e42a7a685 reported cstate change: config changed from index 4 to 5, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:53:57.066179 17335 catalog_manager.cc:5576] TS 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623) does not have the latest schema for tablet acee3683811241d293a7565b02f2e274 (table rebalancer_dynamic_table [id=492ccb5ecf704a8a878f6ef6acb0d927]). Expected version 1 got 0
I20240502 01:53:57.067070 22073 tablet.cc:1722] T acee3683811241d293a7565b02f2e274 P 6a36c952c51542c0bf7ea3ec2857a9f9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:53:57.067749 22073 tablet.cc:1620] T acee3683811241d293a7565b02f2e274 P 6a36c952c51542c0bf7ea3ec2857a9f9: MemRowSet was empty: no flush needed.
W20240502 01:53:57.068671 20300 consensus_peers.cc:485] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): Couldn't send request to peer 3fceccefba274b1dbbd8216af8b0bc29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 31a4ca0cc44e492fbd13a70ce38add35. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:57.072486 20233 raft_consensus.cc:1261] T 31a4ca0cc44e492fbd13a70ce38add35 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:57.073444 17866 raft_consensus.cc:1261] T 31a4ca0cc44e492fbd13a70ce38add35 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:57.073577 21582 consensus_queue.cc:1035] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:53:57.076671 17335 catalog_manager.cc:5576] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083) does not have the latest schema for tablet acee3683811241d293a7565b02f2e274 (table rebalancer_dynamic_table [id=492ccb5ecf704a8a878f6ef6acb0d927]). Expected version 1 got 0
I20240502 01:53:57.077399 21579 consensus_queue.cc:1035] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:53:57.082775 22077 tablet.cc:1722] T acee3683811241d293a7565b02f2e274 P bb31bceba76d466db24b9341c04900c3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:53:57.083652 22077 tablet.cc:1620] T acee3683811241d293a7565b02f2e274 P bb31bceba76d466db24b9341c04900c3: MemRowSet was empty: no flush needed.
I20240502 01:53:57.083655 21579 raft_consensus.cc:2932] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } } }
I20240502 01:53:57.088058 22075 tablet.cc:1722] T acee3683811241d293a7565b02f2e274 P 2f306549407b4bb281ccf2994035e4f1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:53:57.088732 22075 tablet.cc:1620] T acee3683811241d293a7565b02f2e274 P 2f306549407b4bb281ccf2994035e4f1: MemRowSet was empty: no flush needed.
I20240502 01:53:57.090621 17335 catalog_manager.cc:5576] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083) does not have the latest schema for tablet acee3683811241d293a7565b02f2e274 (table rebalancer_dynamic_table [id=492ccb5ecf704a8a878f6ef6acb0d927]). Expected version 1 got 0
I20240502 01:53:57.091303 17332 catalog_manager.cc:5576] TS 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623) does not have the latest schema for tablet acee3683811241d293a7565b02f2e274 (table rebalancer_dynamic_table [id=492ccb5ecf704a8a878f6ef6acb0d927]). Expected version 1 got 0
I20240502 01:53:57.091729 17334 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=492ccb5ecf704a8a878f6ef6acb0d927] alter complete (version 1)
I20240502 01:53:57.094256 21861 tablet_copy_client.cc:308] T 10ec78a08ab04678897a6892988e515f P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.66:38531
I20240502 01:53:57.094295 20231 raft_consensus.cc:2932] T 31a4ca0cc44e492fbd13a70ce38add35 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } } }
I20240502 01:53:57.099632 17627 tablet_copy_service.cc:140] P 3fceccefba274b1dbbd8216af8b0bc29: Received BeginTabletCopySession request for tablet 10ec78a08ab04678897a6892988e515f from peer f19cc6cb1e76460aa6711e49a5775091 ({username='slave'} at 127.11.42.72:53885)
I20240502 01:53:57.099815 17627 tablet_copy_service.cc:161] P 3fceccefba274b1dbbd8216af8b0bc29: Beginning new tablet copy session on tablet 10ec78a08ab04678897a6892988e515f from peer f19cc6cb1e76460aa6711e49a5775091 at {username='slave'} at 127.11.42.72:53885: session id = f19cc6cb1e76460aa6711e49a5775091-10ec78a08ab04678897a6892988e515f
I20240502 01:53:57.100695 17865 raft_consensus.cc:2932] T 31a4ca0cc44e492fbd13a70ce38add35 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } } }
I20240502 01:53:57.102563 17627 tablet_copy_source_session.cc:205] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:57.103122 21969 raft_consensus.cc:1050] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29: attempting to promote NON_VOTER f19cc6cb1e76460aa6711e49a5775091 to VOTER
I20240502 01:53:57.103215 17327 catalog_manager.cc:5484] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: config changed from index -1 to 2, NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) added. New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:57.103940 21969 consensus_queue.cc:237] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } }
I20240502 01:53:57.104336 21861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10ec78a08ab04678897a6892988e515f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:57.110877 21861 tablet_copy_client.cc:783] T 10ec78a08ab04678897a6892988e515f P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:57.111122 21861 tablet_copy_client.cc:647] T 10ec78a08ab04678897a6892988e515f P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:57.112665 21861 tablet_copy_client.cc:523] T 10ec78a08ab04678897a6892988e515f P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:57.115126 21861 tablet_bootstrap.cc:492] T 10ec78a08ab04678897a6892988e515f P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:53:57.120131 20628 raft_consensus.cc:1261] T 4aefc61cb95f4e949e7a23a1c677749e P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:57.120373 17735 raft_consensus.cc:1261] T 4aefc61cb95f4e949e7a23a1c677749e P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:57.121165 17865 raft_consensus.cc:1261] T 4aefc61cb95f4e949e7a23a1c677749e P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:57.121227 21972 consensus_queue.cc:1035] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:57.121799 21972 consensus_queue.cc:1035] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:57.122097 21973 consensus_queue.cc:1035] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:57.141290 17335 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_extra_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:57.153049 20211 tablet_service.cc:1467] Processing CreateTablet for tablet 246cec994d274786aad7672f2a30e5d8 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=448fbe791a344997b564da10a31bc5d2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:57.153350 20212 tablet_service.cc:1467] Processing CreateTablet for tablet 81f335c023444796ab933e19d809cc03 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=448fbe791a344997b564da10a31bc5d2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:57.153525 20211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 246cec994d274786aad7672f2a30e5d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:57.153577 17716 tablet_service.cc:1467] Processing CreateTablet for tablet 246cec994d274786aad7672f2a30e5d8 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=448fbe791a344997b564da10a31bc5d2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:57.153791 20212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81f335c023444796ab933e19d809cc03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:57.154029 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 246cec994d274786aad7672f2a30e5d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:57.154484 20477 tablet_service.cc:1467] Processing CreateTablet for tablet adc2b57af23c41aebd1258ff8b60d074 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=448fbe791a344997b564da10a31bc5d2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:57.154511 20476 tablet_service.cc:1467] Processing CreateTablet for tablet 246cec994d274786aad7672f2a30e5d8 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=448fbe791a344997b564da10a31bc5d2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:57.154928 20477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adc2b57af23c41aebd1258ff8b60d074. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:57.157236 20476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 246cec994d274786aad7672f2a30e5d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:57.157397 20347 tablet_service.cc:1467] Processing CreateTablet for tablet adc2b57af23c41aebd1258ff8b60d074 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=448fbe791a344997b564da10a31bc5d2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:57.157845 20347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adc2b57af23c41aebd1258ff8b60d074. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:57.159353 21982 raft_consensus.cc:1050] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29: attempting to promote NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 to VOTER
I20240502 01:53:57.160118 21982 consensus_queue.cc:237] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } }
I20240502 01:53:57.161399 20610 tablet_service.cc:1467] Processing CreateTablet for tablet adc2b57af23c41aebd1258ff8b60d074 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=448fbe791a344997b564da10a31bc5d2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:57.161837 20610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adc2b57af23c41aebd1258ff8b60d074. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:57.165434 20609 tablet_service.cc:1467] Processing CreateTablet for tablet 81f335c023444796ab933e19d809cc03 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=448fbe791a344997b564da10a31bc5d2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:57.165810 20609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81f335c023444796ab933e19d809cc03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:57.167150 21914 tablet_bootstrap.cc:492] T 246cec994d274786aad7672f2a30e5d8 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:53:57.168788 21914 tablet_bootstrap.cc:654] T 246cec994d274786aad7672f2a30e5d8 P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.181416 20477 tablet_service.cc:1467] Processing CreateTablet for tablet 81f335c023444796ab933e19d809cc03 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=448fbe791a344997b564da10a31bc5d2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:57.181984 20477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81f335c023444796ab933e19d809cc03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:57.188802 21914 tablet_bootstrap.cc:492] T 246cec994d274786aad7672f2a30e5d8 P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:53:57.189065 21914 ts_tablet_manager.cc:1367] T 246cec994d274786aad7672f2a30e5d8 P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.006s
I20240502 01:53:57.189950 21914 raft_consensus.cc:348] T 246cec994d274786aad7672f2a30e5d8 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:57.190171 21914 raft_consensus.cc:374] T 246cec994d274786aad7672f2a30e5d8 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.190261 21914 raft_consensus.cc:729] T 246cec994d274786aad7672f2a30e5d8 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.190526 21914 consensus_queue.cc:260] T 246cec994d274786aad7672f2a30e5d8 P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:57.195605 21914 ts_tablet_manager.cc:1398] T 246cec994d274786aad7672f2a30e5d8 P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 01:53:57.197299 21914 tablet_bootstrap.cc:492] T adc2b57af23c41aebd1258ff8b60d074 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:53:57.200949 22092 tablet_bootstrap.cc:492] T 246cec994d274786aad7672f2a30e5d8 P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:53:57.200961 17327 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:53:57.201943 21914 tablet_bootstrap.cc:654] T adc2b57af23c41aebd1258ff8b60d074 P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.213629 22092 tablet_bootstrap.cc:654] T 246cec994d274786aad7672f2a30e5d8 P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.216189 22073 tablet.cc:1722] T acee3683811241d293a7565b02f2e274 P 6a36c952c51542c0bf7ea3ec2857a9f9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:53:57.216683 22073 tablet.cc:1620] T acee3683811241d293a7565b02f2e274 P 6a36c952c51542c0bf7ea3ec2857a9f9: MemRowSet was empty: no flush needed.
I20240502 01:53:57.216940 22075 tablet.cc:1722] T acee3683811241d293a7565b02f2e274 P 2f306549407b4bb281ccf2994035e4f1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:53:57.217572 22075 tablet.cc:1620] T acee3683811241d293a7565b02f2e274 P 2f306549407b4bb281ccf2994035e4f1: MemRowSet was empty: no flush needed.
I20240502 01:53:57.223654 21914 tablet_bootstrap.cc:492] T adc2b57af23c41aebd1258ff8b60d074 P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:53:57.223924 21914 ts_tablet_manager.cc:1367] T adc2b57af23c41aebd1258ff8b60d074 P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.002s
I20240502 01:53:57.223959 22077 tablet.cc:1722] T acee3683811241d293a7565b02f2e274 P bb31bceba76d466db24b9341c04900c3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:53:57.224458 22077 tablet.cc:1620] T acee3683811241d293a7565b02f2e274 P bb31bceba76d466db24b9341c04900c3: MemRowSet was empty: no flush needed.
I20240502 01:53:57.224749 21914 raft_consensus.cc:348] T adc2b57af23c41aebd1258ff8b60d074 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:57.224951 21914 raft_consensus.cc:374] T adc2b57af23c41aebd1258ff8b60d074 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.225050 21914 raft_consensus.cc:729] T adc2b57af23c41aebd1258ff8b60d074 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.225392 21914 consensus_queue.cc:260] T adc2b57af23c41aebd1258ff8b60d074 P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:57.226048 21914 ts_tablet_manager.cc:1398] T adc2b57af23c41aebd1258ff8b60d074 P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:53:57.231575 17736 consensus_queue.cc:237] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } }
I20240502 01:53:57.234035 17335 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=492ccb5ecf704a8a878f6ef6acb0d927] alter complete (version 2)
I20240502 01:53:57.234148 20629 raft_consensus.cc:1261] T 5df83f83d2674155afba93c2d992af0a P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:53:57.234896 21530 consensus_queue.cc:1035] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:53:57.236552 17475 raft_consensus.cc:1261] T 5df83f83d2674155afba93c2d992af0a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:53:57.238775 22037 consensus_queue.cc:1035] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:53:57.241604 22037 raft_consensus.cc:2932] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:57.245975 21969 raft_consensus.cc:2932] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:57.246767 17864 raft_consensus.cc:2932] T 4aefc61cb95f4e949e7a23a1c677749e P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:57.248788 17475 raft_consensus.cc:2932] T 5df83f83d2674155afba93c2d992af0a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:57.252650 17736 raft_consensus.cc:2932] T 4aefc61cb95f4e949e7a23a1c677749e P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:57.253470 17335 catalog_manager.cc:5484] T 4aefc61cb95f4e949e7a23a1c677749e P bb31bceba76d466db24b9341c04900c3 reported cstate change: config changed from index 4 to 5, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:57.257982 20629 raft_consensus.cc:2932] T 5df83f83d2674155afba93c2d992af0a P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:57.268410 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5df83f83d2674155afba93c2d992af0a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:57.268545 17335 catalog_manager.cc:5484] T 5df83f83d2674155afba93c2d992af0a P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: config changed from index 5 to 6, VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) evicted. New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:57.269201 22099 tablet_bootstrap.cc:492] T 246cec994d274786aad7672f2a30e5d8 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:57.271191 22099 tablet_bootstrap.cc:654] T 246cec994d274786aad7672f2a30e5d8 P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.274951 17846 tablet_service.cc:1514] Processing DeleteTablet for tablet 5df83f83d2674155afba93c2d992af0a with delete_type TABLET_DATA_TOMBSTONED (TS bb31bceba76d466db24b9341c04900c3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47226
I20240502 01:53:57.276324 22100 ts_tablet_manager.cc:902] T 8e7f88b6c509477fb0900024e7de8f00 P fdc2ac372492454c85387589a91f63af: Initiating tablet copy from peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:53:57.281855 22095 tablet_bootstrap.cc:492] T adc2b57af23c41aebd1258ff8b60d074 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:53:57.283913 22095 tablet_bootstrap.cc:654] T adc2b57af23c41aebd1258ff8b60d074 P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.286613 22094 tablet_bootstrap.cc:492] T 81f335c023444796ab933e19d809cc03 P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:53:57.290241 22094 tablet_bootstrap.cc:654] T 81f335c023444796ab933e19d809cc03 P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.290498 22104 tablet_replica.cc:330] T 5df83f83d2674155afba93c2d992af0a P bb31bceba76d466db24b9341c04900c3: stopping tablet replica
I20240502 01:53:57.290999 22104 raft_consensus.cc:2229] T 5df83f83d2674155afba93c2d992af0a P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:57.291657 20366 raft_consensus.cc:1261] T 1fdd104f1a28446cbc25d4ca16d03d7d P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:57.292331 21634 consensus_queue.cc:1035] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:57.292546 17736 raft_consensus.cc:1261] T 1fdd104f1a28446cbc25d4ca16d03d7d P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:57.292933 22095 tablet_bootstrap.cc:492] T adc2b57af23c41aebd1258ff8b60d074 P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:53:57.293195 22095 ts_tablet_manager.cc:1367] T adc2b57af23c41aebd1258ff8b60d074 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240502 01:53:57.294108 22095 raft_consensus.cc:348] T adc2b57af23c41aebd1258ff8b60d074 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:57.294327 21634 consensus_queue.cc:1035] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:57.295382 22094 tablet_bootstrap.cc:492] T 81f335c023444796ab933e19d809cc03 P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:53:57.295564 22094 ts_tablet_manager.cc:1367] T 81f335c023444796ab933e19d809cc03 P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240502 01:53:57.298359 21914 tablet_bootstrap.cc:492] T 81f335c023444796ab933e19d809cc03 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:53:57.299356 22094 raft_consensus.cc:348] T 81f335c023444796ab933e19d809cc03 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:57.299919 21914 tablet_bootstrap.cc:654] T 81f335c023444796ab933e19d809cc03 P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.300292 22094 raft_consensus.cc:374] T 81f335c023444796ab933e19d809cc03 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.300634 22094 raft_consensus.cc:729] T 81f335c023444796ab933e19d809cc03 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.295502 17865 raft_consensus.cc:1261] T 1fdd104f1a28446cbc25d4ca16d03d7d P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:57.305348 22104 raft_consensus.cc:2258] T 5df83f83d2674155afba93c2d992af0a P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:57.306483 21982 consensus_queue.cc:1035] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:57.307302 22104 ts_tablet_manager.cc:1875] T 5df83f83d2674155afba93c2d992af0a P bb31bceba76d466db24b9341c04900c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:57.307343 21914 tablet_bootstrap.cc:492] T 81f335c023444796ab933e19d809cc03 P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:53:57.307848 21914 ts_tablet_manager.cc:1367] T 81f335c023444796ab933e19d809cc03 P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20240502 01:53:57.308473 22095 raft_consensus.cc:374] T adc2b57af23c41aebd1258ff8b60d074 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.308596 22095 raft_consensus.cc:729] T adc2b57af23c41aebd1258ff8b60d074 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.310704 20497 raft_consensus.cc:1203] T 5124abbb09db4be0955c6bc831af08fa P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:53:57.313730 21914 raft_consensus.cc:348] T 81f335c023444796ab933e19d809cc03 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:57.313925 21914 raft_consensus.cc:374] T 81f335c023444796ab933e19d809cc03 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.314003 21914 raft_consensus.cc:729] T 81f335c023444796ab933e19d809cc03 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.314254 21914 consensus_queue.cc:260] T 81f335c023444796ab933e19d809cc03 P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:57.315316 22092 tablet_bootstrap.cc:492] T 246cec994d274786aad7672f2a30e5d8 P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:53:57.315464 22092 ts_tablet_manager.cc:1367] T 246cec994d274786aad7672f2a30e5d8 P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.115s user 0.008s sys 0.016s
I20240502 01:53:57.315614 21914 ts_tablet_manager.cc:1398] T 81f335c023444796ab933e19d809cc03 P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240502 01:53:57.316185 22092 raft_consensus.cc:348] T 246cec994d274786aad7672f2a30e5d8 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:57.316385 22092 raft_consensus.cc:374] T 246cec994d274786aad7672f2a30e5d8 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.316473 22092 raft_consensus.cc:729] T 246cec994d274786aad7672f2a30e5d8 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.316488 22099 tablet_bootstrap.cc:492] T 246cec994d274786aad7672f2a30e5d8 P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:57.316745 22099 ts_tablet_manager.cc:1367] T 246cec994d274786aad7672f2a30e5d8 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.048s user 0.005s sys 0.000s
I20240502 01:53:57.316716 22092 consensus_queue.cc:260] T 246cec994d274786aad7672f2a30e5d8 P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:57.317318 22092 ts_tablet_manager.cc:1398] T 246cec994d274786aad7672f2a30e5d8 P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:57.308844 22095 consensus_queue.cc:260] T adc2b57af23c41aebd1258ff8b60d074 P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:57.318416 22094 consensus_queue.cc:260] T 81f335c023444796ab933e19d809cc03 P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:57.319975 22094 ts_tablet_manager.cc:1398] T 81f335c023444796ab933e19d809cc03 P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20240502 01:53:57.320247 22094 tablet_bootstrap.cc:492] T adc2b57af23c41aebd1258ff8b60d074 P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:53:57.323235 22092 tablet_bootstrap.cc:492] T 81f335c023444796ab933e19d809cc03 P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:53:57.327898 22092 tablet_bootstrap.cc:654] T 81f335c023444796ab933e19d809cc03 P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.328022 20628 raft_consensus.cc:2932] T 4aefc61cb95f4e949e7a23a1c677749e P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:57.329349 17605 consensus_queue.cc:237] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } }
I20240502 01:53:57.329799 22095 ts_tablet_manager.cc:1398] T adc2b57af23c41aebd1258ff8b60d074 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.036s user 0.002s sys 0.000s
I20240502 01:53:57.331596 22094 tablet_bootstrap.cc:654] T adc2b57af23c41aebd1258ff8b60d074 P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.332397 17864 raft_consensus.cc:1261] T 4aefc61cb95f4e949e7a23a1c677749e P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:53:57.332870 20629 consensus_queue.cc:237] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:53:57.334285 21982 consensus_queue.cc:1035] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:53:57.334545 22099 raft_consensus.cc:348] T 246cec994d274786aad7672f2a30e5d8 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:57.334779 22099 raft_consensus.cc:374] T 246cec994d274786aad7672f2a30e5d8 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.334878 22099 raft_consensus.cc:729] T 246cec994d274786aad7672f2a30e5d8 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.336687 17335 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:53:57.336700 21634 raft_consensus.cc:2932] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:57.337626 22092 tablet_bootstrap.cc:492] T 81f335c023444796ab933e19d809cc03 P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:53:57.338649 21429 raft_consensus.cc:482] T 81f335c023444796ab933e19d809cc03 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:57.338842 21429 raft_consensus.cc:504] T 81f335c023444796ab933e19d809cc03 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:57.339485 21429 leader_election.cc:288] T 81f335c023444796ab933e19d809cc03 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:53:57.339980 22092 ts_tablet_manager.cc:1367] T 81f335c023444796ab933e19d809cc03 P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.001s
I20240502 01:53:57.340698 17319 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4aefc61cb95f4e949e7a23a1c677749e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:57.340507 21972 raft_consensus.cc:2932] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:53:57.341581 22099 consensus_queue.cc:260] T 246cec994d274786aad7672f2a30e5d8 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:57.341780 20497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81f335c023444796ab933e19d809cc03" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
I20240502 01:53:57.342011 20497 raft_consensus.cc:2454] T 81f335c023444796ab933e19d809cc03 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 0.
I20240502 01:53:57.344223 22099 ts_tablet_manager.cc:1398] T 246cec994d274786aad7672f2a30e5d8 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20240502 01:53:57.345559 20565 leader_election.cc:302] T 81f335c023444796ab933e19d809cc03 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:53:57.345885 21429 raft_consensus.cc:2781] T 81f335c023444796ab933e19d809cc03 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:57.346002 21429 raft_consensus.cc:482] T 81f335c023444796ab933e19d809cc03 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:57.346086 21429 raft_consensus.cc:3037] T 81f335c023444796ab933e19d809cc03 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.346714 17864 raft_consensus.cc:2932] T 4aefc61cb95f4e949e7a23a1c677749e P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:57.347462 17333 catalog_manager.cc:5484] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: config changed from index 5 to 6, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:57.348600 21429 raft_consensus.cc:504] T 81f335c023444796ab933e19d809cc03 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:57.349218 21429 leader_election.cc:288] T 81f335c023444796ab933e19d809cc03 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 election: Requested vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:53:57.350247 20498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81f335c023444796ab933e19d809cc03" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
I20240502 01:53:57.350447 20498 raft_consensus.cc:3037] T 81f335c023444796ab933e19d809cc03 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.352371 21861 tablet_bootstrap.cc:492] T 10ec78a08ab04678897a6892988e515f P f19cc6cb1e76460aa6711e49a5775091: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:57.352744 21861 tablet_bootstrap.cc:492] T 10ec78a08ab04678897a6892988e515f P f19cc6cb1e76460aa6711e49a5775091: Bootstrap complete.
I20240502 01:53:57.353068 21861 ts_tablet_manager.cc:1367] T 10ec78a08ab04678897a6892988e515f P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.238s user 0.048s sys 0.001s
I20240502 01:53:57.353788 22073 tablet.cc:1722] T acee3683811241d293a7565b02f2e274 P 6a36c952c51542c0bf7ea3ec2857a9f9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:53:57.354043 21861 raft_consensus.cc:348] T 10ec78a08ab04678897a6892988e515f P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:53:57.354156 20630 raft_consensus.cc:1261] T 4aefc61cb95f4e949e7a23a1c677749e P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:53:57.354286 21861 raft_consensus.cc:729] T 10ec78a08ab04678897a6892988e515f P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: LEARNER
I20240502 01:53:57.354290 22077 tablet.cc:1722] T acee3683811241d293a7565b02f2e274 P bb31bceba76d466db24b9341c04900c3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:53:57.354481 21861 consensus_queue.cc:260] T 10ec78a08ab04678897a6892988e515f P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:53:57.355674 20498 raft_consensus.cc:2454] T 81f335c023444796ab933e19d809cc03 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 1.
I20240502 01:53:57.356348 21982 consensus_queue.cc:1035] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:53:57.357076 21861 ts_tablet_manager.cc:1398] T 10ec78a08ab04678897a6892988e515f P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:53:57.357348 17716 tablet_service.cc:1514] Processing DeleteTablet for tablet 4aefc61cb95f4e949e7a23a1c677749e with delete_type TABLET_DATA_TOMBSTONED (TS dfa1152987644bd781b46c7e42a7a685 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52198
I20240502 01:53:57.357993 20565 leader_election.cc:302] T 81f335c023444796ab933e19d809cc03 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:53:57.358294 21429 raft_consensus.cc:2781] T 81f335c023444796ab933e19d809cc03 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:57.358430 21429 raft_consensus.cc:686] T 81f335c023444796ab933e19d809cc03 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Becoming Leader. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Running, Role: LEADER
I20240502 01:53:57.358673 21429 consensus_queue.cc:237] T 81f335c023444796ab933e19d809cc03 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:57.360164 17627 tablet_copy_service.cc:342] P 3fceccefba274b1dbbd8216af8b0bc29: Request end of tablet copy session f19cc6cb1e76460aa6711e49a5775091-10ec78a08ab04678897a6892988e515f received from {username='slave'} at 127.11.42.72:53885
I20240502 01:53:57.360327 17627 tablet_copy_service.cc:433] P 3fceccefba274b1dbbd8216af8b0bc29: ending tablet copy session f19cc6cb1e76460aa6711e49a5775091-10ec78a08ab04678897a6892988e515f on tablet 10ec78a08ab04678897a6892988e515f with peer f19cc6cb1e76460aa6711e49a5775091
I20240502 01:53:57.360782 22075 tablet.cc:1722] T acee3683811241d293a7565b02f2e274 P 2f306549407b4bb281ccf2994035e4f1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:53:57.363093 21697 raft_consensus.cc:482] T 246cec994d274786aad7672f2a30e5d8 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:57.363051 20630 raft_consensus.cc:2932] T 4aefc61cb95f4e949e7a23a1c677749e P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:57.363265 21697 raft_consensus.cc:504] T 246cec994d274786aad7672f2a30e5d8 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:57.363255 17865 raft_consensus.cc:2932] T 1fdd104f1a28446cbc25d4ca16d03d7d P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:53:57.363940 21697 leader_election.cc:288] T 246cec994d274786aad7672f2a30e5d8 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:53:57.364464 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "246cec994d274786aad7672f2a30e5d8" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:53:57.364655 17735 raft_consensus.cc:2454] T 246cec994d274786aad7672f2a30e5d8 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 0.
I20240502 01:53:57.365094 20433 leader_election.cc:302] T 246cec994d274786aad7672f2a30e5d8 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:57.366021 21697 raft_consensus.cc:2781] T 246cec994d274786aad7672f2a30e5d8 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:57.366220 21697 raft_consensus.cc:482] T 246cec994d274786aad7672f2a30e5d8 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:57.366315 21697 raft_consensus.cc:3037] T 246cec994d274786aad7672f2a30e5d8 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.366873 17736 raft_consensus.cc:2932] T 1fdd104f1a28446cbc25d4ca16d03d7d P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:53:57.368395 20365 raft_consensus.cc:1203] T 71cbc012efe14fdb937302796a9f84e5 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:53:57.368448 17333 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table_new_name [id=492ccb5ecf704a8a878f6ef6acb0d927] alter complete (version 3)
I20240502 01:53:57.369556 21697 raft_consensus.cc:504] T 246cec994d274786aad7672f2a30e5d8 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:57.370040 17327 catalog_manager.cc:5484] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: config changed from index 4 to 5, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:57.370975 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "246cec994d274786aad7672f2a30e5d8" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:53:57.371155 17735 raft_consensus.cc:3037] T 246cec994d274786aad7672f2a30e5d8 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.344102 22100 tablet_copy_client.cc:308] T 8e7f88b6c509477fb0900024e7de8f00 P fdc2ac372492454c85387589a91f63af: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.67:36115
I20240502 01:53:57.346241 20232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81f335c023444796ab933e19d809cc03" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:53:57.374179 17735 raft_consensus.cc:2454] T 246cec994d274786aad7672f2a30e5d8 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 1.
I20240502 01:53:57.374197 17757 tablet_copy_service.cc:140] P dfa1152987644bd781b46c7e42a7a685: Received BeginTabletCopySession request for tablet 8e7f88b6c509477fb0900024e7de8f00 from peer fdc2ac372492454c85387589a91f63af ({username='slave'} at 127.11.42.69:44021)
I20240502 01:53:57.374141 22092 raft_consensus.cc:348] T 81f335c023444796ab933e19d809cc03 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:57.374367 22092 raft_consensus.cc:374] T 81f335c023444796ab933e19d809cc03 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.374389 17757 tablet_copy_service.cc:161] P dfa1152987644bd781b46c7e42a7a685: Beginning new tablet copy session on tablet 8e7f88b6c509477fb0900024e7de8f00 from peer fdc2ac372492454c85387589a91f63af at {username='slave'} at 127.11.42.69:44021: session id = fdc2ac372492454c85387589a91f63af-8e7f88b6c509477fb0900024e7de8f00
I20240502 01:53:57.374449 22092 raft_consensus.cc:729] T 81f335c023444796ab933e19d809cc03 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.374673 22092 consensus_queue.cc:260] T 81f335c023444796ab933e19d809cc03 P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:57.374843 20433 leader_election.cc:302] T 246cec994d274786aad7672f2a30e5d8 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:57.375098 21924 raft_consensus.cc:2781] T 246cec994d274786aad7672f2a30e5d8 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:57.375229 21924 raft_consensus.cc:686] T 246cec994d274786aad7672f2a30e5d8 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Running, Role: LEADER
I20240502 01:53:57.375447 21924 consensus_queue.cc:237] T 246cec994d274786aad7672f2a30e5d8 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:57.379204 22104 ts_tablet_manager.cc:1888] T 5df83f83d2674155afba93c2d992af0a P bb31bceba76d466db24b9341c04900c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:53:57.379338 22104 log.cc:1196] T 5df83f83d2674155afba93c2d992af0a P bb31bceba76d466db24b9341c04900c3: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/wals/5df83f83d2674155afba93c2d992af0a
I20240502 01:53:57.380091 17322 catalog_manager.cc:4830] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): tablet 5df83f83d2674155afba93c2d992af0a (table rebalance_test_table_7 [id=c6c3a53b24874f31aa38118254253dcb]) successfully deleted
W20240502 01:53:57.380348 20565 leader_election.cc:341] T 81f335c023444796ab933e19d809cc03 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:53:57.383563 17757 tablet_copy_source_session.cc:205] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:57.383935 20365 raft_consensus.cc:2932] T 1fdd104f1a28446cbc25d4ca16d03d7d P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:53:57.386808 22100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e7f88b6c509477fb0900024e7de8f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:57.386329 17327 catalog_manager.cc:5484] T 81f335c023444796ab933e19d809cc03 P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: term changed from 0 to 1, leader changed from <none> to f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72). New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:57.389474 17605 consensus_queue.cc:237] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } }
I20240502 01:53:57.392089 17735 raft_consensus.cc:1261] T 1fdd104f1a28446cbc25d4ca16d03d7d P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:53:57.393191 20366 raft_consensus.cc:1261] T 1fdd104f1a28446cbc25d4ca16d03d7d P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:53:57.393680 21972 consensus_queue.cc:1035] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:53:57.396517 22092 ts_tablet_manager.cc:1398] T 81f335c023444796ab933e19d809cc03 P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.023s user 0.000s sys 0.002s
I20240502 01:53:57.396088 21634 consensus_queue.cc:1035] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240502 01:53:57.396687 20232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "246cec994d274786aad7672f2a30e5d8" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:53:57.396932 20232 raft_consensus.cc:3037] T 246cec994d274786aad7672f2a30e5d8 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.399746 20232 raft_consensus.cc:2454] T 246cec994d274786aad7672f2a30e5d8 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 1.
I20240502 01:53:57.399917 21972 raft_consensus.cc:2932] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:53:57.400540 22100 tablet_copy_client.cc:783] T 8e7f88b6c509477fb0900024e7de8f00 P fdc2ac372492454c85387589a91f63af: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:57.400719 22100 tablet_copy_client.cc:647] T 8e7f88b6c509477fb0900024e7de8f00 P fdc2ac372492454c85387589a91f63af: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:57.401989 21697 leader_election.cc:288] T 246cec994d274786aad7672f2a30e5d8 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 election: Requested vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:53:57.360893 20231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81f335c023444796ab933e19d809cc03" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:53:57.365840 20233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "246cec994d274786aad7672f2a30e5d8" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:53:57.403923 20233 raft_consensus.cc:2362] T 246cec994d274786aad7672f2a30e5d8 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 1. Re-sending same reply.
I20240502 01:53:57.404155 17319 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1fdd104f1a28446cbc25d4ca16d03d7d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:57.404536 22100 tablet_copy_client.cc:523] T 8e7f88b6c509477fb0900024e7de8f00 P fdc2ac372492454c85387589a91f63af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:57.404549 20231 raft_consensus.cc:3037] T 81f335c023444796ab933e19d809cc03 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.405061 17735 raft_consensus.cc:2932] T 1fdd104f1a28446cbc25d4ca16d03d7d P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:53:57.405722 17334 catalog_manager.cc:5484] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: config changed from index 5 to 6, VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) evicted. New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:57.410539 22100 tablet_bootstrap.cc:492] T 8e7f88b6c509477fb0900024e7de8f00 P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:53:57.414985 17846 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fdd104f1a28446cbc25d4ca16d03d7d with delete_type TABLET_DATA_TOMBSTONED (TS bb31bceba76d466db24b9341c04900c3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47226
I20240502 01:53:57.414753 20366 raft_consensus.cc:2932] T 1fdd104f1a28446cbc25d4ca16d03d7d P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:53:57.415683 22104 tablet_replica.cc:330] T 1fdd104f1a28446cbc25d4ca16d03d7d P bb31bceba76d466db24b9341c04900c3: stopping tablet replica
I20240502 01:53:57.415930 22104 raft_consensus.cc:2229] T 1fdd104f1a28446cbc25d4ca16d03d7d P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:57.416244 22104 raft_consensus.cc:2258] T 1fdd104f1a28446cbc25d4ca16d03d7d P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:57.417380 22104 ts_tablet_manager.cc:1875] T 1fdd104f1a28446cbc25d4ca16d03d7d P bb31bceba76d466db24b9341c04900c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:57.424415 22104 ts_tablet_manager.cc:1888] T 1fdd104f1a28446cbc25d4ca16d03d7d P bb31bceba76d466db24b9341c04900c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:53:57.425704 20231 raft_consensus.cc:2454] T 81f335c023444796ab933e19d809cc03 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 1.
I20240502 01:53:57.426878 22104 log.cc:1196] T 1fdd104f1a28446cbc25d4ca16d03d7d P bb31bceba76d466db24b9341c04900c3: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/wals/1fdd104f1a28446cbc25d4ca16d03d7d
I20240502 01:53:57.427788 17322 catalog_manager.cc:4830] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): tablet 1fdd104f1a28446cbc25d4ca16d03d7d (table rebalance_test_table_13 [id=dbf97af96f07462286eb1fabeddf25fa]) successfully deleted
I20240502 01:53:57.431720 22094 tablet_bootstrap.cc:492] T adc2b57af23c41aebd1258ff8b60d074 P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:53:57.431906 22094 ts_tablet_manager.cc:1367] T adc2b57af23c41aebd1258ff8b60d074 P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.112s user 0.000s sys 0.008s
I20240502 01:53:57.432662 22094 raft_consensus.cc:348] T adc2b57af23c41aebd1258ff8b60d074 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:57.432888 22094 raft_consensus.cc:374] T adc2b57af23c41aebd1258ff8b60d074 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.432957 22094 raft_consensus.cc:729] T adc2b57af23c41aebd1258ff8b60d074 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.433212 22094 consensus_queue.cc:260] T adc2b57af23c41aebd1258ff8b60d074 P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:57.433730 22094 ts_tablet_manager.cc:1398] T adc2b57af23c41aebd1258ff8b60d074 P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:57.439970 17334 catalog_manager.cc:5484] T 246cec994d274786aad7672f2a30e5d8 P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71). New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:57.446183 20497 raft_consensus.cc:1261] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:57.447609 21932 consensus_queue.cc:1035] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:53:57.449748 20365 raft_consensus.cc:1261] T cf0237a4c65b495ba892871dd850bed9 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:53:57.452268 20564 consensus_peers.cc:485] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cf0237a4c65b495ba892871dd850bed9. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:57.453059 21429 consensus_queue.cc:1035] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:53:57.459321 22122 raft_consensus.cc:1050] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1: attempting to promote NON_VOTER 4da690c0fa13407a85c0689ce8de9ade to VOTER
I20240502 01:53:57.459787 22119 tablet_replica.cc:330] T 4aefc61cb95f4e949e7a23a1c677749e P dfa1152987644bd781b46c7e42a7a685: stopping tablet replica
I20240502 01:53:57.453061 20629 raft_consensus.cc:1203] T 10ec78a08ab04678897a6892988e515f P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:53:57.460402 22122 consensus_queue.cc:237] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } }
I20240502 01:53:57.464236 22119 raft_consensus.cc:2229] T 4aefc61cb95f4e949e7a23a1c677749e P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:57.464753 22119 raft_consensus.cc:2258] T 4aefc61cb95f4e949e7a23a1c677749e P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:57.471294 22119 ts_tablet_manager.cc:1875] T 4aefc61cb95f4e949e7a23a1c677749e P dfa1152987644bd781b46c7e42a7a685: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:57.480365 17327 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:53:57.480556 17327 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:53:57.481557 22100 tablet_bootstrap.cc:492] T 8e7f88b6c509477fb0900024e7de8f00 P fdc2ac372492454c85387589a91f63af: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:57.481966 22100 tablet_bootstrap.cc:492] T 8e7f88b6c509477fb0900024e7de8f00 P fdc2ac372492454c85387589a91f63af: Bootstrap complete.
I20240502 01:53:57.482240 22100 ts_tablet_manager.cc:1367] T 8e7f88b6c509477fb0900024e7de8f00 P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.072s user 0.052s sys 0.000s
I20240502 01:53:57.483824 17327 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet acee3683811241d293a7565b02f2e274
I20240502 01:53:57.484921 20347 tablet_service.cc:1514] Processing DeleteTablet for tablet acee3683811241d293a7565b02f2e274 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:57 UTC) from {username='slave'} at 127.0.0.1:40196
I20240502 01:53:57.485952 22100 raft_consensus.cc:348] T 8e7f88b6c509477fb0900024e7de8f00 P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } }
I20240502 01:53:57.486204 22100 raft_consensus.cc:729] T 8e7f88b6c509477fb0900024e7de8f00 P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: LEARNER
I20240502 01:53:57.486223 17846 tablet_service.cc:1514] Processing DeleteTablet for tablet acee3683811241d293a7565b02f2e274 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:57 UTC) from {username='slave'} at 127.0.0.1:47226
I20240502 01:53:57.486507 22104 tablet_replica.cc:330] T acee3683811241d293a7565b02f2e274 P bb31bceba76d466db24b9341c04900c3: stopping tablet replica
I20240502 01:53:57.486436 22100 consensus_queue.cc:260] T 8e7f88b6c509477fb0900024e7de8f00 P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } }
I20240502 01:53:57.486707 22104 raft_consensus.cc:2229] T acee3683811241d293a7565b02f2e274 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:57.486938 22104 raft_consensus.cc:2258] T acee3683811241d293a7565b02f2e274 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:57.487016 22100 ts_tablet_manager.cc:1398] T 8e7f88b6c509477fb0900024e7de8f00 P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:53:57.487936 22104 ts_tablet_manager.cc:1875] T acee3683811241d293a7565b02f2e274 P bb31bceba76d466db24b9341c04900c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:57.489918 20477 tablet_service.cc:1514] Processing DeleteTablet for tablet acee3683811241d293a7565b02f2e274 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:57 UTC) from {username='slave'} at 127.0.0.1:35018
I20240502 01:53:57.490669 22128 tablet_replica.cc:330] T acee3683811241d293a7565b02f2e274 P 2f306549407b4bb281ccf2994035e4f1: stopping tablet replica
I20240502 01:53:57.490924 22128 raft_consensus.cc:2229] T acee3683811241d293a7565b02f2e274 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:57.491242 22128 raft_consensus.cc:2258] T acee3683811241d293a7565b02f2e274 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:57.492038 22128 ts_tablet_manager.cc:1875] T acee3683811241d293a7565b02f2e274 P 2f306549407b4bb281ccf2994035e4f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:57.492446 17757 tablet_copy_service.cc:342] P dfa1152987644bd781b46c7e42a7a685: Request end of tablet copy session fdc2ac372492454c85387589a91f63af-8e7f88b6c509477fb0900024e7de8f00 received from {username='slave'} at 127.11.42.69:44021
I20240502 01:53:57.492558 17757 tablet_copy_service.cc:433] P dfa1152987644bd781b46c7e42a7a685: ending tablet copy session fdc2ac372492454c85387589a91f63af-8e7f88b6c509477fb0900024e7de8f00 on tablet 8e7f88b6c509477fb0900024e7de8f00 with peer fdc2ac372492454c85387589a91f63af
I20240502 01:53:57.494131 22104 ts_tablet_manager.cc:1888] T acee3683811241d293a7565b02f2e274 P bb31bceba76d466db24b9341c04900c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:53:57.494258 22104 log.cc:1196] T acee3683811241d293a7565b02f2e274 P bb31bceba76d466db24b9341c04900c3: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/wals/acee3683811241d293a7565b02f2e274
I20240502 01:53:57.494742 22104 ts_tablet_manager.cc:1909] T acee3683811241d293a7565b02f2e274 P bb31bceba76d466db24b9341c04900c3: Deleting consensus metadata
I20240502 01:53:57.495702 17322 catalog_manager.cc:4830] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): tablet acee3683811241d293a7565b02f2e274 (table rebalancer_dynamic_table_new_name [id=492ccb5ecf704a8a878f6ef6acb0d927]) successfully deleted
I20240502 01:53:57.497490 21429 raft_consensus.cc:2932] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:53:57.501554 22128 ts_tablet_manager.cc:1888] T acee3683811241d293a7565b02f2e274 P 2f306549407b4bb281ccf2994035e4f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:53:57.501786 22128 log.cc:1196] T acee3683811241d293a7565b02f2e274 P 2f306549407b4bb281ccf2994035e4f1: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal/wals/acee3683811241d293a7565b02f2e274
I20240502 01:53:57.502291 22128 ts_tablet_manager.cc:1909] T acee3683811241d293a7565b02f2e274 P 2f306549407b4bb281ccf2994035e4f1: Deleting consensus metadata
I20240502 01:53:57.504770 17322 catalog_manager.cc:4830] TS 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): tablet acee3683811241d293a7565b02f2e274 (table rebalancer_dynamic_table_new_name [id=492ccb5ecf704a8a878f6ef6acb0d927]) successfully deleted
I20240502 01:53:57.510771 17334 catalog_manager.cc:5484] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:57.513518 20233 raft_consensus.cc:1261] T 617443200e2c47b69d63ef056294f40c P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:53:57.514097 21697 consensus_queue.cc:1035] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:53:57.516424 22119 ts_tablet_manager.cc:1888] T 4aefc61cb95f4e949e7a23a1c677749e P dfa1152987644bd781b46c7e42a7a685: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:53:57.516614 22119 log.cc:1196] T 4aefc61cb95f4e949e7a23a1c677749e P dfa1152987644bd781b46c7e42a7a685: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/wals/4aefc61cb95f4e949e7a23a1c677749e
I20240502 01:53:57.517516 17321 catalog_manager.cc:4830] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): tablet 4aefc61cb95f4e949e7a23a1c677749e (table rebalance_test_table_13 [id=dbf97af96f07462286eb1fabeddf25fa]) successfully deleted
I20240502 01:53:57.522840 20497 raft_consensus.cc:2932] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:53:57.524808 17475 raft_consensus.cc:1261] T 617443200e2c47b69d63ef056294f40c P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEARNER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:53:57.525918 20628 raft_consensus.cc:1261] T 617443200e2c47b69d63ef056294f40c P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:53:57.527451 21924 consensus_queue.cc:1035] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:53:57.529749 20365 raft_consensus.cc:2932] T cf0237a4c65b495ba892871dd850bed9 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:53:57.536994 21924 raft_consensus.cc:482] T adc2b57af23c41aebd1258ff8b60d074 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:57.537169 21924 raft_consensus.cc:504] T adc2b57af23c41aebd1258ff8b60d074 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:57.537741 21924 leader_election.cc:288] T adc2b57af23c41aebd1258ff8b60d074 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:53:57.538426 21924 consensus_queue.cc:1035] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:53:57.539726 20365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adc2b57af23c41aebd1258ff8b60d074" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:53:57.540019 20365 raft_consensus.cc:2454] T adc2b57af23c41aebd1258ff8b60d074 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 0.
I20240502 01:53:57.541584 20628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adc2b57af23c41aebd1258ff8b60d074" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091" is_pre_election: true
I20240502 01:53:57.541775 20628 raft_consensus.cc:2454] T adc2b57af23c41aebd1258ff8b60d074 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 0.
I20240502 01:53:57.552654 20433 leader_election.cc:302] T adc2b57af23c41aebd1258ff8b60d074 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:53:57.552968 21924 raft_consensus.cc:2781] T adc2b57af23c41aebd1258ff8b60d074 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:57.553121 21924 raft_consensus.cc:482] T adc2b57af23c41aebd1258ff8b60d074 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:57.553257 21924 raft_consensus.cc:3037] T adc2b57af23c41aebd1258ff8b60d074 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.557246 21924 raft_consensus.cc:504] T adc2b57af23c41aebd1258ff8b60d074 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:57.557968 21924 leader_election.cc:288] T adc2b57af23c41aebd1258ff8b60d074 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 election: Requested vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:53:57.560032 20365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adc2b57af23c41aebd1258ff8b60d074" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
I20240502 01:53:57.560276 20365 raft_consensus.cc:3037] T adc2b57af23c41aebd1258ff8b60d074 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.564772 20365 raft_consensus.cc:2454] T adc2b57af23c41aebd1258ff8b60d074 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 1.
I20240502 01:53:57.568826 20630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adc2b57af23c41aebd1258ff8b60d074" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091"
I20240502 01:53:57.569069 20630 raft_consensus.cc:3037] T adc2b57af23c41aebd1258ff8b60d074 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.571704 20630 raft_consensus.cc:2454] T adc2b57af23c41aebd1258ff8b60d074 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 1.
I20240502 01:53:57.572258 20433 leader_election.cc:302] T adc2b57af23c41aebd1258ff8b60d074 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:53:57.576295 22134 ts_tablet_manager.cc:902] T 31a4ca0cc44e492fbd13a70ce38add35 P 3fceccefba274b1dbbd8216af8b0bc29: Initiating tablet copy from peer 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:53:57.578711 21697 raft_consensus.cc:2781] T adc2b57af23c41aebd1258ff8b60d074 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:57.578864 21697 raft_consensus.cc:686] T adc2b57af23c41aebd1258ff8b60d074 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Running, Role: LEADER
I20240502 01:53:57.579106 21697 consensus_queue.cc:237] T adc2b57af23c41aebd1258ff8b60d074 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:57.587819 17334 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:53:57.595486 17327 catalog_manager.cc:5484] T adc2b57af23c41aebd1258ff8b60d074 P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71). New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:57.596307 17716 tablet_service.cc:1467] Processing CreateTablet for tablet 7131c9bab38e45fa984a84ea6e21dcc5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=624bf9e5554748a88cb806a0e83f5ad4]), partition=
I20240502 01:53:57.596701 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7131c9bab38e45fa984a84ea6e21dcc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:57.601982 20477 tablet_service.cc:1467] Processing CreateTablet for tablet 7131c9bab38e45fa984a84ea6e21dcc5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=624bf9e5554748a88cb806a0e83f5ad4]), partition=
I20240502 01:53:57.602322 20477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7131c9bab38e45fa984a84ea6e21dcc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:57.605237 21697 raft_consensus.cc:2932] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:53:57.605695 20211 tablet_service.cc:1467] Processing CreateTablet for tablet 7131c9bab38e45fa984a84ea6e21dcc5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=624bf9e5554748a88cb806a0e83f5ad4]), partition=
I20240502 01:53:57.606031 20211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7131c9bab38e45fa984a84ea6e21dcc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:57.610101 17474 raft_consensus.cc:2932] T 617443200e2c47b69d63ef056294f40c P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:53:57.614879 22127 tablet_replica.cc:330] T acee3683811241d293a7565b02f2e274 P 6a36c952c51542c0bf7ea3ec2857a9f9: stopping tablet replica
I20240502 01:53:57.615101 22127 raft_consensus.cc:2229] T acee3683811241d293a7565b02f2e274 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:57.615468 22127 raft_consensus.cc:2258] T acee3683811241d293a7565b02f2e274 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:57.616201 17327 catalog_manager.cc:5484] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:57.620273 22132 ts_tablet_manager.cc:902] T 03bd47783e884a70bfea1dfab4f2e563 P 2f306549407b4bb281ccf2994035e4f1: Initiating tablet copy from peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:53:57.628756 22092 tablet_bootstrap.cc:492] T 7131c9bab38e45fa984a84ea6e21dcc5 P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:53:57.632058 22092 tablet_bootstrap.cc:654] T 7131c9bab38e45fa984a84ea6e21dcc5 P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.632620 22127 ts_tablet_manager.cc:1875] T acee3683811241d293a7565b02f2e274 P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:57.632357 20628 raft_consensus.cc:2932] T 617443200e2c47b69d63ef056294f40c P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:53:57.635550 20498 consensus_queue.cc:237] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } }
I20240502 01:53:57.639354 17474 raft_consensus.cc:1261] T 617443200e2c47b69d63ef056294f40c P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:57.639300 20231 raft_consensus.cc:2932] T 617443200e2c47b69d63ef056294f40c P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:53:57.642350 20628 raft_consensus.cc:1261] T 617443200e2c47b69d63ef056294f40c P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:57.643213 21924 consensus_queue.cc:1035] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:57.643957 21924 consensus_queue.cc:1035] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:57.647394 21216 raft_consensus.cc:2932] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:53:57.651377 17322 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 617443200e2c47b69d63ef056294f40c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:57.653693 21914 tablet_bootstrap.cc:492] T 7131c9bab38e45fa984a84ea6e21dcc5 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:53:57.654675 17334 catalog_manager.cc:5484] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: config changed from index 3 to 4, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:57.655393 21914 tablet_bootstrap.cc:654] T 7131c9bab38e45fa984a84ea6e21dcc5 P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.658391 17474 raft_consensus.cc:2932] T 617443200e2c47b69d63ef056294f40c P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:53:57.665666 20628 raft_consensus.cc:2932] T 617443200e2c47b69d63ef056294f40c P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:53:57.666100 22099 tablet_bootstrap.cc:492] T 7131c9bab38e45fa984a84ea6e21dcc5 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:57.667748 22099 tablet_bootstrap.cc:654] T 7131c9bab38e45fa984a84ea6e21dcc5 P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:57.674234 22127 ts_tablet_manager.cc:1888] T acee3683811241d293a7565b02f2e274 P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:53:57.674430 22127 log.cc:1196] T acee3683811241d293a7565b02f2e274 P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal/wals/acee3683811241d293a7565b02f2e274
I20240502 01:53:57.675066 22127 ts_tablet_manager.cc:1909] T acee3683811241d293a7565b02f2e274 P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting consensus metadata
I20240502 01:53:57.676297 17321 catalog_manager.cc:4830] TS 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): tablet acee3683811241d293a7565b02f2e274 (table rebalancer_dynamic_table_new_name [id=492ccb5ecf704a8a878f6ef6acb0d927]) successfully deleted
I20240502 01:53:57.683298 20211 tablet_service.cc:1514] Processing DeleteTablet for tablet 617443200e2c47b69d63ef056294f40c with delete_type TABLET_DATA_TOMBSTONED (TS fdc2ac372492454c85387589a91f63af not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44624
I20240502 01:53:57.694679 20232 raft_consensus.cc:1203] T 8e7f88b6c509477fb0900024e7de8f00 P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:53:57.696413 20365 consensus_queue.cc:237] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:53:57.730779 22150 tablet_replica.cc:330] T 617443200e2c47b69d63ef056294f40c P fdc2ac372492454c85387589a91f63af: stopping tablet replica
I20240502 01:53:57.731065 22150 raft_consensus.cc:2229] T 617443200e2c47b69d63ef056294f40c P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:57.731551 22150 raft_consensus.cc:2258] T 617443200e2c47b69d63ef056294f40c P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:57.738792 21914 tablet_bootstrap.cc:492] T 7131c9bab38e45fa984a84ea6e21dcc5 P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:53:57.739030 21914 ts_tablet_manager.cc:1367] T 7131c9bab38e45fa984a84ea6e21dcc5 P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.085s user 0.006s sys 0.006s
I20240502 01:53:57.739786 21914 raft_consensus.cc:348] T 7131c9bab38e45fa984a84ea6e21dcc5 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:57.740144 21914 raft_consensus.cc:374] T 7131c9bab38e45fa984a84ea6e21dcc5 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.740283 21914 raft_consensus.cc:729] T 7131c9bab38e45fa984a84ea6e21dcc5 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.740620 21914 consensus_queue.cc:260] T 7131c9bab38e45fa984a84ea6e21dcc5 P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:57.741541 21914 ts_tablet_manager.cc:1398] T 7131c9bab38e45fa984a84ea6e21dcc5 P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:53:57.742498 22092 tablet_bootstrap.cc:492] T 7131c9bab38e45fa984a84ea6e21dcc5 P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:53:57.742691 22092 ts_tablet_manager.cc:1367] T 7131c9bab38e45fa984a84ea6e21dcc5 P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.114s user 0.009s sys 0.003s
I20240502 01:53:57.743451 22092 raft_consensus.cc:348] T 7131c9bab38e45fa984a84ea6e21dcc5 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:57.743693 22092 raft_consensus.cc:374] T 7131c9bab38e45fa984a84ea6e21dcc5 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.743805 22092 raft_consensus.cc:729] T 7131c9bab38e45fa984a84ea6e21dcc5 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.744166 22092 consensus_queue.cc:260] T 7131c9bab38e45fa984a84ea6e21dcc5 P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:57.753587 22099 tablet_bootstrap.cc:492] T 7131c9bab38e45fa984a84ea6e21dcc5 P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:57.753803 22099 ts_tablet_manager.cc:1367] T 7131c9bab38e45fa984a84ea6e21dcc5 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.088s user 0.008s sys 0.004s
I20240502 01:53:57.754559 22099 raft_consensus.cc:348] T 7131c9bab38e45fa984a84ea6e21dcc5 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:57.754892 22099 raft_consensus.cc:374] T 7131c9bab38e45fa984a84ea6e21dcc5 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:57.755013 22099 raft_consensus.cc:729] T 7131c9bab38e45fa984a84ea6e21dcc5 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:57.755307 22099 consensus_queue.cc:260] T 7131c9bab38e45fa984a84ea6e21dcc5 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:57.756016 22099 ts_tablet_manager.cc:1398] T 7131c9bab38e45fa984a84ea6e21dcc5 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:57.758369 22150 ts_tablet_manager.cc:1875] T 617443200e2c47b69d63ef056294f40c P fdc2ac372492454c85387589a91f63af: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:57.760344 22092 ts_tablet_manager.cc:1398] T 7131c9bab38e45fa984a84ea6e21dcc5 P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
W20240502 01:53:57.760605 20302 consensus_peers.cc:485] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a212c17ba64842c99b3d6a78a3b7cc8a. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:57.762039 17474 raft_consensus.cc:1261] T a212c17ba64842c99b3d6a78a3b7cc8a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:57.762622 22081 consensus_queue.cc:1035] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:53:57.771818 20231 raft_consensus.cc:1261] T a212c17ba64842c99b3d6a78a3b7cc8a P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:57.772331 22081 consensus_queue.cc:1035] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:53:57.777304 22081 raft_consensus.cc:2932] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:53:57.782565 17475 raft_consensus.cc:2932] T a212c17ba64842c99b3d6a78a3b7cc8a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:53:57.787868 22134 tablet_copy_client.cc:308] T 31a4ca0cc44e492fbd13a70ce38add35 P 3fceccefba274b1dbbd8216af8b0bc29: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.70:39155
I20240502 01:53:57.788038 22150 ts_tablet_manager.cc:1888] T 617443200e2c47b69d63ef056294f40c P fdc2ac372492454c85387589a91f63af: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:53:57.788172 22150 log.cc:1196] T 617443200e2c47b69d63ef056294f40c P fdc2ac372492454c85387589a91f63af: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal/wals/617443200e2c47b69d63ef056294f40c
I20240502 01:53:57.788821 20387 tablet_copy_service.cc:140] P 6a36c952c51542c0bf7ea3ec2857a9f9: Received BeginTabletCopySession request for tablet 31a4ca0cc44e492fbd13a70ce38add35 from peer 3fceccefba274b1dbbd8216af8b0bc29 ({username='slave'} at 127.11.42.66:36601)
I20240502 01:53:57.788991 20387 tablet_copy_service.cc:161] P 6a36c952c51542c0bf7ea3ec2857a9f9: Beginning new tablet copy session on tablet 31a4ca0cc44e492fbd13a70ce38add35 from peer 3fceccefba274b1dbbd8216af8b0bc29 at {username='slave'} at 127.11.42.66:36601: session id = 3fceccefba274b1dbbd8216af8b0bc29-31a4ca0cc44e492fbd13a70ce38add35
I20240502 01:53:57.789248 17334 catalog_manager.cc:5484] T a212c17ba64842c99b3d6a78a3b7cc8a P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: config changed from index -1 to 2, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:53:57.793570 20387 tablet_copy_source_session.cc:205] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:57.796496 22134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31a4ca0cc44e492fbd13a70ce38add35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:57.804206 20232 raft_consensus.cc:2932] T a212c17ba64842c99b3d6a78a3b7cc8a P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:53:57.806383 22134 tablet_copy_client.cc:783] T 31a4ca0cc44e492fbd13a70ce38add35 P 3fceccefba274b1dbbd8216af8b0bc29: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:57.806630 22134 tablet_copy_client.cc:647] T 31a4ca0cc44e492fbd13a70ce38add35 P 3fceccefba274b1dbbd8216af8b0bc29: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:57.807912 17322 catalog_manager.cc:4830] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): tablet 617443200e2c47b69d63ef056294f40c (table rebalancer_extra_table_8 [id=c5f998b2cdaf41f6a49ea6ae65a975b8]) successfully deleted
I20240502 01:53:57.814294 22132 tablet_copy_client.cc:308] T 03bd47783e884a70bfea1dfab4f2e563 P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.68:44083
I20240502 01:53:57.814996 17887 tablet_copy_service.cc:140] P bb31bceba76d466db24b9341c04900c3: Received BeginTabletCopySession request for tablet 03bd47783e884a70bfea1dfab4f2e563 from peer 2f306549407b4bb281ccf2994035e4f1 ({username='slave'} at 127.11.42.71:48285)
I20240502 01:53:57.815158 17887 tablet_copy_service.cc:161] P bb31bceba76d466db24b9341c04900c3: Beginning new tablet copy session on tablet 03bd47783e884a70bfea1dfab4f2e563 from peer 2f306549407b4bb281ccf2994035e4f1 at {username='slave'} at 127.11.42.71:48285: session id = 2f306549407b4bb281ccf2994035e4f1-03bd47783e884a70bfea1dfab4f2e563
I20240502 01:53:57.816563 22134 tablet_copy_client.cc:523] T 31a4ca0cc44e492fbd13a70ce38add35 P 3fceccefba274b1dbbd8216af8b0bc29: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:57.817827 17887 tablet_copy_source_session.cc:205] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:57.819136 22132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03bd47783e884a70bfea1dfab4f2e563. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:57.819883 22134 tablet_bootstrap.cc:492] T 31a4ca0cc44e492fbd13a70ce38add35 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:57.828413 22134 tablet_bootstrap.cc:492] T 31a4ca0cc44e492fbd13a70ce38add35 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:57.828842 22134 tablet_bootstrap.cc:492] T 31a4ca0cc44e492fbd13a70ce38add35 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap complete.
I20240502 01:53:57.829102 22134 ts_tablet_manager.cc:1367] T 31a4ca0cc44e492fbd13a70ce38add35 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.005s
I20240502 01:53:57.829761 22132 tablet_copy_client.cc:783] T 03bd47783e884a70bfea1dfab4f2e563 P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:57.830034 22132 tablet_copy_client.cc:647] T 03bd47783e884a70bfea1dfab4f2e563 P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:57.829841 22134 raft_consensus.cc:348] T 31a4ca0cc44e492fbd13a70ce38add35 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } }
I20240502 01:53:57.830718 22134 raft_consensus.cc:729] T 31a4ca0cc44e492fbd13a70ce38add35 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: LEARNER
I20240502 01:53:57.830973 22134 consensus_queue.cc:260] T 31a4ca0cc44e492fbd13a70ce38add35 P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } }
I20240502 01:53:57.831573 22134 ts_tablet_manager.cc:1398] T 31a4ca0cc44e492fbd13a70ce38add35 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:57.831712 22132 tablet_copy_client.cc:523] T 03bd47783e884a70bfea1dfab4f2e563 P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:57.832263 20387 tablet_copy_service.cc:342] P 6a36c952c51542c0bf7ea3ec2857a9f9: Request end of tablet copy session 3fceccefba274b1dbbd8216af8b0bc29-31a4ca0cc44e492fbd13a70ce38add35 received from {username='slave'} at 127.11.42.66:36601
I20240502 01:53:57.832398 20387 tablet_copy_service.cc:433] P 6a36c952c51542c0bf7ea3ec2857a9f9: ending tablet copy session 3fceccefba274b1dbbd8216af8b0bc29-31a4ca0cc44e492fbd13a70ce38add35 on tablet 31a4ca0cc44e492fbd13a70ce38add35 with peer 3fceccefba274b1dbbd8216af8b0bc29
I20240502 01:53:57.835954 22132 tablet_bootstrap.cc:492] T 03bd47783e884a70bfea1dfab4f2e563 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:53:57.836086 21697 consensus_queue.cc:1035] T 246cec994d274786aad7672f2a30e5d8 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:57.843531 21674 raft_consensus.cc:1050] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685: attempting to promote NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 to VOTER
I20240502 01:53:57.844406 21674 consensus_queue.cc:237] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } }
I20240502 01:53:57.846239 22058 raft_consensus.cc:1050] T 5124abbb09db4be0955c6bc831af08fa P bb31bceba76d466db24b9341c04900c3: attempting to promote NON_VOTER 2f306549407b4bb281ccf2994035e4f1 to VOTER
I20240502 01:53:57.847076 22058 consensus_queue.cc:237] T 5124abbb09db4be0955c6bc831af08fa P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } }
I20240502 01:53:57.865077 20365 consensus_queue.cc:237] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } }
I20240502 01:53:57.884569 20366 raft_consensus.cc:1261] T 71cbc012efe14fdb937302796a9f84e5 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:57.885252 22037 consensus_queue.cc:1035] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:57.893307 17475 raft_consensus.cc:1261] T 71cbc012efe14fdb937302796a9f84e5 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:57.893307 17865 raft_consensus.cc:1261] T 71cbc012efe14fdb937302796a9f84e5 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:57.893488 20496 raft_consensus.cc:1261] T 5124abbb09db4be0955c6bc831af08fa P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:57.894075 21530 consensus_queue.cc:1035] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:57.894496 17735 raft_consensus.cc:1261] T 5124abbb09db4be0955c6bc831af08fa P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:57.894671 21530 consensus_queue.cc:1035] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:57.895337 21667 consensus_queue.cc:1035] T 5124abbb09db4be0955c6bc831af08fa P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:57.898205 21667 consensus_queue.cc:1035] T 5124abbb09db4be0955c6bc831af08fa P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:57.898322 21530 raft_consensus.cc:2932] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:53:57.904050 17475 raft_consensus.cc:2932] T 71cbc012efe14fdb937302796a9f84e5 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:53:57.905380 17332 catalog_manager.cc:5484] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 reported cstate change: config changed from index 4 to 5, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:57.908911 21667 raft_consensus.cc:2932] T 5124abbb09db4be0955c6bc831af08fa P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:53:57.912353 17734 raft_consensus.cc:2932] T 5124abbb09db4be0955c6bc831af08fa P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:53:57.913864 22132 tablet_bootstrap.cc:492] T 03bd47783e884a70bfea1dfab4f2e563 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:57.914280 22132 tablet_bootstrap.cc:492] T 03bd47783e884a70bfea1dfab4f2e563 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap complete.
I20240502 01:53:57.914669 22132 ts_tablet_manager.cc:1367] T 03bd47783e884a70bfea1dfab4f2e563 P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.079s user 0.062s sys 0.007s
I20240502 01:53:57.911695 20366 raft_consensus.cc:2932] T 71cbc012efe14fdb937302796a9f84e5 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:53:57.915455 22132 raft_consensus.cc:348] T 03bd47783e884a70bfea1dfab4f2e563 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
I20240502 01:53:57.915702 22132 raft_consensus.cc:729] T 03bd47783e884a70bfea1dfab4f2e563 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: LEARNER
I20240502 01:53:57.916137 22132 consensus_queue.cc:260] T 03bd47783e884a70bfea1dfab4f2e563 P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
I20240502 01:53:57.916772 22132 ts_tablet_manager.cc:1398] T 03bd47783e884a70bfea1dfab4f2e563 P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:57.917440 17887 tablet_copy_service.cc:342] P bb31bceba76d466db24b9341c04900c3: Request end of tablet copy session 2f306549407b4bb281ccf2994035e4f1-03bd47783e884a70bfea1dfab4f2e563 received from {username='slave'} at 127.11.42.71:48285
I20240502 01:53:57.917584 17887 tablet_copy_service.cc:433] P bb31bceba76d466db24b9341c04900c3: ending tablet copy session 2f306549407b4bb281ccf2994035e4f1-03bd47783e884a70bfea1dfab4f2e563 on tablet 03bd47783e884a70bfea1dfab4f2e563 with peer 2f306549407b4bb281ccf2994035e4f1
I20240502 01:53:57.920712 17734 consensus_queue.cc:237] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } }
I20240502 01:53:57.922735 20367 raft_consensus.cc:1261] T 71cbc012efe14fdb937302796a9f84e5 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:53:57.922884 17475 raft_consensus.cc:1261] T 71cbc012efe14fdb937302796a9f84e5 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:53:57.923557 22037 consensus_queue.cc:1035] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:53:57.924258 22037 consensus_queue.cc:1035] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:53:57.924561 17334 catalog_manager.cc:5484] T 5124abbb09db4be0955c6bc831af08fa P dfa1152987644bd781b46c7e42a7a685 reported cstate change: config changed from index 4 to 5, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:53:57.926353 22037 raft_consensus.cc:2932] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
W20240502 01:53:57.926884 20302 consensus_peers.cc:485] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Couldn't send request to peer 4da690c0fa13407a85c0689ce8de9ade. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1d8e2e0f6ed34e4cb004e1c5309c0f1f. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:57.927917 20496 raft_consensus.cc:1261] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:57.928303 20629 raft_consensus.cc:1261] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:57.928916 22081 consensus_queue.cc:1035] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:53:57.929528 22156 consensus_queue.cc:1035] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:53:57.929414 20367 raft_consensus.cc:2932] T 71cbc012efe14fdb937302796a9f84e5 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:53:57.932459 21429 consensus_queue.cc:1035] T 81f335c023444796ab933e19d809cc03 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:57.942389 17605 raft_consensus.cc:1261] T 5124abbb09db4be0955c6bc831af08fa P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:57.943418 21216 consensus_queue.cc:1035] T 246cec994d274786aad7672f2a30e5d8 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:57.944272 22156 raft_consensus.cc:2932] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } } }
I20240502 01:53:57.945469 22058 consensus_queue.cc:1035] T 5124abbb09db4be0955c6bc831af08fa P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:57.945695 20628 raft_consensus.cc:2932] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } } }
I20240502 01:53:57.946731 20498 raft_consensus.cc:1203] T 03bd47783e884a70bfea1dfab4f2e563 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:53:57.944801 17334 catalog_manager.cc:5484] T 71cbc012efe14fdb937302796a9f84e5 P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: config changed from index 5 to 6, VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) evicted. New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:53:57.957197 17332 catalog_manager.cc:5484] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: config changed from index -1 to 2, NON_VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) added. New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } } }
I20240502 01:53:57.959436 21429 consensus_queue.cc:1035] T 81f335c023444796ab933e19d809cc03 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:57.969360 17846 tablet_service.cc:1514] Processing DeleteTablet for tablet 71cbc012efe14fdb937302796a9f84e5 with delete_type TABLET_DATA_TOMBSTONED (TS bb31bceba76d466db24b9341c04900c3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47226
I20240502 01:53:57.969674 22104 tablet_replica.cc:330] T 71cbc012efe14fdb937302796a9f84e5 P bb31bceba76d466db24b9341c04900c3: stopping tablet replica
I20240502 01:53:57.969882 22104 raft_consensus.cc:2229] T 71cbc012efe14fdb937302796a9f84e5 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:57.970023 22104 pending_rounds.cc:62] T 71cbc012efe14fdb937302796a9f84e5 P bb31bceba76d466db24b9341c04900c3: Trying to abort 1 pending ops.
I20240502 01:53:57.970118 22104 pending_rounds.cc:68] T 71cbc012efe14fdb937302796a9f84e5 P bb31bceba76d466db24b9341c04900c3: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023062375808462848 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "71cbc012efe14fdb937302796a9f84e5" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } } }
I20240502 01:53:57.970481 22104 raft_consensus.cc:2894] T 71cbc012efe14fdb937302796a9f84e5 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240502 01:53:57.970619 22104 raft_consensus.cc:2866] T 71cbc012efe14fdb937302796a9f84e5 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 01:53:57.970706 22104 raft_consensus.cc:2258] T 71cbc012efe14fdb937302796a9f84e5 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:57.973135 21674 raft_consensus.cc:482] T 7131c9bab38e45fa984a84ea6e21dcc5 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:57.973325 21674 raft_consensus.cc:504] T 7131c9bab38e45fa984a84ea6e21dcc5 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:57.973939 21674 leader_election.cc:288] T 7131c9bab38e45fa984a84ea6e21dcc5 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:53:57.974520 20498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7131c9bab38e45fa984a84ea6e21dcc5" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
I20240502 01:53:57.974731 20498 raft_consensus.cc:2454] T 7131c9bab38e45fa984a84ea6e21dcc5 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:53:57.975000 20230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7131c9bab38e45fa984a84ea6e21dcc5" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:53:57.975095 17673 leader_election.cc:302] T 7131c9bab38e45fa984a84ea6e21dcc5 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:57.975222 20230 raft_consensus.cc:2454] T 7131c9bab38e45fa984a84ea6e21dcc5 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:53:57.975387 21674 raft_consensus.cc:2781] T 7131c9bab38e45fa984a84ea6e21dcc5 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:57.975471 21674 raft_consensus.cc:482] T 7131c9bab38e45fa984a84ea6e21dcc5 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:57.975593 21674 raft_consensus.cc:3037] T 7131c9bab38e45fa984a84ea6e21dcc5 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.978260 21674 raft_consensus.cc:504] T 7131c9bab38e45fa984a84ea6e21dcc5 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:57.978871 21674 leader_election.cc:288] T 7131c9bab38e45fa984a84ea6e21dcc5 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Requested vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:53:57.980530 20498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7131c9bab38e45fa984a84ea6e21dcc5" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
I20240502 01:53:57.980724 20498 raft_consensus.cc:3037] T 7131c9bab38e45fa984a84ea6e21dcc5 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.981187 20230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7131c9bab38e45fa984a84ea6e21dcc5" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:53:57.981369 20230 raft_consensus.cc:3037] T 7131c9bab38e45fa984a84ea6e21dcc5 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:57.983528 20498 raft_consensus.cc:2454] T 7131c9bab38e45fa984a84ea6e21dcc5 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:53:57.983966 17673 leader_election.cc:302] T 7131c9bab38e45fa984a84ea6e21dcc5 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:57.984144 20230 raft_consensus.cc:2454] T 7131c9bab38e45fa984a84ea6e21dcc5 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:53:57.984248 21674 raft_consensus.cc:2781] T 7131c9bab38e45fa984a84ea6e21dcc5 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:57.984769 21674 raft_consensus.cc:686] T 7131c9bab38e45fa984a84ea6e21dcc5 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Becoming Leader. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Running, Role: LEADER
I20240502 01:53:57.985091 21674 consensus_queue.cc:237] T 7131c9bab38e45fa984a84ea6e21dcc5 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:57.985729 22104 ts_tablet_manager.cc:1875] T 71cbc012efe14fdb937302796a9f84e5 P bb31bceba76d466db24b9341c04900c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:58.012490 21697 consensus_queue.cc:1035] T adc2b57af23c41aebd1258ff8b60d074 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:58.025471 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 71cbc012efe14fdb937302796a9f84e5 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:58.027678 21924 consensus_queue.cc:1035] T adc2b57af23c41aebd1258ff8b60d074 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:58.036067 22104 ts_tablet_manager.cc:1888] T 71cbc012efe14fdb937302796a9f84e5 P bb31bceba76d466db24b9341c04900c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:53:58.036327 22104 log.cc:1196] T 71cbc012efe14fdb937302796a9f84e5 P bb31bceba76d466db24b9341c04900c3: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/wals/71cbc012efe14fdb937302796a9f84e5
I20240502 01:53:58.039381 20497 raft_consensus.cc:2932] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } } }
I20240502 01:53:58.048461 17604 raft_consensus.cc:1203] T 31a4ca0cc44e492fbd13a70ce38add35 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:53:58.049947 17322 catalog_manager.cc:4830] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): tablet 71cbc012efe14fdb937302796a9f84e5 (table rebalance_test_table_19 [id=b69eb497d10340fea8df98ad03a1e465]) successfully deleted
I20240502 01:53:58.058758 20496 raft_consensus.cc:2932] T 5124abbb09db4be0955c6bc831af08fa P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:53:58.067971 17865 consensus_queue.cc:237] T 5124abbb09db4be0955c6bc831af08fa P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } }
I20240502 01:53:58.080914 22174 ts_tablet_manager.cc:902] T cf0237a4c65b495ba892871dd850bed9 P dfa1152987644bd781b46c7e42a7a685: Initiating tablet copy from peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:53:58.083546 17736 raft_consensus.cc:1261] T 5124abbb09db4be0955c6bc831af08fa P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:53:58.088392 22169 consensus_queue.cc:1035] T 5124abbb09db4be0955c6bc831af08fa P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:53:58.071874 17605 raft_consensus.cc:2932] T 5124abbb09db4be0955c6bc831af08fa P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:53:58.092408 21667 raft_consensus.cc:2932] T 5124abbb09db4be0955c6bc831af08fa P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:53:58.096565 17735 raft_consensus.cc:2932] T 5124abbb09db4be0955c6bc831af08fa P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:53:58.102227 17322 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5124abbb09db4be0955c6bc831af08fa with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:58.102273 20496 raft_consensus.cc:1261] T 5124abbb09db4be0955c6bc831af08fa P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:53:58.103031 21667 consensus_queue.cc:1035] T 5124abbb09db4be0955c6bc831af08fa P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:53:58.107153 20496 raft_consensus.cc:2932] T 5124abbb09db4be0955c6bc831af08fa P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:53:58.108582 17475 raft_consensus.cc:2932] T 71cbc012efe14fdb937302796a9f84e5 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:53:58.112007 17334 catalog_manager.cc:5484] T 5124abbb09db4be0955c6bc831af08fa P bb31bceba76d466db24b9341c04900c3 reported cstate change: config changed from index 5 to 6, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240502 01:53:58.126924 17542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.42.67:36115, user_credentials={real_user=slave}} blocked reactor thread for 63547.9us
I20240502 01:53:58.129302 17587 tablet_service.cc:1514] Processing DeleteTablet for tablet 5124abbb09db4be0955c6bc831af08fa with delete_type TABLET_DATA_TOMBSTONED (TS 3fceccefba274b1dbbd8216af8b0bc29 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55232
I20240502 01:53:58.129958 22184 raft_consensus.cc:1050] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29: attempting to promote NON_VOTER f19cc6cb1e76460aa6711e49a5775091 to VOTER
I20240502 01:53:58.130704 22184 consensus_queue.cc:237] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } }
I20240502 01:53:58.140139 17335 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_extra_table_20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:58.157672 20609 tablet_service.cc:1467] Processing CreateTablet for tablet 75bb4e19cf57405fabcb532b55a29ced (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=0e1c9ce53d7c47f7a42d19e60342c557]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:58.157964 17846 tablet_service.cc:1467] Processing CreateTablet for tablet 75bb4e19cf57405fabcb532b55a29ced (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=0e1c9ce53d7c47f7a42d19e60342c557]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:58.158193 20609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75bb4e19cf57405fabcb532b55a29ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:58.158416 17846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75bb4e19cf57405fabcb532b55a29ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:58.163352 20347 tablet_service.cc:1467] Processing CreateTablet for tablet 75bb4e19cf57405fabcb532b55a29ced (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=0e1c9ce53d7c47f7a42d19e60342c557]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:58.163833 20347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75bb4e19cf57405fabcb532b55a29ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:58.176964 22210 tablet_replica.cc:330] T 5124abbb09db4be0955c6bc831af08fa P 3fceccefba274b1dbbd8216af8b0bc29: stopping tablet replica
I20240502 01:53:58.177254 22210 raft_consensus.cc:2229] T 5124abbb09db4be0955c6bc831af08fa P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:58.177536 22210 raft_consensus.cc:2258] T 5124abbb09db4be0955c6bc831af08fa P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:58.178546 22210 ts_tablet_manager.cc:1875] T 5124abbb09db4be0955c6bc831af08fa P 3fceccefba274b1dbbd8216af8b0bc29: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:58.180038 20477 tablet_service.cc:1467] Processing CreateTablet for tablet 2714919062f5465c8cde1a3c4720978a (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=0e1c9ce53d7c47f7a42d19e60342c557]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:58.180111 20211 tablet_service.cc:1467] Processing CreateTablet for tablet 2714919062f5465c8cde1a3c4720978a (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=0e1c9ce53d7c47f7a42d19e60342c557]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:58.180480 20211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2714919062f5465c8cde1a3c4720978a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:58.180491 20477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2714919062f5465c8cde1a3c4720978a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:58.181397 20345 tablet_service.cc:1467] Processing CreateTablet for tablet 2714919062f5465c8cde1a3c4720978a (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=0e1c9ce53d7c47f7a42d19e60342c557]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:58.181769 20345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2714919062f5465c8cde1a3c4720978a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:58.180049 20476 tablet_service.cc:1467] Processing CreateTablet for tablet d10d76ac7b094fe9b230b1afcbed9ca6 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=0e1c9ce53d7c47f7a42d19e60342c557]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:58.185429 20212 tablet_service.cc:1467] Processing CreateTablet for tablet d10d76ac7b094fe9b230b1afcbed9ca6 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=0e1c9ce53d7c47f7a42d19e60342c557]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:58.185786 20476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d10d76ac7b094fe9b230b1afcbed9ca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:58.185813 20212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d10d76ac7b094fe9b230b1afcbed9ca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:58.189363 20346 tablet_service.cc:1467] Processing CreateTablet for tablet d10d76ac7b094fe9b230b1afcbed9ca6 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=0e1c9ce53d7c47f7a42d19e60342c557]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:58.189734 20346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d10d76ac7b094fe9b230b1afcbed9ca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:58.192989 21914 tablet_bootstrap.cc:492] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:53:58.194496 21914 tablet_bootstrap.cc:654] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:58.199476 22092 tablet_bootstrap.cc:492] T 2714919062f5465c8cde1a3c4720978a P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:53:58.201004 22092 tablet_bootstrap.cc:654] T 2714919062f5465c8cde1a3c4720978a P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:58.203704 17334 catalog_manager.cc:5484] T 7131c9bab38e45fa984a84ea6e21dcc5 P dfa1152987644bd781b46c7e42a7a685 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfa1152987644bd781b46c7e42a7a685 (127.11.42.67). New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:58.211710 22092 tablet_bootstrap.cc:492] T 2714919062f5465c8cde1a3c4720978a P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:53:58.211889 22092 ts_tablet_manager.cc:1367] T 2714919062f5465c8cde1a3c4720978a P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240502 01:53:58.212489 22210 ts_tablet_manager.cc:1888] T 5124abbb09db4be0955c6bc831af08fa P 3fceccefba274b1dbbd8216af8b0bc29: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:53:58.212610 22210 log.cc:1196] T 5124abbb09db4be0955c6bc831af08fa P 3fceccefba274b1dbbd8216af8b0bc29: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/wals/5124abbb09db4be0955c6bc831af08fa
I20240502 01:53:58.213603 22092 raft_consensus.cc:348] T 2714919062f5465c8cde1a3c4720978a P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:58.213948 22092 raft_consensus.cc:374] T 2714919062f5465c8cde1a3c4720978a P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:58.214071 22092 raft_consensus.cc:729] T 2714919062f5465c8cde1a3c4720978a P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:53:58.211966 22215 tablet_bootstrap.cc:492] T d10d76ac7b094fe9b230b1afcbed9ca6 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:53:58.214401 22092 consensus_queue.cc:260] T 2714919062f5465c8cde1a3c4720978a P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:58.214732 21580 raft_consensus.cc:482] T 2714919062f5465c8cde1a3c4720978a P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:58.214877 21580 raft_consensus.cc:504] T 2714919062f5465c8cde1a3c4720978a P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:58.215555 21580 leader_election.cc:288] T 2714919062f5465c8cde1a3c4720978a P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:53:58.216007 22215 tablet_bootstrap.cc:654] T d10d76ac7b094fe9b230b1afcbed9ca6 P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:58.217094 20366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2714919062f5465c8cde1a3c4720978a" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
W20240502 01:53:58.218132 20167 leader_election.cc:341] T 2714919062f5465c8cde1a3c4720978a P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:53:58.218720 20496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2714919062f5465c8cde1a3c4720978a" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
I20240502 01:53:58.219401 22092 ts_tablet_manager.cc:1398] T 2714919062f5465c8cde1a3c4720978a P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 01:53:58.219712 22092 tablet_bootstrap.cc:492] T d10d76ac7b094fe9b230b1afcbed9ca6 P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
W20240502 01:53:58.220690 20168 leader_election.cc:341] T 2714919062f5465c8cde1a3c4720978a P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:53:58.220836 20168 leader_election.cc:302] T 2714919062f5465c8cde1a3c4720978a P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdc2ac372492454c85387589a91f63af; no voters: 2f306549407b4bb281ccf2994035e4f1, 6a36c952c51542c0bf7ea3ec2857a9f9
I20240502 01:53:58.221352 22217 raft_consensus.cc:2726] T 2714919062f5465c8cde1a3c4720978a P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:53:58.221446 22092 tablet_bootstrap.cc:654] T d10d76ac7b094fe9b230b1afcbed9ca6 P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:58.225741 20628 raft_consensus.cc:1261] T 10ec78a08ab04678897a6892988e515f P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:58.225842 22092 tablet_bootstrap.cc:492] T d10d76ac7b094fe9b230b1afcbed9ca6 P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:53:58.225976 22092 ts_tablet_manager.cc:1367] T d10d76ac7b094fe9b230b1afcbed9ca6 P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240502 01:53:58.226778 22092 raft_consensus.cc:348] T d10d76ac7b094fe9b230b1afcbed9ca6 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:58.226981 22092 raft_consensus.cc:374] T d10d76ac7b094fe9b230b1afcbed9ca6 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:58.227080 22092 raft_consensus.cc:729] T d10d76ac7b094fe9b230b1afcbed9ca6 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:53:58.227120 17734 raft_consensus.cc:1261] T 10ec78a08ab04678897a6892988e515f P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:58.227320 22092 consensus_queue.cc:260] T d10d76ac7b094fe9b230b1afcbed9ca6 P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:58.227849 22092 ts_tablet_manager.cc:1398] T d10d76ac7b094fe9b230b1afcbed9ca6 P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:53:58.228132 17475 raft_consensus.cc:1261] T 10ec78a08ab04678897a6892988e515f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:58.228298 22201 consensus_queue.cc:1035] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:58.229228 22201 consensus_queue.cc:1035] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:58.234812 22213 tablet_bootstrap.cc:492] T 75bb4e19cf57405fabcb532b55a29ced P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:53:58.236318 22201 consensus_queue.cc:1035] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:58.236994 22212 tablet_bootstrap.cc:492] T 75bb4e19cf57405fabcb532b55a29ced P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:58.238840 22212 tablet_bootstrap.cc:654] T 75bb4e19cf57405fabcb532b55a29ced P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:58.239765 22213 tablet_bootstrap.cc:654] T 75bb4e19cf57405fabcb532b55a29ced P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:58.240617 21914 tablet_bootstrap.cc:492] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:53:58.240788 21914 ts_tablet_manager.cc:1367] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.048s user 0.010s sys 0.017s
I20240502 01:53:58.242722 21914 raft_consensus.cc:348] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:58.243723 21914 raft_consensus.cc:374] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:58.243832 21914 raft_consensus.cc:729] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:53:58.244484 21914 consensus_queue.cc:260] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:58.247929 22215 tablet_bootstrap.cc:492] T d10d76ac7b094fe9b230b1afcbed9ca6 P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:53:58.248172 22215 ts_tablet_manager.cc:1367] T d10d76ac7b094fe9b230b1afcbed9ca6 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.036s user 0.001s sys 0.003s
I20240502 01:53:58.249413 21914 ts_tablet_manager.cc:1398] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 01:53:58.249076 22215 raft_consensus.cc:348] T d10d76ac7b094fe9b230b1afcbed9ca6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:58.250824 22215 raft_consensus.cc:374] T d10d76ac7b094fe9b230b1afcbed9ca6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:58.250970 22215 raft_consensus.cc:729] T d10d76ac7b094fe9b230b1afcbed9ca6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:53:58.251269 22215 consensus_queue.cc:260] T d10d76ac7b094fe9b230b1afcbed9ca6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:58.254307 22215 ts_tablet_manager.cc:1398] T d10d76ac7b094fe9b230b1afcbed9ca6 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:53:58.254637 22215 tablet_bootstrap.cc:492] T 2714919062f5465c8cde1a3c4720978a P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:53:58.257982 21914 tablet_bootstrap.cc:492] T 2714919062f5465c8cde1a3c4720978a P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:53:58.261544 17319 catalog_manager.cc:4830] TS 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): tablet 5124abbb09db4be0955c6bc831af08fa (table rebalance_test_table_1 [id=e431affbb02a4744b67a522521a3ee04]) successfully deleted
I20240502 01:53:58.262877 21914 tablet_bootstrap.cc:654] T 2714919062f5465c8cde1a3c4720978a P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:58.264787 22215 tablet_bootstrap.cc:654] T 2714919062f5465c8cde1a3c4720978a P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:58.272178 22174 tablet_copy_client.cc:308] T cf0237a4c65b495ba892871dd850bed9 P dfa1152987644bd781b46c7e42a7a685: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.72:42411
I20240502 01:53:58.273428 21914 tablet_bootstrap.cc:492] T 2714919062f5465c8cde1a3c4720978a P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:53:58.273589 21914 ts_tablet_manager.cc:1367] T 2714919062f5465c8cde1a3c4720978a P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20240502 01:53:58.275378 21914 raft_consensus.cc:348] T 2714919062f5465c8cde1a3c4720978a P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:58.275601 21914 raft_consensus.cc:374] T 2714919062f5465c8cde1a3c4720978a P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:58.275667 21914 raft_consensus.cc:729] T 2714919062f5465c8cde1a3c4720978a P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:53:58.276825 21914 consensus_queue.cc:260] T 2714919062f5465c8cde1a3c4720978a P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:58.281862 21914 ts_tablet_manager.cc:1398] T 2714919062f5465c8cde1a3c4720978a P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 01:53:58.281944 21674 raft_consensus.cc:1050] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685: attempting to promote NON_VOTER fdc2ac372492454c85387589a91f63af to VOTER
I20240502 01:53:58.289894 20650 tablet_copy_service.cc:140] P f19cc6cb1e76460aa6711e49a5775091: Received BeginTabletCopySession request for tablet cf0237a4c65b495ba892871dd850bed9 from peer dfa1152987644bd781b46c7e42a7a685 ({username='slave'} at 127.11.42.67:36277)
I20240502 01:53:58.290076 20650 tablet_copy_service.cc:161] P f19cc6cb1e76460aa6711e49a5775091: Beginning new tablet copy session on tablet cf0237a4c65b495ba892871dd850bed9 from peer dfa1152987644bd781b46c7e42a7a685 at {username='slave'} at 127.11.42.67:36277: session id = dfa1152987644bd781b46c7e42a7a685-cf0237a4c65b495ba892871dd850bed9
I20240502 01:53:58.290120 21674 consensus_queue.cc:237] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } }
I20240502 01:53:58.292991 20650 tablet_copy_source_session.cc:205] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:58.297385 22174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf0237a4c65b495ba892871dd850bed9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:58.308027 22202 raft_consensus.cc:2932] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:58.308888 22230 ts_tablet_manager.cc:902] T a212c17ba64842c99b3d6a78a3b7cc8a P f19cc6cb1e76460aa6711e49a5775091: Initiating tablet copy from peer 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:53:58.314272 22213 tablet_bootstrap.cc:492] T 75bb4e19cf57405fabcb532b55a29ced P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:53:58.314426 22213 ts_tablet_manager.cc:1367] T 75bb4e19cf57405fabcb532b55a29ced P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.080s user 0.013s sys 0.018s
I20240502 01:53:58.315373 22213 raft_consensus.cc:348] T 75bb4e19cf57405fabcb532b55a29ced P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:58.315580 22213 raft_consensus.cc:374] T 75bb4e19cf57405fabcb532b55a29ced P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:58.315666 22213 raft_consensus.cc:729] T 75bb4e19cf57405fabcb532b55a29ced P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:53:58.315898 22213 consensus_queue.cc:260] T 75bb4e19cf57405fabcb532b55a29ced P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:58.316453 22213 ts_tablet_manager.cc:1398] T 75bb4e19cf57405fabcb532b55a29ced P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:53:58.316581 17334 catalog_manager.cc:5484] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: config changed from index 4 to 5, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:58.317716 22212 tablet_bootstrap.cc:492] T 75bb4e19cf57405fabcb532b55a29ced P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:58.317867 22212 ts_tablet_manager.cc:1367] T 75bb4e19cf57405fabcb532b55a29ced P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.081s user 0.015s sys 0.020s
I20240502 01:53:58.318657 22212 raft_consensus.cc:348] T 75bb4e19cf57405fabcb532b55a29ced P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:58.318933 22212 raft_consensus.cc:374] T 75bb4e19cf57405fabcb532b55a29ced P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:58.320478 22212 raft_consensus.cc:729] T 75bb4e19cf57405fabcb532b55a29ced P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:58.320847 22212 consensus_queue.cc:260] T 75bb4e19cf57405fabcb532b55a29ced P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:58.327466 22212 ts_tablet_manager.cc:1398] T 75bb4e19cf57405fabcb532b55a29ced P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.009s user 0.000s sys 0.003s
I20240502 01:53:58.324714 17475 raft_consensus.cc:2932] T 10ec78a08ab04678897a6892988e515f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:58.336300 22215 tablet_bootstrap.cc:492] T 2714919062f5465c8cde1a3c4720978a P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:53:58.336526 22215 ts_tablet_manager.cc:1367] T 2714919062f5465c8cde1a3c4720978a P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.082s user 0.002s sys 0.002s
I20240502 01:53:58.337128 17605 consensus_queue.cc:237] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } }
I20240502 01:53:58.337505 22215 raft_consensus.cc:348] T 2714919062f5465c8cde1a3c4720978a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:58.337805 22215 raft_consensus.cc:374] T 2714919062f5465c8cde1a3c4720978a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:58.337925 22215 raft_consensus.cc:729] T 2714919062f5465c8cde1a3c4720978a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:53:58.338240 22215 consensus_queue.cc:260] T 2714919062f5465c8cde1a3c4720978a P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:58.341630 20628 raft_consensus.cc:1261] T 10ec78a08ab04678897a6892988e515f P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:53:58.342141 20629 raft_consensus.cc:2932] T 10ec78a08ab04678897a6892988e515f P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:58.343149 22215 ts_tablet_manager.cc:1398] T 2714919062f5465c8cde1a3c4720978a P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:53:58.343987 17605 raft_consensus.cc:1261] T 8e7f88b6c509477fb0900024e7de8f00 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:58.344053 17865 raft_consensus.cc:1261] T 8e7f88b6c509477fb0900024e7de8f00 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:58.344314 22215 tablet_bootstrap.cc:492] T 75bb4e19cf57405fabcb532b55a29ced P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:53:58.344691 22037 consensus_queue.cc:1035] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:58.344831 20233 raft_consensus.cc:1261] T 8e7f88b6c509477fb0900024e7de8f00 P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:58.345253 22037 consensus_queue.cc:1035] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:58.345587 22190 consensus_queue.cc:1035] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:58.346473 22215 tablet_bootstrap.cc:654] T 75bb4e19cf57405fabcb532b55a29ced P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:58.347067 22201 consensus_queue.cc:1035] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:58.351469 22201 raft_consensus.cc:2932] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:58.353363 21667 raft_consensus.cc:1050] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3: attempting to promote NON_VOTER 2f306549407b4bb281ccf2994035e4f1 to VOTER
I20240502 01:53:58.354262 21667 consensus_queue.cc:237] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } }
I20240502 01:53:58.357545 22215 tablet_bootstrap.cc:492] T 75bb4e19cf57405fabcb532b55a29ced P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:53:58.357743 22215 ts_tablet_manager.cc:1367] T 75bb4e19cf57405fabcb532b55a29ced P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.004s
I20240502 01:53:58.358409 17319 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 10ec78a08ab04678897a6892988e515f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:58.358737 17332 catalog_manager.cc:5484] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: config changed from index 5 to 6, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:58.359429 22215 raft_consensus.cc:348] T 75bb4e19cf57405fabcb532b55a29ced P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:58.359196 20630 raft_consensus.cc:2932] T 10ec78a08ab04678897a6892988e515f P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:58.359629 22215 raft_consensus.cc:374] T 75bb4e19cf57405fabcb532b55a29ced P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:58.359707 22215 raft_consensus.cc:729] T 75bb4e19cf57405fabcb532b55a29ced P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:53:58.359956 22215 consensus_queue.cc:260] T 75bb4e19cf57405fabcb532b55a29ced P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:58.361649 22215 ts_tablet_manager.cc:1398] T 75bb4e19cf57405fabcb532b55a29ced P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 01:53:58.367398 17475 raft_consensus.cc:1261] T 10ec78a08ab04678897a6892988e515f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:53:58.369530 17716 tablet_service.cc:1514] Processing DeleteTablet for tablet 10ec78a08ab04678897a6892988e515f with delete_type TABLET_DATA_TOMBSTONED (TS dfa1152987644bd781b46c7e42a7a685 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52198
I20240502 01:53:58.370052 22201 consensus_queue.cc:1035] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:53:58.371659 17475 raft_consensus.cc:2932] T 10ec78a08ab04678897a6892988e515f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:58.376272 22037 raft_consensus.cc:2932] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:53:58.378253 21934 raft_consensus.cc:482] T 75bb4e19cf57405fabcb532b55a29ced P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:58.378403 21934 raft_consensus.cc:504] T 75bb4e19cf57405fabcb532b55a29ced P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:58.379086 21934 leader_election.cc:288] T 75bb4e19cf57405fabcb532b55a29ced P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:53:58.379918 17864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75bb4e19cf57405fabcb532b55a29ced" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:53:58.380172 17864 raft_consensus.cc:2454] T 75bb4e19cf57405fabcb532b55a29ced P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 0.
I20240502 01:53:58.380620 20565 leader_election.cc:302] T 75bb4e19cf57405fabcb532b55a29ced P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb31bceba76d466db24b9341c04900c3, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:53:58.380929 21934 raft_consensus.cc:2781] T 75bb4e19cf57405fabcb532b55a29ced P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:58.381063 21934 raft_consensus.cc:482] T 75bb4e19cf57405fabcb532b55a29ced P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:58.381155 21934 raft_consensus.cc:3037] T 75bb4e19cf57405fabcb532b55a29ced P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:58.381428 20233 raft_consensus.cc:2932] T 8e7f88b6c509477fb0900024e7de8f00 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:53:58.383450 17605 raft_consensus.cc:2932] T 8e7f88b6c509477fb0900024e7de8f00 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:53:58.383988 21934 raft_consensus.cc:504] T 75bb4e19cf57405fabcb532b55a29ced P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:58.385048 17864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75bb4e19cf57405fabcb532b55a29ced" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:53:58.385259 17864 raft_consensus.cc:3037] T 75bb4e19cf57405fabcb532b55a29ced P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:58.387137 17327 catalog_manager.cc:5484] T 8e7f88b6c509477fb0900024e7de8f00 P fdc2ac372492454c85387589a91f63af reported cstate change: config changed from index 4 to 5, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:53:58.388177 17864 raft_consensus.cc:2454] T 75bb4e19cf57405fabcb532b55a29ced P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 1.
I20240502 01:53:58.388701 20565 leader_election.cc:302] T 75bb4e19cf57405fabcb532b55a29ced P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb31bceba76d466db24b9341c04900c3, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:53:58.389117 21429 raft_consensus.cc:2781] T 75bb4e19cf57405fabcb532b55a29ced P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:58.389282 21429 raft_consensus.cc:686] T 75bb4e19cf57405fabcb532b55a29ced P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Becoming Leader. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Running, Role: LEADER
I20240502 01:53:58.389562 21429 consensus_queue.cc:237] T 75bb4e19cf57405fabcb532b55a29ced P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:58.391166 20366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75bb4e19cf57405fabcb532b55a29ced" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:53:58.391420 20366 raft_consensus.cc:2454] T 75bb4e19cf57405fabcb532b55a29ced P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 0.
I20240502 01:53:58.395226 20366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75bb4e19cf57405fabcb532b55a29ced" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
I20240502 01:53:58.395458 20366 raft_consensus.cc:3037] T 75bb4e19cf57405fabcb532b55a29ced P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:58.396374 17734 consensus_queue.cc:237] T 7e9be84ef6a043818f166f6b47898855 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
I20240502 01:53:58.398106 20366 raft_consensus.cc:2454] T 75bb4e19cf57405fabcb532b55a29ced P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 1.
I20240502 01:53:58.398975 21934 leader_election.cc:288] T 75bb4e19cf57405fabcb532b55a29ced P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 election: Requested vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:53:58.400115 22241 tablet_replica.cc:330] T 10ec78a08ab04678897a6892988e515f P dfa1152987644bd781b46c7e42a7a685: stopping tablet replica
I20240502 01:53:58.402140 22241 raft_consensus.cc:2229] T 10ec78a08ab04678897a6892988e515f P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:58.406093 20496 raft_consensus.cc:1261] T 03bd47783e884a70bfea1dfab4f2e563 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:58.407321 22058 consensus_queue.cc:1035] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:58.407732 17475 raft_consensus.cc:1261] T 03bd47783e884a70bfea1dfab4f2e563 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:58.409150 22233 ts_tablet_manager.cc:902] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 4da690c0fa13407a85c0689ce8de9ade: Initiating tablet copy from peer 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:53:58.409377 22058 consensus_queue.cc:1035] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:58.415302 21579 raft_consensus.cc:1050] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9: attempting to promote NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 to VOTER
I20240502 01:53:58.416059 21579 consensus_queue.cc:237] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } }
I20240502 01:53:58.417098 17334 catalog_manager.cc:5484] T 75bb4e19cf57405fabcb532b55a29ced P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: term changed from 0 to 1, leader changed from <none> to f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72). New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:58.404455 22241 pending_rounds.cc:62] T 10ec78a08ab04678897a6892988e515f P dfa1152987644bd781b46c7e42a7a685: Trying to abort 1 pending ops.
I20240502 01:53:58.420922 17735 raft_consensus.cc:1261] T 03bd47783e884a70bfea1dfab4f2e563 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:58.420645 22241 pending_rounds.cc:68] T 10ec78a08ab04678897a6892988e515f P dfa1152987644bd781b46c7e42a7a685: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023062376981565440 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "10ec78a08ab04678897a6892988e515f" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } } }
I20240502 01:53:58.421151 22241 raft_consensus.cc:2894] T 10ec78a08ab04678897a6892988e515f P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240502 01:53:58.421289 22241 raft_consensus.cc:2866] T 10ec78a08ab04678897a6892988e515f P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 01:53:58.421399 22241 raft_consensus.cc:2258] T 10ec78a08ab04678897a6892988e515f P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:58.422008 21667 consensus_queue.cc:1035] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:58.422961 17864 raft_consensus.cc:2932] T 8e7f88b6c509477fb0900024e7de8f00 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:53:58.428817 22230 tablet_copy_client.cc:308] T a212c17ba64842c99b3d6a78a3b7cc8a P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.70:39155
I20240502 01:53:58.431061 20387 tablet_copy_service.cc:140] P 6a36c952c51542c0bf7ea3ec2857a9f9: Received BeginTabletCopySession request for tablet a212c17ba64842c99b3d6a78a3b7cc8a from peer f19cc6cb1e76460aa6711e49a5775091 ({username='slave'} at 127.11.42.72:36559)
I20240502 01:53:58.431218 20387 tablet_copy_service.cc:161] P 6a36c952c51542c0bf7ea3ec2857a9f9: Beginning new tablet copy session on tablet a212c17ba64842c99b3d6a78a3b7cc8a from peer f19cc6cb1e76460aa6711e49a5775091 at {username='slave'} at 127.11.42.72:36559: session id = f19cc6cb1e76460aa6711e49a5775091-a212c17ba64842c99b3d6a78a3b7cc8a
I20240502 01:53:58.435583 20387 tablet_copy_source_session.cc:205] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:58.436775 22230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a212c17ba64842c99b3d6a78a3b7cc8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:58.439568 22174 tablet_copy_client.cc:783] T cf0237a4c65b495ba892871dd850bed9 P dfa1152987644bd781b46c7e42a7a685: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:58.439868 22174 tablet_copy_client.cc:647] T cf0237a4c65b495ba892871dd850bed9 P dfa1152987644bd781b46c7e42a7a685: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:58.439582 22241 ts_tablet_manager.cc:1875] T 10ec78a08ab04678897a6892988e515f P dfa1152987644bd781b46c7e42a7a685: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 01:53:58.440753 17673 consensus_peers.cc:485] T 7e9be84ef6a043818f166f6b47898855 P dfa1152987644bd781b46c7e42a7a685 -> Peer 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): Couldn't send request to peer 2f306549407b4bb281ccf2994035e4f1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7e9be84ef6a043818f166f6b47898855. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:58.441135 17605 raft_consensus.cc:1261] T 7e9be84ef6a043818f166f6b47898855 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:58.441440 17864 raft_consensus.cc:1261] T 7e9be84ef6a043818f166f6b47898855 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:58.441944 21530 consensus_queue.cc:1035] T 7e9be84ef6a043818f166f6b47898855 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:58.442510 21530 consensus_queue.cc:1035] T 7e9be84ef6a043818f166f6b47898855 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:58.442835 17734 consensus_queue.cc:237] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } }
I20240502 01:53:58.443652 22174 tablet_copy_client.cc:523] T cf0237a4c65b495ba892871dd850bed9 P dfa1152987644bd781b46c7e42a7a685: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:58.445823 20233 raft_consensus.cc:1261] T 8e7f88b6c509477fb0900024e7de8f00 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:53:58.446547 21530 consensus_queue.cc:1035] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:53:58.446801 17865 raft_consensus.cc:1261] T 8e7f88b6c509477fb0900024e7de8f00 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:53:58.447939 22230 tablet_copy_client.cc:783] T a212c17ba64842c99b3d6a78a3b7cc8a P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:58.448819 21674 consensus_queue.cc:1035] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:53:58.449843 22230 tablet_copy_client.cc:647] T a212c17ba64842c99b3d6a78a3b7cc8a P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:58.449604 21530 raft_consensus.cc:2932] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:53:58.451189 17334 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:53:58.452937 22190 raft_consensus.cc:2932] T 7e9be84ef6a043818f166f6b47898855 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:53:58.454735 20233 raft_consensus.cc:2932] T 8e7f88b6c509477fb0900024e7de8f00 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:53:58.458925 17606 raft_consensus.cc:2932] T 7e9be84ef6a043818f166f6b47898855 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:53:58.461638 17335 catalog_manager.cc:5484] T 8e7f88b6c509477fb0900024e7de8f00 P fdc2ac372492454c85387589a91f63af reported cstate change: config changed from index 5 to 6, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:53:58.464025 17327 catalog_manager.cc:5484] T 7e9be84ef6a043818f166f6b47898855 P dfa1152987644bd781b46c7e42a7a685 reported cstate change: config changed from index -1 to 4, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:58.467535 20496 raft_consensus.cc:1261] T 7131c9bab38e45fa984a84ea6e21dcc5 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:58.468174 22037 consensus_queue.cc:1035] T 7131c9bab38e45fa984a84ea6e21dcc5 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:58.469620 20233 raft_consensus.cc:1261] T 7131c9bab38e45fa984a84ea6e21dcc5 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:58.471859 22037 consensus_queue.cc:1035] T 7131c9bab38e45fa984a84ea6e21dcc5 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:58.479002 22230 tablet_copy_client.cc:523] T a212c17ba64842c99b3d6a78a3b7cc8a P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:58.478833 17865 raft_consensus.cc:2932] T 8e7f88b6c509477fb0900024e7de8f00 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:53:58.482533 22156 raft_consensus.cc:482] T 2714919062f5465c8cde1a3c4720978a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:58.482686 22156 raft_consensus.cc:504] T 2714919062f5465c8cde1a3c4720978a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:58.483148 22230 tablet_bootstrap.cc:492] T a212c17ba64842c99b3d6a78a3b7cc8a P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:53:58.483414 22156 leader_election.cc:288] T 2714919062f5465c8cde1a3c4720978a P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:53:58.484287 20231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2714919062f5465c8cde1a3c4720978a" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:53:58.484547 20231 raft_consensus.cc:2454] T 2714919062f5465c8cde1a3c4720978a P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 0.
I20240502 01:53:58.485289 20498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2714919062f5465c8cde1a3c4720978a" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
I20240502 01:53:58.485491 20498 raft_consensus.cc:2454] T 2714919062f5465c8cde1a3c4720978a P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 0.
I20240502 01:53:58.486119 20303 leader_election.cc:302] T 2714919062f5465c8cde1a3c4720978a P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 6a36c952c51542c0bf7ea3ec2857a9f9; no voters:
I20240502 01:53:58.486538 22156 raft_consensus.cc:2781] T 2714919062f5465c8cde1a3c4720978a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:58.486637 22156 raft_consensus.cc:482] T 2714919062f5465c8cde1a3c4720978a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:58.486708 22156 raft_consensus.cc:3037] T 2714919062f5465c8cde1a3c4720978a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:58.489665 22156 raft_consensus.cc:504] T 2714919062f5465c8cde1a3c4720978a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:58.490290 22156 leader_election.cc:288] T 2714919062f5465c8cde1a3c4720978a P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Requested vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:53:58.495440 20231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2714919062f5465c8cde1a3c4720978a" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:53:58.495697 20231 raft_consensus.cc:3037] T 2714919062f5465c8cde1a3c4720978a P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:58.498451 20231 raft_consensus.cc:2454] T 2714919062f5465c8cde1a3c4720978a P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 1.
I20240502 01:53:58.499246 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8e7f88b6c509477fb0900024e7de8f00 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:58.499903 20498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2714919062f5465c8cde1a3c4720978a" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
I20240502 01:53:58.500097 20498 raft_consensus.cc:3037] T 2714919062f5465c8cde1a3c4720978a P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:58.502849 20498 raft_consensus.cc:2454] T 2714919062f5465c8cde1a3c4720978a P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 1.
I20240502 01:53:58.503827 20303 leader_election.cc:302] T 2714919062f5465c8cde1a3c4720978a P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 6a36c952c51542c0bf7ea3ec2857a9f9; no voters:
I20240502 01:53:58.505154 22156 raft_consensus.cc:2781] T 2714919062f5465c8cde1a3c4720978a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:58.506139 22156 raft_consensus.cc:686] T 2714919062f5465c8cde1a3c4720978a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Becoming Leader. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Running, Role: LEADER
I20240502 01:53:58.507452 17587 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e7f88b6c509477fb0900024e7de8f00 with delete_type TABLET_DATA_TOMBSTONED (TS 3fceccefba274b1dbbd8216af8b0bc29 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55232
I20240502 01:53:58.507853 22210 tablet_replica.cc:330] T 8e7f88b6c509477fb0900024e7de8f00 P 3fceccefba274b1dbbd8216af8b0bc29: stopping tablet replica
I20240502 01:53:58.508080 22210 raft_consensus.cc:2229] T 8e7f88b6c509477fb0900024e7de8f00 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:58.508311 22210 raft_consensus.cc:2258] T 8e7f88b6c509477fb0900024e7de8f00 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:58.509307 22210 ts_tablet_manager.cc:1875] T 8e7f88b6c509477fb0900024e7de8f00 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:58.508183 22156 consensus_queue.cc:237] T 2714919062f5465c8cde1a3c4720978a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:53:58.517328 22210 ts_tablet_manager.cc:1888] T 8e7f88b6c509477fb0900024e7de8f00 P 3fceccefba274b1dbbd8216af8b0bc29: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:53:58.517499 22210 log.cc:1196] T 8e7f88b6c509477fb0900024e7de8f00 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/wals/8e7f88b6c509477fb0900024e7de8f00
I20240502 01:53:58.518337 17319 catalog_manager.cc:4830] TS 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): tablet 8e7f88b6c509477fb0900024e7de8f00 (table rebalance_test_table_1 [id=e431affbb02a4744b67a522521a3ee04]) successfully deleted
I20240502 01:53:58.520931 22174 tablet_bootstrap.cc:492] T cf0237a4c65b495ba892871dd850bed9 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:58.537472 22258 tablet.cc:1722] T 7131c9bab38e45fa984a84ea6e21dcc5 P 2f306549407b4bb281ccf2994035e4f1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:53:58.537994 22258 tablet.cc:1620] T 7131c9bab38e45fa984a84ea6e21dcc5 P 2f306549407b4bb281ccf2994035e4f1: MemRowSet was empty: no flush needed.
I20240502 01:53:58.539273 22169 raft_consensus.cc:2932] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:53:58.547385 20497 raft_consensus.cc:2932] T 03bd47783e884a70bfea1dfab4f2e563 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:53:58.554358 17473 raft_consensus.cc:2932] T 03bd47783e884a70bfea1dfab4f2e563 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:53:58.556157 17333 catalog_manager.cc:5576] TS 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623) does not have the latest schema for tablet 7131c9bab38e45fa984a84ea6e21dcc5 (table rebalancer_dynamic_table [id=624bf9e5554748a88cb806a0e83f5ad4]). Expected version 1 got 0
I20240502 01:53:58.558079 22259 tablet.cc:1722] T 7131c9bab38e45fa984a84ea6e21dcc5 P fdc2ac372492454c85387589a91f63af: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:53:58.558981 22259 tablet.cc:1620] T 7131c9bab38e45fa984a84ea6e21dcc5 P fdc2ac372492454c85387589a91f63af: MemRowSet was empty: no flush needed.
I20240502 01:53:58.559954 17332 catalog_manager.cc:5484] T 2714919062f5465c8cde1a3c4720978a P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70). New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:58.561132 17333 catalog_manager.cc:5484] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 reported cstate change: config changed from index 4 to 5, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:58.566215 17334 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=624bf9e5554748a88cb806a0e83f5ad4] alter complete (version 1)
I20240502 01:53:58.534994 22257 tablet.cc:1722] T 7131c9bab38e45fa984a84ea6e21dcc5 P dfa1152987644bd781b46c7e42a7a685: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:53:58.571280 17330 catalog_manager.cc:5576] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115) does not have the latest schema for tablet 7131c9bab38e45fa984a84ea6e21dcc5 (table rebalancer_dynamic_table [id=624bf9e5554748a88cb806a0e83f5ad4]). Expected version 1 got 0
I20240502 01:53:58.571583 22257 tablet.cc:1620] T 7131c9bab38e45fa984a84ea6e21dcc5 P dfa1152987644bd781b46c7e42a7a685: MemRowSet was empty: no flush needed.
I20240502 01:53:58.571946 17327 catalog_manager.cc:5576] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339) does not have the latest schema for tablet 7131c9bab38e45fa984a84ea6e21dcc5 (table rebalancer_dynamic_table [id=624bf9e5554748a88cb806a0e83f5ad4]). Expected version 1 got 0
I20240502 01:53:58.576490 17864 raft_consensus.cc:2932] T 7e9be84ef6a043818f166f6b47898855 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:53:58.578096 17604 raft_consensus.cc:1261] T 31a4ca0cc44e492fbd13a70ce38add35 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEARNER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:53:58.578285 22230 tablet_bootstrap.cc:492] T a212c17ba64842c99b3d6a78a3b7cc8a P f19cc6cb1e76460aa6711e49a5775091: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:58.578758 20233 raft_consensus.cc:1261] T 31a4ca0cc44e492fbd13a70ce38add35 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:53:58.578745 21579 consensus_queue.cc:1035] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:53:58.579427 22230 tablet_bootstrap.cc:492] T a212c17ba64842c99b3d6a78a3b7cc8a P f19cc6cb1e76460aa6711e49a5775091: Bootstrap complete.
I20240502 01:53:58.579341 22081 consensus_queue.cc:1035] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:53:58.579703 22230 ts_tablet_manager.cc:1367] T a212c17ba64842c99b3d6a78a3b7cc8a P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.097s user 0.025s sys 0.014s
I20240502 01:53:58.579818 17865 raft_consensus.cc:1261] T 31a4ca0cc44e492fbd13a70ce38add35 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:53:58.585660 21579 consensus_queue.cc:1035] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:53:58.587306 22230 raft_consensus.cc:348] T a212c17ba64842c99b3d6a78a3b7cc8a P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:53:58.588910 22230 raft_consensus.cc:729] T a212c17ba64842c99b3d6a78a3b7cc8a P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: LEARNER
I20240502 01:53:58.589175 22230 consensus_queue.cc:260] T a212c17ba64842c99b3d6a78a3b7cc8a P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:53:58.591693 22233 tablet_copy_client.cc:308] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 4da690c0fa13407a85c0689ce8de9ade: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.70:39155
I20240502 01:53:58.592463 20387 tablet_copy_service.cc:140] P 6a36c952c51542c0bf7ea3ec2857a9f9: Received BeginTabletCopySession request for tablet 1d8e2e0f6ed34e4cb004e1c5309c0f1f from peer 4da690c0fa13407a85c0689ce8de9ade ({username='slave'} at 127.11.42.65:57879)
I20240502 01:53:58.592636 20387 tablet_copy_service.cc:161] P 6a36c952c51542c0bf7ea3ec2857a9f9: Beginning new tablet copy session on tablet 1d8e2e0f6ed34e4cb004e1c5309c0f1f from peer 4da690c0fa13407a85c0689ce8de9ade at {username='slave'} at 127.11.42.65:57879: session id = 4da690c0fa13407a85c0689ce8de9ade-1d8e2e0f6ed34e4cb004e1c5309c0f1f
I20240502 01:53:58.593360 22230 ts_tablet_manager.cc:1398] T a212c17ba64842c99b3d6a78a3b7cc8a P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.014s user 0.002s sys 0.001s
I20240502 01:53:58.595388 20387 tablet_copy_source_session.cc:205] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 6a36c952c51542c0bf7ea3ec2857a9f9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:58.595675 21697 raft_consensus.cc:482] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:58.595849 21697 raft_consensus.cc:504] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:58.596460 21697 leader_election.cc:288] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:53:58.597106 20231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d10d76ac7b094fe9b230b1afcbed9ca6" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:53:58.597354 20231 raft_consensus.cc:2454] T d10d76ac7b094fe9b230b1afcbed9ca6 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 0.
I20240502 01:53:58.597765 20434 leader_election.cc:302] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:53:58.598477 20366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d10d76ac7b094fe9b230b1afcbed9ca6" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:53:58.598687 20366 raft_consensus.cc:2454] T d10d76ac7b094fe9b230b1afcbed9ca6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 0.
I20240502 01:53:58.599340 21697 raft_consensus.cc:2781] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:58.599416 20387 tablet_copy_service.cc:342] P 6a36c952c51542c0bf7ea3ec2857a9f9: Request end of tablet copy session f19cc6cb1e76460aa6711e49a5775091-a212c17ba64842c99b3d6a78a3b7cc8a received from {username='slave'} at 127.11.42.72:36559
I20240502 01:53:58.599440 21697 raft_consensus.cc:482] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:58.599522 20387 tablet_copy_service.cc:433] P 6a36c952c51542c0bf7ea3ec2857a9f9: ending tablet copy session f19cc6cb1e76460aa6711e49a5775091-a212c17ba64842c99b3d6a78a3b7cc8a on tablet a212c17ba64842c99b3d6a78a3b7cc8a with peer f19cc6cb1e76460aa6711e49a5775091
I20240502 01:53:58.599532 21697 raft_consensus.cc:3037] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:58.599619 22233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d8e2e0f6ed34e4cb004e1c5309c0f1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:58.599629 22081 raft_consensus.cc:2932] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:53:58.600437 17865 raft_consensus.cc:2932] T 31a4ca0cc44e492fbd13a70ce38add35 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:53:58.602038 21697 raft_consensus.cc:504] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:58.602594 21697 leader_election.cc:288] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 election: Requested vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:53:58.603705 20366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d10d76ac7b094fe9b230b1afcbed9ca6" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
I20240502 01:53:58.603899 20366 raft_consensus.cc:3037] T d10d76ac7b094fe9b230b1afcbed9ca6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:58.606635 20366 raft_consensus.cc:2454] T d10d76ac7b094fe9b230b1afcbed9ca6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 1.
I20240502 01:53:58.607182 20433 leader_election.cc:302] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 6a36c952c51542c0bf7ea3ec2857a9f9; no voters:
I20240502 01:53:58.607522 20231 raft_consensus.cc:2932] T 31a4ca0cc44e492fbd13a70ce38add35 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:53:58.609030 17735 raft_consensus.cc:2932] T 03bd47783e884a70bfea1dfab4f2e563 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:53:58.609922 21697 raft_consensus.cc:2781] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:58.610126 21697 raft_consensus.cc:686] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Running, Role: LEADER
I20240502 01:53:58.610615 21697 consensus_queue.cc:237] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:53:58.613374 20233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d10d76ac7b094fe9b230b1afcbed9ca6" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:53:58.613627 20233 raft_consensus.cc:3037] T d10d76ac7b094fe9b230b1afcbed9ca6 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:58.613456 17604 raft_consensus.cc:2932] T 31a4ca0cc44e492fbd13a70ce38add35 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:53:58.616906 20233 raft_consensus.cc:2454] T d10d76ac7b094fe9b230b1afcbed9ca6 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 1.
I20240502 01:53:58.617080 17327 catalog_manager.cc:5484] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:58.621659 22241 ts_tablet_manager.cc:1888] T 10ec78a08ab04678897a6892988e515f P dfa1152987644bd781b46c7e42a7a685: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:53:58.621843 22241 log.cc:1196] T 10ec78a08ab04678897a6892988e515f P dfa1152987644bd781b46c7e42a7a685: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/wals/10ec78a08ab04678897a6892988e515f
I20240502 01:53:58.622783 17321 catalog_manager.cc:4830] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): tablet 10ec78a08ab04678897a6892988e515f (table rebalance_test_table_1 [id=e431affbb02a4744b67a522521a3ee04]) successfully deleted
I20240502 01:53:58.622572 17333 catalog_manager.cc:5484] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71). New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:58.630012 22233 tablet_copy_client.cc:783] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 4da690c0fa13407a85c0689ce8de9ade: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:58.630290 22233 tablet_copy_client.cc:647] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 4da690c0fa13407a85c0689ce8de9ade: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:58.633965 20366 consensus_queue.cc:237] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } }
I20240502 01:53:58.636760 17604 raft_consensus.cc:1261] T 31a4ca0cc44e492fbd13a70ce38add35 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:58.636943 20233 raft_consensus.cc:1261] T 31a4ca0cc44e492fbd13a70ce38add35 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:58.637367 21579 consensus_queue.cc:1035] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:58.638561 21579 consensus_queue.cc:1035] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:58.638564 17864 consensus_queue.cc:237] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } }
I20240502 01:53:58.641192 20498 raft_consensus.cc:1261] T 03bd47783e884a70bfea1dfab4f2e563 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:53:58.641767 22081 raft_consensus.cc:2932] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:53:58.642392 22169 consensus_queue.cc:1035] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:53:58.643777 22233 tablet_copy_client.cc:523] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 4da690c0fa13407a85c0689ce8de9ade: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:58.644591 21667 raft_consensus.cc:2932] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:53:58.645296 20498 raft_consensus.cc:2932] T 03bd47783e884a70bfea1dfab4f2e563 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:53:58.646767 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 31a4ca0cc44e492fbd13a70ce38add35 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:58.646502 17606 raft_consensus.cc:2932] T 31a4ca0cc44e492fbd13a70ce38add35 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:53:58.648149 17735 raft_consensus.cc:1261] T 03bd47783e884a70bfea1dfab4f2e563 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:53:58.649112 22169 consensus_queue.cc:1035] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:53:58.650749 17734 raft_consensus.cc:2932] T 03bd47783e884a70bfea1dfab4f2e563 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:53:58.652366 17334 catalog_manager.cc:5484] T 31a4ca0cc44e492fbd13a70ce38add35 P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: config changed from index 3 to 4, VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) evicted. New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:53:58.658331 20233 raft_consensus.cc:2932] T 31a4ca0cc44e492fbd13a70ce38add35 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:53:58.659068 17327 catalog_manager.cc:5484] T 03bd47783e884a70bfea1dfab4f2e563 P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: config changed from index 5 to 6, VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) evicted. New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:53:58.661972 17322 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 03bd47783e884a70bfea1dfab4f2e563 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:58.662910 17846 tablet_service.cc:1514] Processing DeleteTablet for tablet 31a4ca0cc44e492fbd13a70ce38add35 with delete_type TABLET_DATA_TOMBSTONED (TS bb31bceba76d466db24b9341c04900c3 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47226
I20240502 01:53:58.667533 22277 tablet_replica.cc:330] T 31a4ca0cc44e492fbd13a70ce38add35 P bb31bceba76d466db24b9341c04900c3: stopping tablet replica
I20240502 01:53:58.667872 22277 raft_consensus.cc:2229] T 31a4ca0cc44e492fbd13a70ce38add35 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:58.668258 22277 raft_consensus.cc:2258] T 31a4ca0cc44e492fbd13a70ce38add35 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:58.669395 22277 ts_tablet_manager.cc:1875] T 31a4ca0cc44e492fbd13a70ce38add35 P bb31bceba76d466db24b9341c04900c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:58.670573 22233 tablet_bootstrap.cc:492] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:53:58.672775 17333 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:53:58.682235 22277 ts_tablet_manager.cc:1888] T 31a4ca0cc44e492fbd13a70ce38add35 P bb31bceba76d466db24b9341c04900c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:53:58.686561 22258 tablet.cc:1722] T 7131c9bab38e45fa984a84ea6e21dcc5 P 2f306549407b4bb281ccf2994035e4f1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:53:58.686745 17454 tablet_service.cc:1514] Processing DeleteTablet for tablet 03bd47783e884a70bfea1dfab4f2e563 with delete_type TABLET_DATA_TOMBSTONED (TS 4da690c0fa13407a85c0689ce8de9ade not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54520
I20240502 01:53:58.687048 22258 tablet.cc:1620] T 7131c9bab38e45fa984a84ea6e21dcc5 P 2f306549407b4bb281ccf2994035e4f1: MemRowSet was empty: no flush needed.
I20240502 01:53:58.687539 22259 tablet.cc:1722] T 7131c9bab38e45fa984a84ea6e21dcc5 P fdc2ac372492454c85387589a91f63af: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:53:58.687976 22259 tablet.cc:1620] T 7131c9bab38e45fa984a84ea6e21dcc5 P fdc2ac372492454c85387589a91f63af: MemRowSet was empty: no flush needed.
I20240502 01:53:58.693257 22277 log.cc:1196] T 31a4ca0cc44e492fbd13a70ce38add35 P bb31bceba76d466db24b9341c04900c3: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/wals/31a4ca0cc44e492fbd13a70ce38add35
I20240502 01:53:58.695367 17322 catalog_manager.cc:4830] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): tablet 31a4ca0cc44e492fbd13a70ce38add35 (table rebalancer_extra_table_8 [id=c5f998b2cdaf41f6a49ea6ae65a975b8]) successfully deleted
I20240502 01:53:58.696962 22257 tablet.cc:1722] T 7131c9bab38e45fa984a84ea6e21dcc5 P dfa1152987644bd781b46c7e42a7a685: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:53:58.697517 22257 tablet.cc:1620] T 7131c9bab38e45fa984a84ea6e21dcc5 P dfa1152987644bd781b46c7e42a7a685: MemRowSet was empty: no flush needed.
I20240502 01:53:58.700619 17333 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=624bf9e5554748a88cb806a0e83f5ad4] alter complete (version 2)
I20240502 01:53:58.702443 22174 tablet_bootstrap.cc:492] T cf0237a4c65b495ba892871dd850bed9 P dfa1152987644bd781b46c7e42a7a685: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:58.703392 22174 tablet_bootstrap.cc:492] T cf0237a4c65b495ba892871dd850bed9 P dfa1152987644bd781b46c7e42a7a685: Bootstrap complete.
I20240502 01:53:58.703657 22174 ts_tablet_manager.cc:1367] T cf0237a4c65b495ba892871dd850bed9 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.183s user 0.007s sys 0.001s
I20240502 01:53:58.715739 22174 raft_consensus.cc:348] T cf0237a4c65b495ba892871dd850bed9 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:53:58.718223 22174 raft_consensus.cc:729] T cf0237a4c65b495ba892871dd850bed9 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: LEARNER
I20240502 01:53:58.718322 17864 consensus_queue.cc:237] T de174ba4f7d848718ab8dbeb74369055 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } }
I20240502 01:53:58.719141 22174 consensus_queue.cc:260] T cf0237a4c65b495ba892871dd850bed9 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:53:58.726684 22174 ts_tablet_manager.cc:1398] T cf0237a4c65b495ba892871dd850bed9 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20240502 01:53:58.729830 20650 tablet_copy_service.cc:342] P f19cc6cb1e76460aa6711e49a5775091: Request end of tablet copy session dfa1152987644bd781b46c7e42a7a685-cf0237a4c65b495ba892871dd850bed9 received from {username='slave'} at 127.11.42.67:36277
I20240502 01:53:58.729960 20650 tablet_copy_service.cc:433] P f19cc6cb1e76460aa6711e49a5775091: ending tablet copy session dfa1152987644bd781b46c7e42a7a685-cf0237a4c65b495ba892871dd850bed9 on tablet cf0237a4c65b495ba892871dd850bed9 with peer dfa1152987644bd781b46c7e42a7a685
W20240502 01:53:58.736229 17802 consensus_peers.cc:485] T de174ba4f7d848718ab8dbeb74369055 P bb31bceba76d466db24b9341c04900c3 -> Peer 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): Couldn't send request to peer 6a36c952c51542c0bf7ea3ec2857a9f9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: de174ba4f7d848718ab8dbeb74369055. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:58.737161 17734 raft_consensus.cc:1261] T de174ba4f7d848718ab8dbeb74369055 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:58.737694 22169 consensus_queue.cc:1035] T de174ba4f7d848718ab8dbeb74369055 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:58.737948 22279 tablet_replica.cc:330] T 03bd47783e884a70bfea1dfab4f2e563 P 4da690c0fa13407a85c0689ce8de9ade: stopping tablet replica
I20240502 01:53:58.739877 22279 raft_consensus.cc:2229] T 03bd47783e884a70bfea1dfab4f2e563 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:58.740203 22279 raft_consensus.cc:2258] T 03bd47783e884a70bfea1dfab4f2e563 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:58.744745 17604 raft_consensus.cc:1261] T de174ba4f7d848718ab8dbeb74369055 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:58.745342 22169 consensus_queue.cc:1035] T de174ba4f7d848718ab8dbeb74369055 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:58.746909 22279 ts_tablet_manager.cc:1875] T 03bd47783e884a70bfea1dfab4f2e563 P 4da690c0fa13407a85c0689ce8de9ade: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:58.766064 22177 raft_consensus.cc:2932] T de174ba4f7d848718ab8dbeb74369055 P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } } }
I20240502 01:53:58.766692 17606 raft_consensus.cc:2932] T de174ba4f7d848718ab8dbeb74369055 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } } }
I20240502 01:53:58.775158 17330 catalog_manager.cc:5484] T de174ba4f7d848718ab8dbeb74369055 P bb31bceba76d466db24b9341c04900c3 reported cstate change: config changed from index -1 to 4, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:58.790912 22279 ts_tablet_manager.cc:1888] T 03bd47783e884a70bfea1dfab4f2e563 P 4da690c0fa13407a85c0689ce8de9ade: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:53:58.791061 22279 log.cc:1196] T 03bd47783e884a70bfea1dfab4f2e563 P 4da690c0fa13407a85c0689ce8de9ade: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/03bd47783e884a70bfea1dfab4f2e563
I20240502 01:53:58.791922 17735 raft_consensus.cc:2932] T de174ba4f7d848718ab8dbeb74369055 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } } }
I20240502 01:53:58.793742 22233 tablet_bootstrap.cc:492] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:58.794203 22233 tablet_bootstrap.cc:492] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap complete.
I20240502 01:53:58.794473 22233 ts_tablet_manager.cc:1367] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.124s user 0.033s sys 0.024s
I20240502 01:53:58.806193 17321 catalog_manager.cc:4830] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): tablet 03bd47783e884a70bfea1dfab4f2e563 (table rebalance_test_table_3 [id=b996bd60793c4d3c83bf34f4989da909]) successfully deleted
I20240502 01:53:58.806113 22233 raft_consensus.cc:348] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } }
I20240502 01:53:58.806478 22233 raft_consensus.cc:729] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: LEARNER
I20240502 01:53:58.806783 22233 consensus_queue.cc:260] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } }
I20240502 01:53:58.807507 22233 ts_tablet_manager.cc:1398] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20240502 01:53:58.808271 20387 tablet_copy_service.cc:342] P 6a36c952c51542c0bf7ea3ec2857a9f9: Request end of tablet copy session 4da690c0fa13407a85c0689ce8de9ade-1d8e2e0f6ed34e4cb004e1c5309c0f1f received from {username='slave'} at 127.11.42.65:57879
I20240502 01:53:58.808387 20387 tablet_copy_service.cc:433] P 6a36c952c51542c0bf7ea3ec2857a9f9: ending tablet copy session 4da690c0fa13407a85c0689ce8de9ade-1d8e2e0f6ed34e4cb004e1c5309c0f1f on tablet 1d8e2e0f6ed34e4cb004e1c5309c0f1f with peer 4da690c0fa13407a85c0689ce8de9ade
I20240502 01:53:58.809688 17330 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:53:58.824198 22257 tablet.cc:1722] T 7131c9bab38e45fa984a84ea6e21dcc5 P dfa1152987644bd781b46c7e42a7a685: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:53:58.828883 22258 tablet.cc:1722] T 7131c9bab38e45fa984a84ea6e21dcc5 P 2f306549407b4bb281ccf2994035e4f1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:53:58.829425 22259 tablet.cc:1722] T 7131c9bab38e45fa984a84ea6e21dcc5 P fdc2ac372492454c85387589a91f63af: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:53:58.834178 17330 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table_new_name [id=624bf9e5554748a88cb806a0e83f5ad4] alter complete (version 3)
I20240502 01:53:58.866613 20628 raft_consensus.cc:1203] T a212c17ba64842c99b3d6a78a3b7cc8a P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:53:58.925673 21429 consensus_queue.cc:1035] T 75bb4e19cf57405fabcb532b55a29ced P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:58.931075 21934 consensus_queue.cc:1035] T 75bb4e19cf57405fabcb532b55a29ced P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:58.938915 17330 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:53:58.939121 17330 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:53:58.942756 17330 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet 7131c9bab38e45fa984a84ea6e21dcc5
I20240502 01:53:58.943553 17716 tablet_service.cc:1514] Processing DeleteTablet for tablet 7131c9bab38e45fa984a84ea6e21dcc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:58 UTC) from {username='slave'} at 127.0.0.1:52198
I20240502 01:53:58.943863 22241 tablet_replica.cc:330] T 7131c9bab38e45fa984a84ea6e21dcc5 P dfa1152987644bd781b46c7e42a7a685: stopping tablet replica
I20240502 01:53:58.943852 20211 tablet_service.cc:1514] Processing DeleteTablet for tablet 7131c9bab38e45fa984a84ea6e21dcc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:58 UTC) from {username='slave'} at 127.0.0.1:44624
I20240502 01:53:58.944083 22241 raft_consensus.cc:2229] T 7131c9bab38e45fa984a84ea6e21dcc5 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:58.944226 20477 tablet_service.cc:1514] Processing DeleteTablet for tablet 7131c9bab38e45fa984a84ea6e21dcc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:58 UTC) from {username='slave'} at 127.0.0.1:35018
I20240502 01:53:58.944447 22241 raft_consensus.cc:2258] T 7131c9bab38e45fa984a84ea6e21dcc5 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:58.945506 22241 ts_tablet_manager.cc:1875] T 7131c9bab38e45fa984a84ea6e21dcc5 P dfa1152987644bd781b46c7e42a7a685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:58.947031 21579 consensus_queue.cc:1035] T 2714919062f5465c8cde1a3c4720978a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:58.952126 22241 ts_tablet_manager.cc:1888] T 7131c9bab38e45fa984a84ea6e21dcc5 P dfa1152987644bd781b46c7e42a7a685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:53:58.952301 22241 log.cc:1196] T 7131c9bab38e45fa984a84ea6e21dcc5 P dfa1152987644bd781b46c7e42a7a685: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/wals/7131c9bab38e45fa984a84ea6e21dcc5
I20240502 01:53:58.952809 22241 ts_tablet_manager.cc:1909] T 7131c9bab38e45fa984a84ea6e21dcc5 P dfa1152987644bd781b46c7e42a7a685: Deleting consensus metadata
I20240502 01:53:58.953925 17321 catalog_manager.cc:4830] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): tablet 7131c9bab38e45fa984a84ea6e21dcc5 (table rebalancer_dynamic_table_new_name [id=624bf9e5554748a88cb806a0e83f5ad4]) successfully deleted
I20240502 01:53:58.955358 22337 ts_tablet_manager.cc:902] T 7e9be84ef6a043818f166f6b47898855 P 2f306549407b4bb281ccf2994035e4f1: Initiating tablet copy from peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:53:58.957546 22338 tablet_replica.cc:330] T 7131c9bab38e45fa984a84ea6e21dcc5 P fdc2ac372492454c85387589a91f63af: stopping tablet replica
I20240502 01:53:58.957839 22338 raft_consensus.cc:2229] T 7131c9bab38e45fa984a84ea6e21dcc5 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:58.958123 22338 raft_consensus.cc:2258] T 7131c9bab38e45fa984a84ea6e21dcc5 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:58.959136 22338 ts_tablet_manager.cc:1875] T 7131c9bab38e45fa984a84ea6e21dcc5 P fdc2ac372492454c85387589a91f63af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:58.963565 22081 consensus_queue.cc:1035] T 2714919062f5465c8cde1a3c4720978a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:58.977452 22338 ts_tablet_manager.cc:1888] T 7131c9bab38e45fa984a84ea6e21dcc5 P fdc2ac372492454c85387589a91f63af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:53:58.977653 22338 log.cc:1196] T 7131c9bab38e45fa984a84ea6e21dcc5 P fdc2ac372492454c85387589a91f63af: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal/wals/7131c9bab38e45fa984a84ea6e21dcc5
I20240502 01:53:58.978595 22338 ts_tablet_manager.cc:1909] T 7131c9bab38e45fa984a84ea6e21dcc5 P fdc2ac372492454c85387589a91f63af: Deleting consensus metadata
I20240502 01:53:58.979869 17322 catalog_manager.cc:4830] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): tablet 7131c9bab38e45fa984a84ea6e21dcc5 (table rebalancer_dynamic_table_new_name [id=624bf9e5554748a88cb806a0e83f5ad4]) successfully deleted
I20240502 01:53:58.980755 22339 tablet_replica.cc:330] T 7131c9bab38e45fa984a84ea6e21dcc5 P 2f306549407b4bb281ccf2994035e4f1: stopping tablet replica
I20240502 01:53:58.981068 22339 raft_consensus.cc:2229] T 7131c9bab38e45fa984a84ea6e21dcc5 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:58.981448 22339 raft_consensus.cc:2258] T 7131c9bab38e45fa984a84ea6e21dcc5 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:58.983044 22339 ts_tablet_manager.cc:1875] T 7131c9bab38e45fa984a84ea6e21dcc5 P 2f306549407b4bb281ccf2994035e4f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:58.985920 17734 raft_consensus.cc:1203] T cf0237a4c65b495ba892871dd850bed9 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:53:58.994657 22339 ts_tablet_manager.cc:1888] T 7131c9bab38e45fa984a84ea6e21dcc5 P 2f306549407b4bb281ccf2994035e4f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:53:58.994805 22339 log.cc:1196] T 7131c9bab38e45fa984a84ea6e21dcc5 P 2f306549407b4bb281ccf2994035e4f1: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal/wals/7131c9bab38e45fa984a84ea6e21dcc5
I20240502 01:53:58.995298 22339 ts_tablet_manager.cc:1909] T 7131c9bab38e45fa984a84ea6e21dcc5 P 2f306549407b4bb281ccf2994035e4f1: Deleting consensus metadata
I20240502 01:53:58.996392 17322 catalog_manager.cc:4830] TS 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): tablet 7131c9bab38e45fa984a84ea6e21dcc5 (table rebalancer_dynamic_table_new_name [id=624bf9e5554748a88cb806a0e83f5ad4]) successfully deleted
I20240502 01:53:58.998474 22337 tablet_copy_client.cc:308] T 7e9be84ef6a043818f166f6b47898855 P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.67:36115
I20240502 01:53:58.999225 17757 tablet_copy_service.cc:140] P dfa1152987644bd781b46c7e42a7a685: Received BeginTabletCopySession request for tablet 7e9be84ef6a043818f166f6b47898855 from peer 2f306549407b4bb281ccf2994035e4f1 ({username='slave'} at 127.11.42.71:49495)
I20240502 01:53:58.999395 17757 tablet_copy_service.cc:161] P dfa1152987644bd781b46c7e42a7a685: Beginning new tablet copy session on tablet 7e9be84ef6a043818f166f6b47898855 from peer 2f306549407b4bb281ccf2994035e4f1 at {username='slave'} at 127.11.42.71:49495: session id = 2f306549407b4bb281ccf2994035e4f1-7e9be84ef6a043818f166f6b47898855
I20240502 01:53:59.001636 17757 tablet_copy_source_session.cc:205] T 7e9be84ef6a043818f166f6b47898855 P dfa1152987644bd781b46c7e42a7a685: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:59.002915 22337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e9be84ef6a043818f166f6b47898855. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:59.009173 22337 tablet_copy_client.cc:783] T 7e9be84ef6a043818f166f6b47898855 P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:59.009554 22337 tablet_copy_client.cc:647] T 7e9be84ef6a043818f166f6b47898855 P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:59.012212 22337 tablet_copy_client.cc:523] T 7e9be84ef6a043818f166f6b47898855 P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:59.017149 22337 tablet_bootstrap.cc:492] T 7e9be84ef6a043818f166f6b47898855 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:53:59.047744 17330 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:53:59.056170 20211 tablet_service.cc:1467] Processing CreateTablet for tablet 2259f02e0919491e8077ef5daefe3b02 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4654334e0d494a9c8455f530bb7c550b]), partition=
I20240502 01:53:59.056483 20477 tablet_service.cc:1467] Processing CreateTablet for tablet 2259f02e0919491e8077ef5daefe3b02 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4654334e0d494a9c8455f530bb7c550b]), partition=
I20240502 01:53:59.056526 20211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2259f02e0919491e8077ef5daefe3b02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:59.056805 20477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2259f02e0919491e8077ef5daefe3b02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:59.061110 20609 tablet_service.cc:1467] Processing CreateTablet for tablet 2259f02e0919491e8077ef5daefe3b02 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4654334e0d494a9c8455f530bb7c550b]), partition=
I20240502 01:53:59.061458 20609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2259f02e0919491e8077ef5daefe3b02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:59.072873 22337 tablet_bootstrap.cc:492] T 7e9be84ef6a043818f166f6b47898855 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:59.073359 22337 tablet_bootstrap.cc:492] T 7e9be84ef6a043818f166f6b47898855 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap complete.
I20240502 01:53:59.073652 22337 ts_tablet_manager.cc:1367] T 7e9be84ef6a043818f166f6b47898855 P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.057s user 0.040s sys 0.008s
I20240502 01:53:59.074455 22337 raft_consensus.cc:348] T 7e9be84ef6a043818f166f6b47898855 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
I20240502 01:53:59.074985 22337 raft_consensus.cc:729] T 7e9be84ef6a043818f166f6b47898855 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: LEARNER
I20240502 01:53:59.075171 22365 tablet_bootstrap.cc:492] T 2259f02e0919491e8077ef5daefe3b02 P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:53:59.075213 22337 consensus_queue.cc:260] T 7e9be84ef6a043818f166f6b47898855 P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
I20240502 01:53:59.075853 22337 ts_tablet_manager.cc:1398] T 7e9be84ef6a043818f166f6b47898855 P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:59.076819 17757 tablet_copy_service.cc:342] P dfa1152987644bd781b46c7e42a7a685: Request end of tablet copy session 2f306549407b4bb281ccf2994035e4f1-7e9be84ef6a043818f166f6b47898855 received from {username='slave'} at 127.11.42.71:49495
I20240502 01:53:59.076973 17757 tablet_copy_service.cc:433] P dfa1152987644bd781b46c7e42a7a685: ending tablet copy session 2f306549407b4bb281ccf2994035e4f1-7e9be84ef6a043818f166f6b47898855 on tablet 7e9be84ef6a043818f166f6b47898855 with peer 2f306549407b4bb281ccf2994035e4f1
I20240502 01:53:59.077595 22365 tablet_bootstrap.cc:654] T 2259f02e0919491e8077ef5daefe3b02 P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:59.087147 22364 tablet_bootstrap.cc:492] T 2259f02e0919491e8077ef5daefe3b02 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:53:59.087839 22370 tablet_bootstrap.cc:492] T 2259f02e0919491e8077ef5daefe3b02 P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:53:59.089206 22364 tablet_bootstrap.cc:654] T 2259f02e0919491e8077ef5daefe3b02 P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:59.090381 22365 tablet_bootstrap.cc:492] T 2259f02e0919491e8077ef5daefe3b02 P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:53:59.090636 22365 ts_tablet_manager.cc:1367] T 2259f02e0919491e8077ef5daefe3b02 P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20240502 01:53:59.090780 22370 tablet_bootstrap.cc:654] T 2259f02e0919491e8077ef5daefe3b02 P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:59.091480 22365 raft_consensus.cc:348] T 2259f02e0919491e8077ef5daefe3b02 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:59.091802 22365 raft_consensus.cc:374] T 2259f02e0919491e8077ef5daefe3b02 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:59.092007 22365 raft_consensus.cc:729] T 2259f02e0919491e8077ef5daefe3b02 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:53:59.092340 22365 consensus_queue.cc:260] T 2259f02e0919491e8077ef5daefe3b02 P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:59.093312 22365 ts_tablet_manager.cc:1398] T 2259f02e0919491e8077ef5daefe3b02 P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:59.094564 22370 tablet_bootstrap.cc:492] T 2259f02e0919491e8077ef5daefe3b02 P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:53:59.094719 22370 ts_tablet_manager.cc:1367] T 2259f02e0919491e8077ef5daefe3b02 P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20240502 01:53:59.095466 22370 raft_consensus.cc:348] T 2259f02e0919491e8077ef5daefe3b02 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:59.095678 22370 raft_consensus.cc:374] T 2259f02e0919491e8077ef5daefe3b02 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:59.095849 22370 raft_consensus.cc:729] T 2259f02e0919491e8077ef5daefe3b02 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:53:59.096089 22370 consensus_queue.cc:260] T 2259f02e0919491e8077ef5daefe3b02 P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:59.096675 22370 ts_tablet_manager.cc:1398] T 2259f02e0919491e8077ef5daefe3b02 P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:53:59.099125 22364 tablet_bootstrap.cc:492] T 2259f02e0919491e8077ef5daefe3b02 P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:53:59.099408 22364 ts_tablet_manager.cc:1367] T 2259f02e0919491e8077ef5daefe3b02 P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.005s
I20240502 01:53:59.100107 22364 raft_consensus.cc:348] T 2259f02e0919491e8077ef5daefe3b02 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:59.100374 22364 raft_consensus.cc:374] T 2259f02e0919491e8077ef5daefe3b02 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:59.100520 22364 raft_consensus.cc:729] T 2259f02e0919491e8077ef5daefe3b02 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:53:59.100848 22364 consensus_queue.cc:260] T 2259f02e0919491e8077ef5daefe3b02 P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:59.102525 22364 ts_tablet_manager.cc:1398] T 2259f02e0919491e8077ef5daefe3b02 P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:53:59.112287 21924 consensus_queue.cc:1035] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:59.138871 17606 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f0b3862e504d4b43a60477e5efef29b1"
dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37844
I20240502 01:53:59.139170 17606 raft_consensus.cc:595] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:53:59.140997 21924 consensus_queue.cc:1035] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:59.157831 17330 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_extra_table_22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:59.168021 20211 tablet_service.cc:1467] Processing CreateTablet for tablet b6d89235867f42418f8205167bb29755 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=2947767c2c5e4b8ea491bf140246cfa4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:59.168340 20346 tablet_service.cc:1467] Processing CreateTablet for tablet b6d89235867f42418f8205167bb29755 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=2947767c2c5e4b8ea491bf140246cfa4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:59.168499 20211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6d89235867f42418f8205167bb29755. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:59.168821 20346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6d89235867f42418f8205167bb29755. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:59.169044 20477 tablet_service.cc:1467] Processing CreateTablet for tablet b950e18825ac434ba44586d544667918 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=2947767c2c5e4b8ea491bf140246cfa4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:59.169420 20477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b950e18825ac434ba44586d544667918. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:59.173393 17587 tablet_service.cc:1467] Processing CreateTablet for tablet b6d89235867f42418f8205167bb29755 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=2947767c2c5e4b8ea491bf140246cfa4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:59.173741 17587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6d89235867f42418f8205167bb29755. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:59.175263 22400 tablet_bootstrap.cc:492] T b6d89235867f42418f8205167bb29755 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:53:59.177184 20609 tablet_service.cc:1467] Processing CreateTablet for tablet 2152c07e448b4582abff22b034363c70 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=2947767c2c5e4b8ea491bf140246cfa4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:59.177399 20347 tablet_service.cc:1467] Processing CreateTablet for tablet b950e18825ac434ba44586d544667918 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=2947767c2c5e4b8ea491bf140246cfa4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:59.177613 20609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2152c07e448b4582abff22b034363c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:59.177783 20347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b950e18825ac434ba44586d544667918. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:59.179030 22400 tablet_bootstrap.cc:654] T b6d89235867f42418f8205167bb29755 P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:59.181782 20345 tablet_service.cc:1467] Processing CreateTablet for tablet 2152c07e448b4582abff22b034363c70 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=2947767c2c5e4b8ea491bf140246cfa4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:59.182153 20345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2152c07e448b4582abff22b034363c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:59.183797 22370 tablet_bootstrap.cc:492] T 2152c07e448b4582abff22b034363c70 P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:53:59.185395 22370 tablet_bootstrap.cc:654] T 2152c07e448b4582abff22b034363c70 P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:59.186225 17716 tablet_service.cc:1467] Processing CreateTablet for tablet b950e18825ac434ba44586d544667918 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=2947767c2c5e4b8ea491bf140246cfa4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:59.186661 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b950e18825ac434ba44586d544667918. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:59.190799 22370 tablet_bootstrap.cc:492] T 2152c07e448b4582abff22b034363c70 P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:53:59.190943 17846 tablet_service.cc:1467] Processing CreateTablet for tablet 2152c07e448b4582abff22b034363c70 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=2947767c2c5e4b8ea491bf140246cfa4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:59.191366 17846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2152c07e448b4582abff22b034363c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:59.192584 22370 ts_tablet_manager.cc:1367] T 2152c07e448b4582abff22b034363c70 P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.001s
I20240502 01:53:59.192728 22400 tablet_bootstrap.cc:492] T b6d89235867f42418f8205167bb29755 P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:53:59.192863 22400 ts_tablet_manager.cc:1367] T b6d89235867f42418f8205167bb29755 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.007s
I20240502 01:53:59.193557 22370 raft_consensus.cc:348] T 2152c07e448b4582abff22b034363c70 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:59.193764 22370 raft_consensus.cc:374] T 2152c07e448b4582abff22b034363c70 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:59.193833 22370 raft_consensus.cc:729] T 2152c07e448b4582abff22b034363c70 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:53:59.193712 22400 raft_consensus.cc:348] T b6d89235867f42418f8205167bb29755 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:59.193910 22400 raft_consensus.cc:374] T b6d89235867f42418f8205167bb29755 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:59.193987 22400 raft_consensus.cc:729] T b6d89235867f42418f8205167bb29755 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:53:59.194123 22370 consensus_queue.cc:260] T 2152c07e448b4582abff22b034363c70 P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:59.194211 22400 consensus_queue.cc:260] T b6d89235867f42418f8205167bb29755 P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:59.195703 22400 ts_tablet_manager.cc:1398] T b6d89235867f42418f8205167bb29755 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:53:59.197584 22370 ts_tablet_manager.cc:1398] T 2152c07e448b4582abff22b034363c70 P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 01:53:59.202577 22404 tablet_bootstrap.cc:492] T 2152c07e448b4582abff22b034363c70 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:53:59.204164 22404 tablet_bootstrap.cc:654] T 2152c07e448b4582abff22b034363c70 P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:59.204815 22365 tablet_bootstrap.cc:492] T b6d89235867f42418f8205167bb29755 P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:53:59.204885 22364 tablet_bootstrap.cc:492] T b950e18825ac434ba44586d544667918 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:53:59.206431 22364 tablet_bootstrap.cc:654] T b950e18825ac434ba44586d544667918 P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:59.206425 22365 tablet_bootstrap.cc:654] T b6d89235867f42418f8205167bb29755 P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:59.210866 22400 tablet_bootstrap.cc:492] T 2152c07e448b4582abff22b034363c70 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:53:59.215365 22400 tablet_bootstrap.cc:654] T 2152c07e448b4582abff22b034363c70 P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:59.223013 22403 tablet_bootstrap.cc:492] T b6d89235867f42418f8205167bb29755 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:53:59.224756 22403 tablet_bootstrap.cc:654] T b6d89235867f42418f8205167bb29755 P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:59.243016 22201 raft_consensus.cc:979] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29: : Instructing follower bb31bceba76d466db24b9341c04900c3 to start an election
I20240502 01:53:59.243152 22201 raft_consensus.cc:1067] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Signalling peer bb31bceba76d466db24b9341c04900c3 to start an election
I20240502 01:53:59.244169 17864 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f0b3862e504d4b43a60477e5efef29b1"
dest_uuid: "bb31bceba76d466db24b9341c04900c3"
from {username='slave'} at 127.11.42.66:41833
I20240502 01:53:59.244423 17864 raft_consensus.cc:482] T f0b3862e504d4b43a60477e5efef29b1 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:53:59.244541 17864 raft_consensus.cc:3037] T f0b3862e504d4b43a60477e5efef29b1 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:59.247753 17864 raft_consensus.cc:504] T f0b3862e504d4b43a60477e5efef29b1 P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } }
I20240502 01:53:59.249351 17606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b3862e504d4b43a60477e5efef29b1" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:53:59.249580 17606 raft_consensus.cc:3032] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:53:59.249657 17606 raft_consensus.cc:729] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Running, Role: LEADER
I20240502 01:53:59.249962 17606 consensus_queue.cc:260] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } }
I20240502 01:53:59.250377 17606 raft_consensus.cc:3037] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:59.252065 22408 ts_tablet_manager.cc:902] T de174ba4f7d848718ab8dbeb74369055 P 6a36c952c51542c0bf7ea3ec2857a9f9: Initiating tablet copy from peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:53:59.253878 20233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b3862e504d4b43a60477e5efef29b1" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:53:59.254102 20233 raft_consensus.cc:3037] T f0b3862e504d4b43a60477e5efef29b1 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:59.256845 20233 raft_consensus.cc:2454] T f0b3862e504d4b43a60477e5efef29b1 P fdc2ac372492454c85387589a91f63af [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 2.
I20240502 01:53:59.257900 17734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b3862e504d4b43a60477e5efef29b1" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:53:59.258149 17734 raft_consensus.cc:3037] T f0b3862e504d4b43a60477e5efef29b1 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:59.259764 22408 tablet_copy_client.cc:308] T de174ba4f7d848718ab8dbeb74369055 P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.68:44083
I20240502 01:53:59.260830 17734 raft_consensus.cc:2454] T f0b3862e504d4b43a60477e5efef29b1 P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 2.
I20240502 01:53:59.260995 22411 tablet_bootstrap.cc:492] T b950e18825ac434ba44586d544667918 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:53:59.261432 17887 tablet_copy_service.cc:140] P bb31bceba76d466db24b9341c04900c3: Received BeginTabletCopySession request for tablet de174ba4f7d848718ab8dbeb74369055 from peer 6a36c952c51542c0bf7ea3ec2857a9f9 ({username='slave'} at 127.11.42.70:41203)
I20240502 01:53:59.261612 17887 tablet_copy_service.cc:161] P bb31bceba76d466db24b9341c04900c3: Beginning new tablet copy session on tablet de174ba4f7d848718ab8dbeb74369055 from peer 6a36c952c51542c0bf7ea3ec2857a9f9 at {username='slave'} at 127.11.42.70:41203: session id = 6a36c952c51542c0bf7ea3ec2857a9f9-de174ba4f7d848718ab8dbeb74369055
I20240502 01:53:59.262773 22411 tablet_bootstrap.cc:654] T b950e18825ac434ba44586d544667918 P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:59.264839 17887 tablet_copy_source_session.cc:205] T de174ba4f7d848718ab8dbeb74369055 P bb31bceba76d466db24b9341c04900c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:59.267545 17802 leader_election.cc:302] T f0b3862e504d4b43a60477e5efef29b1 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: bb31bceba76d466db24b9341c04900c3, dfa1152987644bd781b46c7e42a7a685, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:53:59.267935 22177 raft_consensus.cc:2781] T f0b3862e504d4b43a60477e5efef29b1 P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:53:59.268096 22177 raft_consensus.cc:686] T f0b3862e504d4b43a60477e5efef29b1 P bb31bceba76d466db24b9341c04900c3 [term 2 LEADER]: Becoming Leader. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Running, Role: LEADER
I20240502 01:53:59.268373 22177 consensus_queue.cc:237] T f0b3862e504d4b43a60477e5efef29b1 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } }
I20240502 01:53:59.272321 17606 raft_consensus.cc:2454] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 2.
I20240502 01:53:59.272922 17864 leader_election.cc:288] T f0b3862e504d4b43a60477e5efef29b1 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 election: Requested vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:53:59.277683 22408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de174ba4f7d848718ab8dbeb74369055. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:59.283362 17333 catalog_manager.cc:5484] T f0b3862e504d4b43a60477e5efef29b1 P bb31bceba76d466db24b9341c04900c3 reported cstate change: term changed from 1 to 2, leader changed from 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) to bb31bceba76d466db24b9341c04900c3 (127.11.42.68). New cstate: current_term: 2 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:59.284900 22404 tablet_bootstrap.cc:492] T 2152c07e448b4582abff22b034363c70 P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:53:59.285158 22404 ts_tablet_manager.cc:1367] T 2152c07e448b4582abff22b034363c70 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.083s user 0.021s sys 0.029s
I20240502 01:53:59.286093 22404 raft_consensus.cc:348] T 2152c07e448b4582abff22b034363c70 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:59.286396 22404 raft_consensus.cc:374] T 2152c07e448b4582abff22b034363c70 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:59.286535 22404 raft_consensus.cc:729] T 2152c07e448b4582abff22b034363c70 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:53:59.286871 22404 consensus_queue.cc:260] T 2152c07e448b4582abff22b034363c70 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:59.288569 22404 ts_tablet_manager.cc:1398] T 2152c07e448b4582abff22b034363c70 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:53:59.289269 22365 tablet_bootstrap.cc:492] T b6d89235867f42418f8205167bb29755 P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:53:59.289269 22364 tablet_bootstrap.cc:492] T b950e18825ac434ba44586d544667918 P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:53:59.289460 22365 ts_tablet_manager.cc:1367] T b6d89235867f42418f8205167bb29755 P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.085s user 0.006s sys 0.007s
I20240502 01:53:59.289494 22364 ts_tablet_manager.cc:1367] T b950e18825ac434ba44586d544667918 P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.085s user 0.009s sys 0.000s
I20240502 01:53:59.290275 22365 raft_consensus.cc:348] T b6d89235867f42418f8205167bb29755 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:59.290508 22365 raft_consensus.cc:374] T b6d89235867f42418f8205167bb29755 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:59.290417 22364 raft_consensus.cc:348] T b950e18825ac434ba44586d544667918 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:59.290575 22365 raft_consensus.cc:729] T b6d89235867f42418f8205167bb29755 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:53:59.290622 22364 raft_consensus.cc:374] T b950e18825ac434ba44586d544667918 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:59.290688 22364 raft_consensus.cc:729] T b950e18825ac434ba44586d544667918 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:53:59.290822 22365 consensus_queue.cc:260] T b6d89235867f42418f8205167bb29755 P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:59.290902 22364 consensus_queue.cc:260] T b950e18825ac434ba44586d544667918 P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:59.291355 22365 ts_tablet_manager.cc:1398] T b6d89235867f42418f8205167bb29755 P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:53:59.291399 22364 ts_tablet_manager.cc:1398] T b950e18825ac434ba44586d544667918 P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:59.292711 22411 tablet_bootstrap.cc:492] T b950e18825ac434ba44586d544667918 P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:53:59.292912 22411 ts_tablet_manager.cc:1367] T b950e18825ac434ba44586d544667918 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.000s
I20240502 01:53:59.294579 22411 raft_consensus.cc:348] T b950e18825ac434ba44586d544667918 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:59.297083 22411 raft_consensus.cc:374] T b950e18825ac434ba44586d544667918 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:59.298022 22411 raft_consensus.cc:729] T b950e18825ac434ba44586d544667918 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:53:59.298331 22411 consensus_queue.cc:260] T b950e18825ac434ba44586d544667918 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:59.298977 22400 tablet_bootstrap.cc:492] T 2152c07e448b4582abff22b034363c70 P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:53:59.299371 22400 ts_tablet_manager.cc:1367] T 2152c07e448b4582abff22b034363c70 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.089s user 0.002s sys 0.021s
I20240502 01:53:59.299983 22411 ts_tablet_manager.cc:1398] T b950e18825ac434ba44586d544667918 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:53:59.300303 22400 raft_consensus.cc:348] T 2152c07e448b4582abff22b034363c70 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:59.300612 22400 raft_consensus.cc:374] T 2152c07e448b4582abff22b034363c70 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:59.300693 22400 raft_consensus.cc:729] T 2152c07e448b4582abff22b034363c70 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:53:59.300938 22400 consensus_queue.cc:260] T 2152c07e448b4582abff22b034363c70 P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:59.301712 22400 ts_tablet_manager.cc:1398] T 2152c07e448b4582abff22b034363c70 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:53:59.302126 22400 tablet_bootstrap.cc:492] T b950e18825ac434ba44586d544667918 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:53:59.303979 22400 tablet_bootstrap.cc:654] T b950e18825ac434ba44586d544667918 P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:59.309108 22408 tablet_copy_client.cc:783] T de174ba4f7d848718ab8dbeb74369055 P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:59.309417 22408 tablet_copy_client.cc:647] T de174ba4f7d848718ab8dbeb74369055 P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:59.311123 22408 tablet_copy_client.cc:523] T de174ba4f7d848718ab8dbeb74369055 P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:59.313753 22400 tablet_bootstrap.cc:492] T b950e18825ac434ba44586d544667918 P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:53:59.313935 22400 ts_tablet_manager.cc:1367] T b950e18825ac434ba44586d544667918 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.005s
I20240502 01:53:59.314726 22400 raft_consensus.cc:348] T b950e18825ac434ba44586d544667918 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:59.315356 22400 raft_consensus.cc:374] T b950e18825ac434ba44586d544667918 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:59.315510 22400 raft_consensus.cc:729] T b950e18825ac434ba44586d544667918 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:53:59.315775 22400 consensus_queue.cc:260] T b950e18825ac434ba44586d544667918 P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:59.318105 22400 ts_tablet_manager.cc:1398] T b950e18825ac434ba44586d544667918 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240502 01:53:59.318267 22403 tablet_bootstrap.cc:492] T b6d89235867f42418f8205167bb29755 P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:53:59.318419 22403 ts_tablet_manager.cc:1367] T b6d89235867f42418f8205167bb29755 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.096s user 0.004s sys 0.016s
I20240502 01:53:59.319033 22403 raft_consensus.cc:348] T b6d89235867f42418f8205167bb29755 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:59.319171 22403 raft_consensus.cc:374] T b6d89235867f42418f8205167bb29755 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:59.319227 22403 raft_consensus.cc:729] T b6d89235867f42418f8205167bb29755 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:53:59.319420 22403 consensus_queue.cc:260] T b6d89235867f42418f8205167bb29755 P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:59.319926 22403 ts_tablet_manager.cc:1398] T b6d89235867f42418f8205167bb29755 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:53:59.320525 22408 tablet_bootstrap.cc:492] T de174ba4f7d848718ab8dbeb74369055 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:53:59.322861 22217 raft_consensus.cc:482] T 2259f02e0919491e8077ef5daefe3b02 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:59.323050 22217 raft_consensus.cc:504] T 2259f02e0919491e8077ef5daefe3b02 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:59.323797 22217 leader_election.cc:288] T 2259f02e0919491e8077ef5daefe3b02 P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:53:59.326609 20628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2259f02e0919491e8077ef5daefe3b02" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091" is_pre_election: true
I20240502 01:53:59.326913 20628 raft_consensus.cc:2454] T 2259f02e0919491e8077ef5daefe3b02 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 0.
I20240502 01:53:59.328536 20497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2259f02e0919491e8077ef5daefe3b02" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
I20240502 01:53:59.328779 20497 raft_consensus.cc:2454] T 2259f02e0919491e8077ef5daefe3b02 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 0.
I20240502 01:53:59.328903 20167 leader_election.cc:302] T 2259f02e0919491e8077ef5daefe3b02 P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f19cc6cb1e76460aa6711e49a5775091, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:53:59.329209 22217 raft_consensus.cc:2781] T 2259f02e0919491e8077ef5daefe3b02 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:59.329301 22217 raft_consensus.cc:482] T 2259f02e0919491e8077ef5daefe3b02 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:59.329396 22217 raft_consensus.cc:3037] T 2259f02e0919491e8077ef5daefe3b02 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:59.331147 17473 raft_consensus.cc:1203] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:53:59.331935 22217 raft_consensus.cc:504] T 2259f02e0919491e8077ef5daefe3b02 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:59.332489 22217 leader_election.cc:288] T 2259f02e0919491e8077ef5daefe3b02 P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 election: Requested vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:53:59.332898 20628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2259f02e0919491e8077ef5daefe3b02" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091"
I20240502 01:53:59.333148 20628 raft_consensus.cc:3037] T 2259f02e0919491e8077ef5daefe3b02 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:59.335887 20628 raft_consensus.cc:2454] T 2259f02e0919491e8077ef5daefe3b02 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 1.
I20240502 01:53:59.336508 20167 leader_election.cc:302] T 2259f02e0919491e8077ef5daefe3b02 P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f19cc6cb1e76460aa6711e49a5775091, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:53:59.337143 20497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2259f02e0919491e8077ef5daefe3b02" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
I20240502 01:53:59.337376 20497 raft_consensus.cc:3037] T 2259f02e0919491e8077ef5daefe3b02 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:59.340929 20497 raft_consensus.cc:2454] T 2259f02e0919491e8077ef5daefe3b02 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 1.
I20240502 01:53:59.342686 22177 raft_consensus.cc:482] T 2152c07e448b4582abff22b034363c70 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:59.342846 22177 raft_consensus.cc:504] T 2152c07e448b4582abff22b034363c70 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:59.343165 22217 raft_consensus.cc:2781] T 2259f02e0919491e8077ef5daefe3b02 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:59.343497 22177 leader_election.cc:288] T 2152c07e448b4582abff22b034363c70 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:53:59.344154 22217 raft_consensus.cc:686] T 2259f02e0919491e8077ef5daefe3b02 P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Becoming Leader. State: Replica: fdc2ac372492454c85387589a91f63af, State: Running, Role: LEADER
I20240502 01:53:59.344455 22217 consensus_queue.cc:237] T 2259f02e0919491e8077ef5daefe3b02 P fdc2ac372492454c85387589a91f63af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:53:59.344974 20365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2152c07e448b4582abff22b034363c70" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:53:59.345201 20365 raft_consensus.cc:2454] T 2152c07e448b4582abff22b034363c70 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 0.
I20240502 01:53:59.345867 17802 leader_election.cc:302] T 2152c07e448b4582abff22b034363c70 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:53:59.346192 21667 raft_consensus.cc:2781] T 2152c07e448b4582abff22b034363c70 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:59.346298 21667 raft_consensus.cc:482] T 2152c07e448b4582abff22b034363c70 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:59.346381 21667 raft_consensus.cc:3037] T 2152c07e448b4582abff22b034363c70 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:59.348923 21667 raft_consensus.cc:504] T 2152c07e448b4582abff22b034363c70 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:59.349524 21667 leader_election.cc:288] T 2152c07e448b4582abff22b034363c70 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 election: Requested vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:53:59.350412 20628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2152c07e448b4582abff22b034363c70" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091" is_pre_election: true
I20240502 01:53:59.350630 20628 raft_consensus.cc:2454] T 2152c07e448b4582abff22b034363c70 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 0.
I20240502 01:53:59.350600 20365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2152c07e448b4582abff22b034363c70" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
I20240502 01:53:59.350600 20629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2152c07e448b4582abff22b034363c70" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091"
I20240502 01:53:59.350780 20365 raft_consensus.cc:3037] T 2152c07e448b4582abff22b034363c70 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:59.353291 17330 catalog_manager.cc:5484] T 2259f02e0919491e8077ef5daefe3b02 P fdc2ac372492454c85387589a91f63af reported cstate change: term changed from 0 to 1, leader changed from <none> to fdc2ac372492454c85387589a91f63af (127.11.42.69). New cstate: current_term: 1 leader_uuid: "fdc2ac372492454c85387589a91f63af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:59.353581 20365 raft_consensus.cc:2454] T 2152c07e448b4582abff22b034363c70 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 1.
I20240502 01:53:59.353929 17802 leader_election.cc:302] T 2152c07e448b4582abff22b034363c70 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, bb31bceba76d466db24b9341c04900c3; no voters: f19cc6cb1e76460aa6711e49a5775091
I20240502 01:53:59.355000 22177 raft_consensus.cc:2781] T 2152c07e448b4582abff22b034363c70 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:59.355152 22177 raft_consensus.cc:686] T 2152c07e448b4582abff22b034363c70 P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Becoming Leader. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Running, Role: LEADER
I20240502 01:53:59.355401 22177 consensus_queue.cc:237] T 2152c07e448b4582abff22b034363c70 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:59.357050 20497 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "648b1cc4d8b54c5eb0948ca8c1984a1f"
dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33190
I20240502 01:53:59.357302 20497 raft_consensus.cc:595] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:53:59.360365 17330 catalog_manager.cc:5484] T 2152c07e448b4582abff22b034363c70 P bb31bceba76d466db24b9341c04900c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb31bceba76d466db24b9341c04900c3 (127.11.42.68). New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:59.385586 22217 raft_consensus.cc:482] T b6d89235867f42418f8205167bb29755 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:59.385746 22217 raft_consensus.cc:504] T b6d89235867f42418f8205167bb29755 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:59.386535 22217 leader_election.cc:288] T b6d89235867f42418f8205167bb29755 P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:53:59.388026 20365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d89235867f42418f8205167bb29755" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:53:59.388267 20365 raft_consensus.cc:2454] T b6d89235867f42418f8205167bb29755 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 0.
I20240502 01:53:59.388603 17606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d89235867f42418f8205167bb29755" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:53:59.388821 17606 raft_consensus.cc:2454] T b6d89235867f42418f8205167bb29755 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 0.
I20240502 01:53:59.389950 20165 leader_election.cc:302] T b6d89235867f42418f8205167bb29755 P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:53:59.390340 22217 raft_consensus.cc:2781] T b6d89235867f42418f8205167bb29755 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:59.390451 22217 raft_consensus.cc:482] T b6d89235867f42418f8205167bb29755 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:59.390527 22217 raft_consensus.cc:3037] T b6d89235867f42418f8205167bb29755 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:59.393543 22217 raft_consensus.cc:504] T b6d89235867f42418f8205167bb29755 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:59.394629 17606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d89235867f42418f8205167bb29755" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:53:59.394829 17606 raft_consensus.cc:3037] T b6d89235867f42418f8205167bb29755 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:59.396880 21674 raft_consensus.cc:482] T b950e18825ac434ba44586d544667918 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:59.397051 21674 raft_consensus.cc:504] T b950e18825ac434ba44586d544667918 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:59.397372 17606 raft_consensus.cc:2454] T b6d89235867f42418f8205167bb29755 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 1.
I20240502 01:53:59.397730 21674 leader_election.cc:288] T b950e18825ac434ba44586d544667918 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:53:59.398154 20497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b950e18825ac434ba44586d544667918" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
I20240502 01:53:59.398321 20365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b950e18825ac434ba44586d544667918" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:53:59.398368 20497 raft_consensus.cc:2454] T b950e18825ac434ba44586d544667918 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:53:59.398535 20365 raft_consensus.cc:2454] T b950e18825ac434ba44586d544667918 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:53:59.398742 17673 leader_election.cc:302] T b950e18825ac434ba44586d544667918 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:59.399205 21674 raft_consensus.cc:2781] T b950e18825ac434ba44586d544667918 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:59.399307 21674 raft_consensus.cc:482] T b950e18825ac434ba44586d544667918 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:59.399420 21674 raft_consensus.cc:3037] T b950e18825ac434ba44586d544667918 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:59.400421 20165 leader_election.cc:302] T b6d89235867f42418f8205167bb29755 P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:53:59.400733 22429 raft_consensus.cc:2781] T b6d89235867f42418f8205167bb29755 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:59.402091 21674 raft_consensus.cc:504] T b950e18825ac434ba44586d544667918 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:59.403015 20497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b950e18825ac434ba44586d544667918" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
I20240502 01:53:59.403191 20497 raft_consensus.cc:3037] T b950e18825ac434ba44586d544667918 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:59.403438 20365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d89235867f42418f8205167bb29755" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
I20240502 01:53:59.403647 20365 raft_consensus.cc:3037] T b6d89235867f42418f8205167bb29755 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:59.406021 20497 raft_consensus.cc:2454] T b950e18825ac434ba44586d544667918 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:53:59.406343 20365 raft_consensus.cc:2454] T b6d89235867f42418f8205167bb29755 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 1.
I20240502 01:53:59.406481 17673 leader_election.cc:302] T b950e18825ac434ba44586d544667918 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:59.406749 22037 raft_consensus.cc:2781] T b950e18825ac434ba44586d544667918 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:59.406908 22037 raft_consensus.cc:686] T b950e18825ac434ba44586d544667918 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Becoming Leader. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Running, Role: LEADER
I20240502 01:53:59.407192 22037 consensus_queue.cc:237] T b950e18825ac434ba44586d544667918 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:53:59.410120 22217 leader_election.cc:288] T b6d89235867f42418f8205167bb29755 P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 election: Requested vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:53:59.410754 20365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b950e18825ac434ba44586d544667918" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
I20240502 01:53:59.410979 20365 raft_consensus.cc:3037] T b950e18825ac434ba44586d544667918 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:59.411892 21674 leader_election.cc:288] T b950e18825ac434ba44586d544667918 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Requested vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:53:59.417253 20365 raft_consensus.cc:2454] T b950e18825ac434ba44586d544667918 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:53:59.421701 17330 catalog_manager.cc:5484] T b950e18825ac434ba44586d544667918 P dfa1152987644bd781b46c7e42a7a685 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfa1152987644bd781b46c7e42a7a685 (127.11.42.67). New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } }
I20240502 01:53:59.422607 22429 raft_consensus.cc:686] T b6d89235867f42418f8205167bb29755 P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Becoming Leader. State: Replica: fdc2ac372492454c85387589a91f63af, State: Running, Role: LEADER
I20240502 01:53:59.422961 22429 consensus_queue.cc:237] T b6d89235867f42418f8205167bb29755 P fdc2ac372492454c85387589a91f63af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:53:59.426601 22408 tablet_bootstrap.cc:492] T de174ba4f7d848718ab8dbeb74369055 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:59.427052 22408 tablet_bootstrap.cc:492] T de174ba4f7d848718ab8dbeb74369055 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap complete.
I20240502 01:53:59.427364 22408 ts_tablet_manager.cc:1367] T de174ba4f7d848718ab8dbeb74369055 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.107s user 0.037s sys 0.009s
I20240502 01:53:59.428241 22408 raft_consensus.cc:348] T de174ba4f7d848718ab8dbeb74369055 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } }
I20240502 01:53:59.428587 22408 raft_consensus.cc:729] T de174ba4f7d848718ab8dbeb74369055 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: LEARNER
I20240502 01:53:59.428869 22408 consensus_queue.cc:260] T de174ba4f7d848718ab8dbeb74369055 P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } }
I20240502 01:53:59.430409 22408 ts_tablet_manager.cc:1398] T de174ba4f7d848718ab8dbeb74369055 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.003s user 0.002s sys 0.002s
I20240502 01:53:59.431111 17887 tablet_copy_service.cc:342] P bb31bceba76d466db24b9341c04900c3: Request end of tablet copy session 6a36c952c51542c0bf7ea3ec2857a9f9-de174ba4f7d848718ab8dbeb74369055 received from {username='slave'} at 127.11.42.70:41203
I20240502 01:53:59.431233 17887 tablet_copy_service.cc:433] P bb31bceba76d466db24b9341c04900c3: ending tablet copy session 6a36c952c51542c0bf7ea3ec2857a9f9-de174ba4f7d848718ab8dbeb74369055 on tablet de174ba4f7d848718ab8dbeb74369055 with peer 6a36c952c51542c0bf7ea3ec2857a9f9
I20240502 01:53:59.433234 17330 catalog_manager.cc:5484] T b6d89235867f42418f8205167bb29755 P fdc2ac372492454c85387589a91f63af reported cstate change: term changed from 0 to 1, leader changed from <none> to fdc2ac372492454c85387589a91f63af (127.11.42.69). New cstate: current_term: 1 leader_uuid: "fdc2ac372492454c85387589a91f63af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:59.447109 17473 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0618761d30f34cfe9650d13c9275cdcb"
dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54200
I20240502 01:53:59.447345 17473 raft_consensus.cc:595] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Received request to transfer leadership
I20240502 01:53:59.459057 17333 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:53:59.460459 22200 raft_consensus.cc:1050] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091: attempting to promote NON_VOTER dfa1152987644bd781b46c7e42a7a685 to VOTER
I20240502 01:53:59.460646 17606 consensus_queue.cc:237] T 5152c002eadd4d54ab10a093bce45f17 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } }
I20240502 01:53:59.461227 22200 consensus_queue.cc:237] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:53:59.466136 20628 raft_consensus.cc:1261] T 2259f02e0919491e8077ef5daefe3b02 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:59.466696 22217 consensus_queue.cc:1035] T 2259f02e0919491e8077ef5daefe3b02 P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:59.467128 20497 raft_consensus.cc:1261] T 2259f02e0919491e8077ef5daefe3b02 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:59.467780 22217 consensus_queue.cc:1035] T 2259f02e0919491e8077ef5daefe3b02 P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:59.473039 22081 raft_consensus.cc:1050] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9: attempting to promote NON_VOTER f19cc6cb1e76460aa6711e49a5775091 to VOTER
I20240502 01:53:59.473788 22081 consensus_queue.cc:237] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } }
W20240502 01:53:59.474330 17543 consensus_peers.cc:485] T 5152c002eadd4d54ab10a093bce45f17 P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5152c002eadd4d54ab10a093bce45f17. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:59.475332 17864 raft_consensus.cc:1261] T 5152c002eadd4d54ab10a093bce45f17 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:59.475347 17473 raft_consensus.cc:1261] T 5152c002eadd4d54ab10a093bce45f17 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:59.475941 22202 consensus_queue.cc:1035] T 5152c002eadd4d54ab10a093bce45f17 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:59.477243 22202 consensus_queue.cc:1035] T 5152c002eadd4d54ab10a093bce45f17 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:59.477509 17330 catalog_manager.cc:5576] TS 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623) does not have the latest schema for tablet 2259f02e0919491e8077ef5daefe3b02 (table rebalancer_dynamic_table [id=4654334e0d494a9c8455f530bb7c550b]). Expected version 1 got 0
I20240502 01:53:59.479900 20629 raft_consensus.cc:1261] T a212c17ba64842c99b3d6a78a3b7cc8a P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:53:59.480146 17474 raft_consensus.cc:1261] T a212c17ba64842c99b3d6a78a3b7cc8a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:53:59.480540 22156 consensus_queue.cc:1035] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:53:59.480631 20233 raft_consensus.cc:1261] T a212c17ba64842c99b3d6a78a3b7cc8a P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:53:59.481395 22156 consensus_queue.cc:1035] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:53:59.483609 22156 consensus_queue.cc:1035] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:53:59.502897 22463 tablet.cc:1722] T 2259f02e0919491e8077ef5daefe3b02 P fdc2ac372492454c85387589a91f63af: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:53:59.503492 22463 tablet.cc:1620] T 2259f02e0919491e8077ef5daefe3b02 P fdc2ac372492454c85387589a91f63af: MemRowSet was empty: no flush needed.
I20240502 01:53:59.504805 22464 tablet.cc:1722] T 2259f02e0919491e8077ef5daefe3b02 P 2f306549407b4bb281ccf2994035e4f1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:53:59.505261 22464 tablet.cc:1620] T 2259f02e0919491e8077ef5daefe3b02 P 2f306549407b4bb281ccf2994035e4f1: MemRowSet was empty: no flush needed.
I20240502 01:53:59.505242 17330 catalog_manager.cc:5576] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339) does not have the latest schema for tablet 2259f02e0919491e8077ef5daefe3b02 (table rebalancer_dynamic_table [id=4654334e0d494a9c8455f530bb7c550b]). Expected version 1 got 0
I20240502 01:53:59.507377 17330 catalog_manager.cc:5576] TS 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623) does not have the latest schema for tablet 2259f02e0919491e8077ef5daefe3b02 (table rebalancer_dynamic_table [id=4654334e0d494a9c8455f530bb7c550b]). Expected version 1 got 0
I20240502 01:53:59.509502 22202 raft_consensus.cc:2932] T 5152c002eadd4d54ab10a093bce45f17 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:53:59.510944 20497 raft_consensus.cc:1261] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:53:59.511533 22200 consensus_queue.cc:1035] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:53:59.513895 20365 raft_consensus.cc:1261] T cf0237a4c65b495ba892871dd850bed9 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:53:59.514164 17734 raft_consensus.cc:1261] T cf0237a4c65b495ba892871dd850bed9 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:53:59.514299 17865 raft_consensus.cc:2932] T 5152c002eadd4d54ab10a093bce45f17 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:53:59.517654 21934 consensus_queue.cc:1035] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:53:59.518126 17473 raft_consensus.cc:2932] T 5152c002eadd4d54ab10a093bce45f17 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:53:59.519289 21934 consensus_queue.cc:1035] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:53:59.519932 17330 catalog_manager.cc:5484] T 5152c002eadd4d54ab10a093bce45f17 P bb31bceba76d466db24b9341c04900c3 reported cstate change: config changed from index -1 to 4, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:53:59.525269 17333 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=4654334e0d494a9c8455f530bb7c550b] alter complete (version 1)
I20240502 01:53:59.526062 20497 raft_consensus.cc:1203] T 7e9be84ef6a043818f166f6b47898855 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:53:59.545981 22081 raft_consensus.cc:2932] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:59.547173 21934 raft_consensus.cc:2932] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:53:59.551946 20233 raft_consensus.cc:2932] T a212c17ba64842c99b3d6a78a3b7cc8a P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:59.552065 17735 raft_consensus.cc:2932] T cf0237a4c65b495ba892871dd850bed9 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:53:59.552850 17473 raft_consensus.cc:2932] T a212c17ba64842c99b3d6a78a3b7cc8a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:59.559161 17334 catalog_manager.cc:5484] T cf0237a4c65b495ba892871dd850bed9 P dfa1152987644bd781b46c7e42a7a685 reported cstate change: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:53:59.563190 17333 catalog_manager.cc:5576] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411) does not have the latest schema for tablet 2259f02e0919491e8077ef5daefe3b02 (table rebalancer_dynamic_table [id=4654334e0d494a9c8455f530bb7c550b]). Expected version 1 got 0
I20240502 01:53:59.563555 20365 raft_consensus.cc:2932] T cf0237a4c65b495ba892871dd850bed9 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:53:59.559954 17327 catalog_manager.cc:5484] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: config changed from index 2 to 3, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:59.570801 20628 raft_consensus.cc:2932] T a212c17ba64842c99b3d6a78a3b7cc8a P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:59.575771 22392 raft_consensus.cc:979] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1: : Instructing follower 4da690c0fa13407a85c0689ce8de9ade to start an election
I20240502 01:53:59.576037 22392 raft_consensus.cc:1067] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Signalling peer 4da690c0fa13407a85c0689ce8de9ade to start an election
I20240502 01:53:59.577700 17474 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "648b1cc4d8b54c5eb0948ca8c1984a1f"
dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
from {username='slave'} at 127.11.42.71:44055
I20240502 01:53:59.578285 17474 raft_consensus.cc:482] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:53:59.579466 17474 raft_consensus.cc:3037] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:59.580108 20365 consensus_queue.cc:237] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } }
I20240502 01:53:59.584060 17475 raft_consensus.cc:1261] T a212c17ba64842c99b3d6a78a3b7cc8a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:59.584074 20628 raft_consensus.cc:1261] T a212c17ba64842c99b3d6a78a3b7cc8a P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:59.584998 17474 raft_consensus.cc:504] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 4da690c0fa13407a85c0689ce8de9ade [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:53:59.585414 22465 consensus_queue.cc:1035] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:59.585930 22465 consensus_queue.cc:1035] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:59.588447 22081 raft_consensus.cc:2932] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:59.593786 20630 raft_consensus.cc:2932] T a212c17ba64842c99b3d6a78a3b7cc8a P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:59.597935 20498 raft_consensus.cc:2932] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:53:59.598701 20365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "648b1cc4d8b54c5eb0948ca8c1984a1f" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
I20240502 01:53:59.599617 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a212c17ba64842c99b3d6a78a3b7cc8a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:59.600883 20365 raft_consensus.cc:3037] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:59.602381 17334 catalog_manager.cc:5484] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: config changed from index 3 to 4, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:59.604315 20365 raft_consensus.cc:2454] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 2.
I20240502 01:53:59.605741 20497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "648b1cc4d8b54c5eb0948ca8c1984a1f" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
I20240502 01:53:59.605813 17734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "648b1cc4d8b54c5eb0948ca8c1984a1f" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:53:59.605957 20497 raft_consensus.cc:3032] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:53:59.606019 17734 raft_consensus.cc:3037] T 648b1cc4d8b54c5eb0948ca8c1984a1f P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:59.606061 20497 raft_consensus.cc:729] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Running, Role: LEADER
I20240502 01:53:59.606350 20497 consensus_queue.cc:260] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:53:59.606736 20497 raft_consensus.cc:3037] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:59.609602 20211 tablet_service.cc:1514] Processing DeleteTablet for tablet a212c17ba64842c99b3d6a78a3b7cc8a with delete_type TABLET_DATA_TOMBSTONED (TS fdc2ac372492454c85387589a91f63af not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44624
I20240502 01:53:59.612655 17734 raft_consensus.cc:2454] T 648b1cc4d8b54c5eb0948ca8c1984a1f P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 2.
I20240502 01:53:59.613579 20497 raft_consensus.cc:2454] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 2.
I20240502 01:53:59.614176 17412 leader_election.cc:302] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 4da690c0fa13407a85c0689ce8de9ade, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:53:59.624899 17474 leader_election.cc:288] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 election: Requested vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:53:59.626442 20366 raft_consensus.cc:1226] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Rejecting Update request from peer 2f306549407b4bb281ccf2994035e4f1 for earlier term 1. Current term is 2. Ops: []
I20240502 01:53:59.625403 21920 raft_consensus.cc:2781] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 4da690c0fa13407a85c0689ce8de9ade [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:53:59.629360 21920 raft_consensus.cc:686] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 4da690c0fa13407a85c0689ce8de9ade [term 2 LEADER]: Becoming Leader. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Running, Role: LEADER
I20240502 01:53:59.629670 21920 consensus_queue.cc:237] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:53:59.629774 17327 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:53:59.625790 22480 raft_consensus.cc:979] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade: : Instructing follower bb31bceba76d466db24b9341c04900c3 to start an election
I20240502 01:53:59.632036 22481 tablet_replica.cc:330] T a212c17ba64842c99b3d6a78a3b7cc8a P fdc2ac372492454c85387589a91f63af: stopping tablet replica
I20240502 01:53:59.632464 22481 raft_consensus.cc:2229] T a212c17ba64842c99b3d6a78a3b7cc8a P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:59.632908 22481 raft_consensus.cc:2258] T a212c17ba64842c99b3d6a78a3b7cc8a P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:59.636963 22482 raft_consensus.cc:1067] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Signalling peer bb31bceba76d466db24b9341c04900c3 to start an election
I20240502 01:53:59.627303 17473 raft_consensus.cc:2932] T a212c17ba64842c99b3d6a78a3b7cc8a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:53:59.640256 22463 tablet.cc:1722] T 2259f02e0919491e8077ef5daefe3b02 P fdc2ac372492454c85387589a91f63af: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:53:59.640726 22463 tablet.cc:1620] T 2259f02e0919491e8077ef5daefe3b02 P fdc2ac372492454c85387589a91f63af: MemRowSet was empty: no flush needed.
I20240502 01:53:59.641053 22464 tablet.cc:1722] T 2259f02e0919491e8077ef5daefe3b02 P 2f306549407b4bb281ccf2994035e4f1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:53:59.641486 22464 tablet.cc:1620] T 2259f02e0919491e8077ef5daefe3b02 P 2f306549407b4bb281ccf2994035e4f1: MemRowSet was empty: no flush needed.
I20240502 01:53:59.642514 17864 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0618761d30f34cfe9650d13c9275cdcb"
dest_uuid: "bb31bceba76d466db24b9341c04900c3"
from {username='slave'} at 127.11.42.65:36787
I20240502 01:53:59.642794 17864 raft_consensus.cc:482] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:53:59.642925 17864 raft_consensus.cc:3037] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:59.647135 17864 raft_consensus.cc:504] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } }
I20240502 01:53:59.648198 17864 leader_election.cc:288] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 election: Requested vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:53:59.649581 20231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0618761d30f34cfe9650d13c9275cdcb" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:53:59.649811 20231 raft_consensus.cc:3037] T 0618761d30f34cfe9650d13c9275cdcb P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:59.651023 17327 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=4654334e0d494a9c8455f530bb7c550b] alter complete (version 2)
I20240502 01:53:59.651854 17734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0618761d30f34cfe9650d13c9275cdcb" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:53:59.652062 17734 raft_consensus.cc:3037] T 0618761d30f34cfe9650d13c9275cdcb P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:59.652428 20231 raft_consensus.cc:2454] T 0618761d30f34cfe9650d13c9275cdcb P fdc2ac372492454c85387589a91f63af [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 2.
I20240502 01:53:59.655133 17734 raft_consensus.cc:2454] T 0618761d30f34cfe9650d13c9275cdcb P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 2.
I20240502 01:53:59.656163 17474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0618761d30f34cfe9650d13c9275cdcb" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:53:59.656410 17474 raft_consensus.cc:3032] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:53:59.656514 17474 raft_consensus.cc:729] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Running, Role: LEADER
I20240502 01:53:59.656915 17474 consensus_queue.cc:260] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } }
I20240502 01:53:59.657446 17474 raft_consensus.cc:3037] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:59.659493 22481 ts_tablet_manager.cc:1875] T a212c17ba64842c99b3d6a78a3b7cc8a P fdc2ac372492454c85387589a91f63af: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:59.662329 22477 tablet.cc:1722] T 2259f02e0919491e8077ef5daefe3b02 P f19cc6cb1e76460aa6711e49a5775091: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:53:59.662554 17327 catalog_manager.cc:5484] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: term changed from 1 to 2, leader changed from 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) to 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65). New cstate: current_term: 2 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:59.662998 17474 raft_consensus.cc:2454] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 2.
I20240502 01:53:59.664726 22477 tablet.cc:1620] T 2259f02e0919491e8077ef5daefe3b02 P f19cc6cb1e76460aa6711e49a5775091: MemRowSet was empty: no flush needed.
I20240502 01:53:59.665556 17734 raft_consensus.cc:1261] T f0b3862e504d4b43a60477e5efef29b1 P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:53:59.673700 17334 catalog_manager.cc:5576] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411) does not have the latest schema for tablet 2259f02e0919491e8077ef5daefe3b02 (table rebalancer_dynamic_table [id=4654334e0d494a9c8455f530bb7c550b]). Expected version 2 got 0
I20240502 01:53:59.676880 17802 leader_election.cc:302] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, bb31bceba76d466db24b9341c04900c3, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:53:59.677426 21667 raft_consensus.cc:2781] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:53:59.677582 21667 raft_consensus.cc:686] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 [term 2 LEADER]: Becoming Leader. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Running, Role: LEADER
I20240502 01:53:59.677876 21667 consensus_queue.cc:237] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } }
I20240502 01:53:59.678191 22169 consensus_queue.cc:1035] T f0b3862e504d4b43a60477e5efef29b1 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:53:59.680363 22481 ts_tablet_manager.cc:1888] T a212c17ba64842c99b3d6a78a3b7cc8a P fdc2ac372492454c85387589a91f63af: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:53:59.680692 22481 log.cc:1196] T a212c17ba64842c99b3d6a78a3b7cc8a P fdc2ac372492454c85387589a91f63af: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal/wals/a212c17ba64842c99b3d6a78a3b7cc8a
I20240502 01:53:59.681686 17322 catalog_manager.cc:4830] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): tablet a212c17ba64842c99b3d6a78a3b7cc8a (table rebalancer_extra_table_4 [id=dca50b50b02e4a8c8b90e0de6f305bd8]) successfully deleted
I20240502 01:53:59.683046 22477 tablet.cc:1722] T 2259f02e0919491e8077ef5daefe3b02 P f19cc6cb1e76460aa6711e49a5775091: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:53:59.683588 22477 tablet.cc:1620] T 2259f02e0919491e8077ef5daefe3b02 P f19cc6cb1e76460aa6711e49a5775091: MemRowSet was empty: no flush needed.
I20240502 01:53:59.688056 20629 consensus_queue.cc:237] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:53:59.690186 17737 raft_consensus.cc:1261] T cf0237a4c65b495ba892871dd850bed9 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:59.690554 20497 raft_consensus.cc:1261] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:59.690783 22200 consensus_queue.cc:1035] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:59.691131 21934 consensus_queue.cc:1035] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:59.693670 22200 raft_consensus.cc:2932] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:53:59.696224 17865 consensus_queue.cc:237] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:53:59.698750 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cf0237a4c65b495ba892871dd850bed9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:59.699954 17737 raft_consensus.cc:2932] T cf0237a4c65b495ba892871dd850bed9 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:53:59.700670 20497 raft_consensus.cc:2932] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:53:59.701556 17334 catalog_manager.cc:5484] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: config changed from index 3 to 4, VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) evicted. New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:59.705746 20347 tablet_service.cc:1514] Processing DeleteTablet for tablet cf0237a4c65b495ba892871dd850bed9 with delete_type TABLET_DATA_TOMBSTONED (TS 6a36c952c51542c0bf7ea3ec2857a9f9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40196
I20240502 01:53:59.706387 22491 tablet_replica.cc:330] T cf0237a4c65b495ba892871dd850bed9 P 6a36c952c51542c0bf7ea3ec2857a9f9: stopping tablet replica
I20240502 01:53:59.706576 22491 raft_consensus.cc:2229] T cf0237a4c65b495ba892871dd850bed9 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:59.706794 22491 raft_consensus.cc:2258] T cf0237a4c65b495ba892871dd850bed9 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:59.709547 22491 ts_tablet_manager.cc:1875] T cf0237a4c65b495ba892871dd850bed9 P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:59.716154 22491 ts_tablet_manager.cc:1888] T cf0237a4c65b495ba892871dd850bed9 P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:53:59.716312 22491 log.cc:1196] T cf0237a4c65b495ba892871dd850bed9 P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal/wals/cf0237a4c65b495ba892871dd850bed9
I20240502 01:53:59.716136 17327 catalog_manager.cc:5484] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 reported cstate change: term changed from 1 to 2, leader changed from 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) to bb31bceba76d466db24b9341c04900c3 (127.11.42.68). New cstate: current_term: 2 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:59.717154 17321 catalog_manager.cc:4830] TS 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): tablet cf0237a4c65b495ba892871dd850bed9 (table rebalancer_extra_table_2 [id=2444c341261d4eb29c484622291d0517]) successfully deleted
I20240502 01:53:59.718868 17604 raft_consensus.cc:1261] T 7c697210a14d4982a939ea7965cb769d P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:59.719565 22169 consensus_queue.cc:1035] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:59.722362 17737 raft_consensus.cc:1261] T 7c697210a14d4982a939ea7965cb769d P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:59.722241 21667 raft_consensus.cc:2932] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:53:59.722829 17604 raft_consensus.cc:2932] T 7c697210a14d4982a939ea7965cb769d P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:53:59.723141 22177 consensus_queue.cc:1035] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:53:59.723944 17802 consensus_peers.cc:485] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7c697210a14d4982a939ea7965cb769d. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:59.729445 17327 catalog_manager.cc:5484] T 7c697210a14d4982a939ea7965cb769d P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: config changed from index -1 to 4, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:53:59.745488 17737 raft_consensus.cc:2932] T 7c697210a14d4982a939ea7965cb769d P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:53:59.747709 17473 consensus_queue.cc:237] T b39b88755cd74b0294d264215aea99cf P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
I20240502 01:53:59.754271 17604 raft_consensus.cc:1261] T b39b88755cd74b0294d264215aea99cf P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:59.754925 21920 consensus_queue.cc:1035] T b39b88755cd74b0294d264215aea99cf P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:53:59.755275 17412 consensus_peers.cc:485] T b39b88755cd74b0294d264215aea99cf P 4da690c0fa13407a85c0689ce8de9ade -> Peer 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): Couldn't send request to peer 2f306549407b4bb281ccf2994035e4f1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b39b88755cd74b0294d264215aea99cf. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:59.755611 17737 raft_consensus.cc:1261] T b39b88755cd74b0294d264215aea99cf P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:59.756781 21920 consensus_queue.cc:1035] T b39b88755cd74b0294d264215aea99cf P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:59.757839 21920 raft_consensus.cc:2932] T b39b88755cd74b0294d264215aea99cf P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:53:59.758610 17604 raft_consensus.cc:2932] T b39b88755cd74b0294d264215aea99cf P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:53:59.760330 17327 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:53:59.769750 17327 catalog_manager.cc:5484] T b39b88755cd74b0294d264215aea99cf P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: config changed from index -1 to 4, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New cstate: current_term: 1 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:59.770064 17734 raft_consensus.cc:2932] T b39b88755cd74b0294d264215aea99cf P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:53:59.774070 22463 tablet.cc:1722] T 2259f02e0919491e8077ef5daefe3b02 P fdc2ac372492454c85387589a91f63af: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:53:59.774791 22477 tablet.cc:1722] T 2259f02e0919491e8077ef5daefe3b02 P f19cc6cb1e76460aa6711e49a5775091: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:53:59.781132 22464 tablet.cc:1722] T 2259f02e0919491e8077ef5daefe3b02 P 2f306549407b4bb281ccf2994035e4f1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:53:59.784999 17330 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table_new_name [id=4654334e0d494a9c8455f530bb7c550b] alter complete (version 3)
I20240502 01:53:59.800365 22177 consensus_queue.cc:1035] T 2152c07e448b4582abff22b034363c70 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:59.806808 20366 raft_consensus.cc:1203] T de174ba4f7d848718ab8dbeb74369055 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:53:59.816879 20629 raft_consensus.cc:3037] T 2152c07e448b4582abff22b034363c70 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:59.821451 21667 consensus_queue.cc:1035] T 2152c07e448b4582abff22b034363c70 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:59.824529 17865 consensus_queue.cc:237] T 59c288e7279f4119afb6c3bf860e6dc3 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } }
I20240502 01:53:59.830689 22037 consensus_queue.cc:1035] T b950e18825ac434ba44586d544667918 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:59.844745 17604 raft_consensus.cc:1261] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
W20240502 01:53:59.845245 17803 consensus_peers.cc:485] T 59c288e7279f4119afb6c3bf860e6dc3 P bb31bceba76d466db24b9341c04900c3 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 59c288e7279f4119afb6c3bf860e6dc3. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:59.845559 22177 consensus_queue.cc:1035] T f0b3862e504d4b43a60477e5efef29b1 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:53:59.846045 17734 raft_consensus.cc:1261] T 59c288e7279f4119afb6c3bf860e6dc3 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:59.847186 22177 consensus_queue.cc:1035] T 59c288e7279f4119afb6c3bf860e6dc3 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:59.847769 20231 raft_consensus.cc:1261] T f0b3862e504d4b43a60477e5efef29b1 P fdc2ac372492454c85387589a91f63af [term 2 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:53:59.848305 22177 consensus_queue.cc:1035] T f0b3862e504d4b43a60477e5efef29b1 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:53:59.849668 17606 raft_consensus.cc:1261] T 59c288e7279f4119afb6c3bf860e6dc3 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:59.850487 21667 consensus_queue.cc:1035] T 59c288e7279f4119afb6c3bf860e6dc3 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:59.871866 17865 consensus_queue.cc:237] T f0b3862e504d4b43a60477e5efef29b1 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } }
I20240502 01:53:59.873975 20231 raft_consensus.cc:1261] T f0b3862e504d4b43a60477e5efef29b1 P fdc2ac372492454c85387589a91f63af [term 2 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:53:59.874480 22169 consensus_queue.cc:1035] T f0b3862e504d4b43a60477e5efef29b1 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:53:59.876269 22177 raft_consensus.cc:2932] T 59c288e7279f4119afb6c3bf860e6dc3 P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:53:59.876860 21674 consensus_queue.cc:1035] T b950e18825ac434ba44586d544667918 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:59.877498 22169 raft_consensus.cc:2932] T f0b3862e504d4b43a60477e5efef29b1 P bb31bceba76d466db24b9341c04900c3 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:53:59.877910 20231 raft_consensus.cc:2932] T f0b3862e504d4b43a60477e5efef29b1 P fdc2ac372492454c85387589a91f63af [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:53:59.879868 17606 raft_consensus.cc:2932] T 59c288e7279f4119afb6c3bf860e6dc3 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:53:59.881388 17736 raft_consensus.cc:1261] T f0b3862e504d4b43a60477e5efef29b1 P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:53:59.886278 22177 consensus_queue.cc:1035] T f0b3862e504d4b43a60477e5efef29b1 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:53:59.886616 17330 catalog_manager.cc:5484] T f0b3862e504d4b43a60477e5efef29b1 P fdc2ac372492454c85387589a91f63af reported cstate change: config changed from index 5 to 7, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New cstate: current_term: 2 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:53:59.889753 17734 raft_consensus.cc:2932] T 59c288e7279f4119afb6c3bf860e6dc3 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:53:59.890192 17327 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:53:59.890410 17327 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:53:59.893644 17327 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet 2259f02e0919491e8077ef5daefe3b02
I20240502 01:53:59.894505 20477 tablet_service.cc:1514] Processing DeleteTablet for tablet 2259f02e0919491e8077ef5daefe3b02 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:59 UTC) from {username='slave'} at 127.0.0.1:35018
I20240502 01:53:59.894636 20609 tablet_service.cc:1514] Processing DeleteTablet for tablet 2259f02e0919491e8077ef5daefe3b02 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:59 UTC) from {username='slave'} at 127.0.0.1:33698
I20240502 01:53:59.894840 17737 raft_consensus.cc:2932] T f0b3862e504d4b43a60477e5efef29b1 P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:53:59.895191 20211 tablet_service.cc:1514] Processing DeleteTablet for tablet 2259f02e0919491e8077ef5daefe3b02 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:53:59 UTC) from {username='slave'} at 127.0.0.1:44624
I20240502 01:53:59.895480 22481 tablet_replica.cc:330] T 2259f02e0919491e8077ef5daefe3b02 P fdc2ac372492454c85387589a91f63af: stopping tablet replica
I20240502 01:53:59.895694 22481 raft_consensus.cc:2229] T 2259f02e0919491e8077ef5daefe3b02 P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:53:59.896085 22481 raft_consensus.cc:2258] T 2259f02e0919491e8077ef5daefe3b02 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:59.896975 22481 ts_tablet_manager.cc:1875] T 2259f02e0919491e8077ef5daefe3b02 P fdc2ac372492454c85387589a91f63af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:59.909830 17327 catalog_manager.cc:5484] T 59c288e7279f4119afb6c3bf860e6dc3 P dfa1152987644bd781b46c7e42a7a685 reported cstate change: config changed from index -1 to 4, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:53:59.912725 22523 ts_tablet_manager.cc:902] T 5152c002eadd4d54ab10a093bce45f17 P fdc2ac372492454c85387589a91f63af: Initiating tablet copy from peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:53:59.914590 17587 tablet_service.cc:1514] Processing DeleteTablet for tablet f0b3862e504d4b43a60477e5efef29b1 with delete_type TABLET_DATA_TOMBSTONED (TS 3fceccefba274b1dbbd8216af8b0bc29 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:55232
I20240502 01:53:59.915025 22525 tablet_replica.cc:330] T 2259f02e0919491e8077ef5daefe3b02 P 2f306549407b4bb281ccf2994035e4f1: stopping tablet replica
I20240502 01:53:59.915791 22525 raft_consensus.cc:2229] T 2259f02e0919491e8077ef5daefe3b02 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:59.917944 22481 ts_tablet_manager.cc:1888] T 2259f02e0919491e8077ef5daefe3b02 P fdc2ac372492454c85387589a91f63af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:53:59.918085 22481 log.cc:1196] T 2259f02e0919491e8077ef5daefe3b02 P fdc2ac372492454c85387589a91f63af: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal/wals/2259f02e0919491e8077ef5daefe3b02
I20240502 01:53:59.918246 22525 raft_consensus.cc:2258] T 2259f02e0919491e8077ef5daefe3b02 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:59.918563 22481 ts_tablet_manager.cc:1909] T 2259f02e0919491e8077ef5daefe3b02 P fdc2ac372492454c85387589a91f63af: Deleting consensus metadata
I20240502 01:53:59.919330 22525 ts_tablet_manager.cc:1875] T 2259f02e0919491e8077ef5daefe3b02 P 2f306549407b4bb281ccf2994035e4f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:59.919739 17322 catalog_manager.cc:4830] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): tablet 2259f02e0919491e8077ef5daefe3b02 (table rebalancer_dynamic_table_new_name [id=4654334e0d494a9c8455f530bb7c550b]) successfully deleted
I20240502 01:53:59.920305 22275 raft_consensus.cc:1050] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 6a36c952c51542c0bf7ea3ec2857a9f9: attempting to promote NON_VOTER 4da690c0fa13407a85c0689ce8de9ade to VOTER
I20240502 01:53:59.921211 22275 consensus_queue.cc:237] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } }
I20240502 01:53:59.921788 22526 tablet_replica.cc:330] T 2259f02e0919491e8077ef5daefe3b02 P f19cc6cb1e76460aa6711e49a5775091: stopping tablet replica
I20240502 01:53:59.922657 22526 raft_consensus.cc:2229] T 2259f02e0919491e8077ef5daefe3b02 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:59.922938 22526 raft_consensus.cc:2258] T 2259f02e0919491e8077ef5daefe3b02 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:59.925734 22526 ts_tablet_manager.cc:1875] T 2259f02e0919491e8077ef5daefe3b02 P f19cc6cb1e76460aa6711e49a5775091: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:53:59.930224 17473 raft_consensus.cc:1261] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEARNER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20240502 01:53:59.930523 17803 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.42.69:34339, user_credentials={real_user=slave}} blocked reactor thread for 47332.7us
I20240502 01:53:59.930740 22529 tablet_replica.cc:330] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29: stopping tablet replica
I20240502 01:53:59.930967 22529 raft_consensus.cc:2229] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:59.931164 22529 pending_rounds.cc:62] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29: Trying to abort 1 pending ops.
I20240502 01:53:59.932426 22465 consensus_queue.cc:1035] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:53:59.932865 22529 pending_rounds.cc:68] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29: Aborting op as it isn't in flight: id { term: 2 index: 6 } timestamp: 7023062381648699392 op_type: NO_OP noop_request { }
I20240502 01:53:59.933117 22529 raft_consensus.cc:2866] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240502 01:53:59.933269 22529 raft_consensus.cc:2258] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:59.934054 17322 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f0b3862e504d4b43a60477e5efef29b1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:59.936713 22529 ts_tablet_manager.cc:1875] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:59.939708 20629 raft_consensus.cc:1261] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:53:59.939999 20497 raft_consensus.cc:1261] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:53:59.940318 22081 consensus_queue.cc:1035] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:53:59.940584 22465 consensus_queue.cc:1035] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:53:59.946864 17334 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_extra_table_24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:53:59.951530 22525 ts_tablet_manager.cc:1888] T 2259f02e0919491e8077ef5daefe3b02 P 2f306549407b4bb281ccf2994035e4f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:53:59.952337 22525 log.cc:1196] T 2259f02e0919491e8077ef5daefe3b02 P 2f306549407b4bb281ccf2994035e4f1: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal/wals/2259f02e0919491e8077ef5daefe3b02
I20240502 01:53:59.953576 22525 ts_tablet_manager.cc:1909] T 2259f02e0919491e8077ef5daefe3b02 P 2f306549407b4bb281ccf2994035e4f1: Deleting consensus metadata
I20240502 01:53:59.955507 17322 catalog_manager.cc:4830] TS 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): tablet 2259f02e0919491e8077ef5daefe3b02 (table rebalancer_dynamic_table_new_name [id=4654334e0d494a9c8455f530bb7c550b]) successfully deleted
I20240502 01:53:59.956115 22529 ts_tablet_manager.cc:1888] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 01:53:59.957125 22529 log.cc:1196] T f0b3862e504d4b43a60477e5efef29b1 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/wals/f0b3862e504d4b43a60477e5efef29b1
I20240502 01:53:59.958299 22526 ts_tablet_manager.cc:1888] T 2259f02e0919491e8077ef5daefe3b02 P f19cc6cb1e76460aa6711e49a5775091: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:53:59.958431 17319 catalog_manager.cc:4830] TS 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): tablet f0b3862e504d4b43a60477e5efef29b1 (table rebalance_test_table_3 [id=b996bd60793c4d3c83bf34f4989da909]) successfully deleted
I20240502 01:53:59.958681 22526 log.cc:1196] T 2259f02e0919491e8077ef5daefe3b02 P f19cc6cb1e76460aa6711e49a5775091: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal/wals/2259f02e0919491e8077ef5daefe3b02
I20240502 01:53:59.960075 22081 raft_consensus.cc:2932] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:53:59.960844 20347 tablet_service.cc:1467] Processing CreateTablet for tablet f3e4c8f98b5e48e3ba6affc2df08943a (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=6077652d45454150b5c7d7072780d895]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:59.961328 20347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3e4c8f98b5e48e3ba6affc2df08943a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:59.961838 22526 ts_tablet_manager.cc:1909] T 2259f02e0919491e8077ef5daefe3b02 P f19cc6cb1e76460aa6711e49a5775091: Deleting consensus metadata
I20240502 01:53:59.962424 20630 raft_consensus.cc:2932] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:53:59.967165 17474 raft_consensus.cc:2932] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:53:59.969282 17321 catalog_manager.cc:4830] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): tablet 2259f02e0919491e8077ef5daefe3b02 (table rebalancer_dynamic_table_new_name [id=4654334e0d494a9c8455f530bb7c550b]) successfully deleted
I20240502 01:53:59.971338 20211 tablet_service.cc:1467] Processing CreateTablet for tablet 062b9d90fb0444f4be37bf09770c9caf (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=6077652d45454150b5c7d7072780d895]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:59.971346 17846 tablet_service.cc:1467] Processing CreateTablet for tablet 062b9d90fb0444f4be37bf09770c9caf (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=6077652d45454150b5c7d7072780d895]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:59.971781 17846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 062b9d90fb0444f4be37bf09770c9caf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:59.971750 20211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 062b9d90fb0444f4be37bf09770c9caf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:59.973699 17327 catalog_manager.cc:5484] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:53:59.976742 20477 tablet_service.cc:1467] Processing CreateTablet for tablet 062b9d90fb0444f4be37bf09770c9caf (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=6077652d45454150b5c7d7072780d895]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:59.977383 20212 tablet_service.cc:1467] Processing CreateTablet for tablet f3e4c8f98b5e48e3ba6affc2df08943a (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=6077652d45454150b5c7d7072780d895]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:59.977782 20212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3e4c8f98b5e48e3ba6affc2df08943a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:59.978488 20497 raft_consensus.cc:2932] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:53:59.983963 20477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 062b9d90fb0444f4be37bf09770c9caf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:59.984424 20345 tablet_service.cc:1467] Processing CreateTablet for tablet 61ab80ce377a44ac95a9c574e9a615f2 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=6077652d45454150b5c7d7072780d895]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:59.984812 20345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61ab80ce377a44ac95a9c574e9a615f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:59.989738 20476 tablet_service.cc:1467] Processing CreateTablet for tablet f3e4c8f98b5e48e3ba6affc2df08943a (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=6077652d45454150b5c7d7072780d895]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:59.990144 20476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3e4c8f98b5e48e3ba6affc2df08943a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:59.990517 22440 consensus_queue.cc:1035] T b6d89235867f42418f8205167bb29755 P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:59.993963 17845 tablet_service.cc:1467] Processing CreateTablet for tablet 61ab80ce377a44ac95a9c574e9a615f2 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=6077652d45454150b5c7d7072780d895]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:59.994371 17845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61ab80ce377a44ac95a9c574e9a615f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:00.000519 20213 tablet_service.cc:1467] Processing CreateTablet for tablet 61ab80ce377a44ac95a9c574e9a615f2 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=6077652d45454150b5c7d7072780d895]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:00.000944 20213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61ab80ce377a44ac95a9c574e9a615f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:00.002839 17327 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:54:00.018007 17734 raft_consensus.cc:1261] T 648b1cc4d8b54c5eb0948ca8c1984a1f P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:54:00.017757 17845 tablet_service.cc:1467] Processing CreateTablet for tablet 9d551360acc74a56863411fb656b6ee7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8af1254734c2412aa0ec7b7538b83711]), partition=
I20240502 01:54:00.018608 22487 consensus_queue.cc:1035] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:00.019109 17845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d551360acc74a56863411fb656b6ee7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:00.019520 22543 tablet_bootstrap.cc:492] T 062b9d90fb0444f4be37bf09770c9caf P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:54:00.022673 22543 tablet_bootstrap.cc:654] T 062b9d90fb0444f4be37bf09770c9caf P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:00.023507 20345 tablet_service.cc:1467] Processing CreateTablet for tablet 9d551360acc74a56863411fb656b6ee7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8af1254734c2412aa0ec7b7538b83711]), partition=
I20240502 01:54:00.023972 20345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d551360acc74a56863411fb656b6ee7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:00.025671 20476 tablet_service.cc:1467] Processing CreateTablet for tablet 9d551360acc74a56863411fb656b6ee7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8af1254734c2412aa0ec7b7538b83711]), partition=
I20240502 01:54:00.026160 20476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d551360acc74a56863411fb656b6ee7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:00.026232 22542 tablet_bootstrap.cc:492] T 062b9d90fb0444f4be37bf09770c9caf P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:54:00.028223 22542 tablet_bootstrap.cc:654] T 062b9d90fb0444f4be37bf09770c9caf P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:00.028820 22541 tablet_bootstrap.cc:492] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:54:00.030678 22541 tablet_bootstrap.cc:654] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:00.034843 22542 tablet_bootstrap.cc:492] T 062b9d90fb0444f4be37bf09770c9caf P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:54:00.035043 22542 ts_tablet_manager.cc:1367] T 062b9d90fb0444f4be37bf09770c9caf P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240502 01:54:00.035888 22542 raft_consensus.cc:348] T 062b9d90fb0444f4be37bf09770c9caf P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:00.036188 22542 raft_consensus.cc:374] T 062b9d90fb0444f4be37bf09770c9caf P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:00.036351 22542 raft_consensus.cc:729] T 062b9d90fb0444f4be37bf09770c9caf P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:54:00.036679 22542 consensus_queue.cc:260] T 062b9d90fb0444f4be37bf09770c9caf P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:00.037356 22542 ts_tablet_manager.cc:1398] T 062b9d90fb0444f4be37bf09770c9caf P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:00.037520 22217 consensus_queue.cc:1035] T b6d89235867f42418f8205167bb29755 P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:00.037748 22542 tablet_bootstrap.cc:492] T 61ab80ce377a44ac95a9c574e9a615f2 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:54:00.039891 22542 tablet_bootstrap.cc:654] T 61ab80ce377a44ac95a9c574e9a615f2 P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:00.042763 22543 tablet_bootstrap.cc:492] T 062b9d90fb0444f4be37bf09770c9caf P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:54:00.042901 22543 ts_tablet_manager.cc:1367] T 062b9d90fb0444f4be37bf09770c9caf P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.003s
I20240502 01:54:00.043608 22543 raft_consensus.cc:348] T 062b9d90fb0444f4be37bf09770c9caf P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:00.043797 22543 raft_consensus.cc:374] T 062b9d90fb0444f4be37bf09770c9caf P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:00.043884 22543 raft_consensus.cc:729] T 062b9d90fb0444f4be37bf09770c9caf P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:54:00.044117 22543 consensus_queue.cc:260] T 062b9d90fb0444f4be37bf09770c9caf P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:00.044632 22543 ts_tablet_manager.cc:1398] T 062b9d90fb0444f4be37bf09770c9caf P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:54:00.045706 22543 tablet_bootstrap.cc:492] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:54:00.046301 22541 tablet_bootstrap.cc:492] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:54:00.046434 22541 ts_tablet_manager.cc:1367] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.003s
I20240502 01:54:00.047423 22543 tablet_bootstrap.cc:654] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:00.047585 22541 raft_consensus.cc:348] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:00.047848 22541 raft_consensus.cc:374] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:00.047987 22541 raft_consensus.cc:729] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:54:00.048244 22541 consensus_queue.cc:260] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:00.048868 22541 ts_tablet_manager.cc:1398] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240502 01:54:00.049381 22541 tablet_bootstrap.cc:492] T 062b9d90fb0444f4be37bf09770c9caf P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:54:00.052310 22541 tablet_bootstrap.cc:654] T 062b9d90fb0444f4be37bf09770c9caf P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:00.054277 22544 tablet_bootstrap.cc:492] T f3e4c8f98b5e48e3ba6affc2df08943a P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:54:00.062208 20365 consensus_queue.cc:237] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } }
I20240502 01:54:00.064843 20497 raft_consensus.cc:1261] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:00.065151 17473 raft_consensus.cc:1261] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:00.065413 22156 consensus_queue.cc:1035] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:00.065693 22523 tablet_copy_client.cc:308] T 5152c002eadd4d54ab10a093bce45f17 P fdc2ac372492454c85387589a91f63af: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.66:38531
I20240502 01:54:00.066604 17627 tablet_copy_service.cc:140] P 3fceccefba274b1dbbd8216af8b0bc29: Received BeginTabletCopySession request for tablet 5152c002eadd4d54ab10a093bce45f17 from peer fdc2ac372492454c85387589a91f63af ({username='slave'} at 127.11.42.69:59447)
I20240502 01:54:00.066781 17627 tablet_copy_service.cc:161] P 3fceccefba274b1dbbd8216af8b0bc29: Beginning new tablet copy session on tablet 5152c002eadd4d54ab10a093bce45f17 from peer fdc2ac372492454c85387589a91f63af at {username='slave'} at 127.11.42.69:59447: session id = fdc2ac372492454c85387589a91f63af-5152c002eadd4d54ab10a093bce45f17
I20240502 01:54:00.067725 22542 tablet_bootstrap.cc:492] T 61ab80ce377a44ac95a9c574e9a615f2 P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:54:00.067852 22542 ts_tablet_manager.cc:1367] T 61ab80ce377a44ac95a9c574e9a615f2 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.000s
I20240502 01:54:00.068440 22542 raft_consensus.cc:348] T 61ab80ce377a44ac95a9c574e9a615f2 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:00.068619 22542 raft_consensus.cc:374] T 61ab80ce377a44ac95a9c574e9a615f2 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:00.068698 22542 raft_consensus.cc:729] T 61ab80ce377a44ac95a9c574e9a615f2 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:54:00.068909 17627 tablet_copy_source_session.cc:205] T 5152c002eadd4d54ab10a093bce45f17 P 3fceccefba274b1dbbd8216af8b0bc29: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:00.068944 22542 consensus_queue.cc:260] T 61ab80ce377a44ac95a9c574e9a615f2 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:00.069461 22542 ts_tablet_manager.cc:1398] T 61ab80ce377a44ac95a9c574e9a615f2 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:00.069795 22542 tablet_bootstrap.cc:492] T 9d551360acc74a56863411fb656b6ee7 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:54:00.071194 22179 raft_consensus.cc:1050] T 7e9be84ef6a043818f166f6b47898855 P dfa1152987644bd781b46c7e42a7a685: attempting to promote NON_VOTER 2f306549407b4bb281ccf2994035e4f1 to VOTER
I20240502 01:54:00.071429 22542 tablet_bootstrap.cc:654] T 9d551360acc74a56863411fb656b6ee7 P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:00.072063 22179 consensus_queue.cc:237] T 7e9be84ef6a043818f166f6b47898855 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } }
I20240502 01:54:00.074975 22523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5152c002eadd4d54ab10a093bce45f17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:00.075554 22541 tablet_bootstrap.cc:492] T 062b9d90fb0444f4be37bf09770c9caf P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:54:00.075574 22543 tablet_bootstrap.cc:492] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:54:00.075740 22543 ts_tablet_manager.cc:1367] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.031s user 0.000s sys 0.003s
I20240502 01:54:00.075796 22541 ts_tablet_manager.cc:1367] T 062b9d90fb0444f4be37bf09770c9caf P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.001s
I20240502 01:54:00.076638 22543 raft_consensus.cc:348] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:00.076871 22543 raft_consensus.cc:374] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:00.076704 22541 raft_consensus.cc:348] T 062b9d90fb0444f4be37bf09770c9caf P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:00.076953 22543 raft_consensus.cc:729] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:54:00.077132 22541 raft_consensus.cc:374] T 062b9d90fb0444f4be37bf09770c9caf P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:00.077216 22543 consensus_queue.cc:260] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:00.077347 22541 raft_consensus.cc:729] T 062b9d90fb0444f4be37bf09770c9caf P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:54:00.077792 22543 ts_tablet_manager.cc:1398] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:54:00.077716 22541 consensus_queue.cc:260] T 062b9d90fb0444f4be37bf09770c9caf P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:00.078089 22543 tablet_bootstrap.cc:492] T 9d551360acc74a56863411fb656b6ee7 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:54:00.078963 22541 ts_tablet_manager.cc:1398] T 062b9d90fb0444f4be37bf09770c9caf P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:54:00.079295 22541 tablet_bootstrap.cc:492] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:54:00.079753 22543 tablet_bootstrap.cc:654] T 9d551360acc74a56863411fb656b6ee7 P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:00.084055 22275 consensus_queue.cc:1035] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.018s
I20240502 01:54:00.085316 22542 tablet_bootstrap.cc:492] T 9d551360acc74a56863411fb656b6ee7 P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:54:00.086462 22542 ts_tablet_manager.cc:1367] T 9d551360acc74a56863411fb656b6ee7 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20240502 01:54:00.086725 20497 raft_consensus.cc:1261] T 7e9be84ef6a043818f166f6b47898855 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:00.086844 17606 raft_consensus.cc:1261] T 7e9be84ef6a043818f166f6b47898855 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:00.087085 22156 raft_consensus.cc:2932] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:00.087400 22037 consensus_queue.cc:1035] T 7e9be84ef6a043818f166f6b47898855 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:00.087738 22542 raft_consensus.cc:348] T 9d551360acc74a56863411fb656b6ee7 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:00.088006 22542 raft_consensus.cc:374] T 9d551360acc74a56863411fb656b6ee7 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:00.088006 22037 consensus_queue.cc:1035] T 7e9be84ef6a043818f166f6b47898855 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:00.088135 22542 raft_consensus.cc:729] T 9d551360acc74a56863411fb656b6ee7 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:54:00.088433 22542 consensus_queue.cc:260] T 9d551360acc74a56863411fb656b6ee7 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:00.089255 22542 ts_tablet_manager.cc:1398] T 9d551360acc74a56863411fb656b6ee7 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:00.089187 20498 raft_consensus.cc:2932] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:00.094364 22177 raft_consensus.cc:482] T 9d551360acc74a56863411fb656b6ee7 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:00.094552 22177 raft_consensus.cc:504] T 9d551360acc74a56863411fb656b6ee7 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:00.095207 22177 leader_election.cc:288] T 9d551360acc74a56863411fb656b6ee7 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:54:00.095872 22543 tablet_bootstrap.cc:492] T 9d551360acc74a56863411fb656b6ee7 P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:54:00.096031 22543 ts_tablet_manager.cc:1367] T 9d551360acc74a56863411fb656b6ee7 P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.005s
I20240502 01:54:00.095989 20498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d551360acc74a56863411fb656b6ee7" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
I20240502 01:54:00.096807 22543 raft_consensus.cc:348] T 9d551360acc74a56863411fb656b6ee7 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:00.097035 22543 raft_consensus.cc:374] T 9d551360acc74a56863411fb656b6ee7 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:00.097149 22543 raft_consensus.cc:729] T 9d551360acc74a56863411fb656b6ee7 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:54:00.097378 22543 consensus_queue.cc:260] T 9d551360acc74a56863411fb656b6ee7 P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:00.097738 20498 raft_consensus.cc:2454] T 9d551360acc74a56863411fb656b6ee7 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 0.
I20240502 01:54:00.097440 17474 raft_consensus.cc:2932] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:00.098006 22543 ts_tablet_manager.cc:1398] T 9d551360acc74a56863411fb656b6ee7 P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:00.098866 17803 leader_election.cc:302] T 9d551360acc74a56863411fb656b6ee7 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:54:00.099247 22177 raft_consensus.cc:2781] T 9d551360acc74a56863411fb656b6ee7 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:00.099340 22177 raft_consensus.cc:482] T 9d551360acc74a56863411fb656b6ee7 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:00.099514 22177 raft_consensus.cc:3037] T 9d551360acc74a56863411fb656b6ee7 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:00.102372 22177 raft_consensus.cc:504] T 9d551360acc74a56863411fb656b6ee7 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:00.103439 20496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d551360acc74a56863411fb656b6ee7" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
I20240502 01:54:00.103605 20496 raft_consensus.cc:3037] T 9d551360acc74a56863411fb656b6ee7 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:00.106280 20496 raft_consensus.cc:2454] T 9d551360acc74a56863411fb656b6ee7 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 1.
I20240502 01:54:00.107558 17803 leader_election.cc:302] T 9d551360acc74a56863411fb656b6ee7 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:54:00.108050 21667 raft_consensus.cc:2781] T 9d551360acc74a56863411fb656b6ee7 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:00.108176 21667 raft_consensus.cc:686] T 9d551360acc74a56863411fb656b6ee7 P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Becoming Leader. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Running, Role: LEADER
I20240502 01:54:00.108484 21667 consensus_queue.cc:237] T 9d551360acc74a56863411fb656b6ee7 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:00.108837 20365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d551360acc74a56863411fb656b6ee7" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:54:00.109508 20365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d551360acc74a56863411fb656b6ee7" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
W20240502 01:54:00.110277 17802 leader_election.cc:341] T 9d551360acc74a56863411fb656b6ee7 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): Illegal state: must be running to vote when last-logged opid is not known
W20240502 01:54:00.110623 17802 leader_election.cc:341] T 9d551360acc74a56863411fb656b6ee7 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:00.112182 17332 catalog_manager.cc:5484] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: config changed from index 3 to 4, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:00.114660 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1d8e2e0f6ed34e4cb004e1c5309c0f1f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:00.116456 17865 raft_consensus.cc:1261] T 7e9be84ef6a043818f166f6b47898855 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:00.117372 22179 consensus_queue.cc:1035] T 7e9be84ef6a043818f166f6b47898855 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:00.118077 20609 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d8e2e0f6ed34e4cb004e1c5309c0f1f with delete_type TABLET_DATA_TOMBSTONED (TS f19cc6cb1e76460aa6711e49a5775091 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33698
I20240502 01:54:00.118438 22526 tablet_replica.cc:330] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P f19cc6cb1e76460aa6711e49a5775091: stopping tablet replica
I20240502 01:54:00.118643 22526 raft_consensus.cc:2229] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:00.118855 22526 raft_consensus.cc:2258] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:00.119855 22526 ts_tablet_manager.cc:1875] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P f19cc6cb1e76460aa6711e49a5775091: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:00.122099 22177 leader_election.cc:288] T 9d551360acc74a56863411fb656b6ee7 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 election: Requested vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:54:00.122246 17474 raft_consensus.cc:1261] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade [term 2 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:54:00.125644 22177 consensus_queue.cc:1035] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:00.131197 22526 ts_tablet_manager.cc:1888] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P f19cc6cb1e76460aa6711e49a5775091: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:54:00.132694 22544 tablet_bootstrap.cc:654] T f3e4c8f98b5e48e3ba6affc2df08943a P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:00.133870 22526 log.cc:1196] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P f19cc6cb1e76460aa6711e49a5775091: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal/wals/1d8e2e0f6ed34e4cb004e1c5309c0f1f
I20240502 01:54:00.134672 17321 catalog_manager.cc:4830] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): tablet 1d8e2e0f6ed34e4cb004e1c5309c0f1f (table rebalancer_extra_table_10 [id=96aabcc026484fffa9ea0c0dc8d37340]) successfully deleted
I20240502 01:54:00.134832 20365 consensus_queue.cc:237] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } }
I20240502 01:54:00.137066 22541 tablet_bootstrap.cc:654] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:00.148975 22037 raft_consensus.cc:2932] T 7e9be84ef6a043818f166f6b47898855 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:00.152866 17865 raft_consensus.cc:2932] T 7e9be84ef6a043818f166f6b47898855 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:00.158077 20233 raft_consensus.cc:1261] T 0618761d30f34cfe9650d13c9275cdcb P fdc2ac372492454c85387589a91f63af [term 2 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:54:00.160326 22522 consensus_queue.cc:1035] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:00.163029 17606 raft_consensus.cc:2932] T 7e9be84ef6a043818f166f6b47898855 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:00.167727 17334 catalog_manager.cc:5484] T 7e9be84ef6a043818f166f6b47898855 P dfa1152987644bd781b46c7e42a7a685 reported cstate change: config changed from index 4 to 5, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:00.169154 22523 tablet_copy_client.cc:783] T 5152c002eadd4d54ab10a093bce45f17 P fdc2ac372492454c85387589a91f63af: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:00.169466 22523 tablet_copy_client.cc:647] T 5152c002eadd4d54ab10a093bce45f17 P fdc2ac372492454c85387589a91f63af: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:00.170997 22541 tablet_bootstrap.cc:492] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:54:00.171209 22541 ts_tablet_manager.cc:1367] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.092s user 0.007s sys 0.004s
I20240502 01:54:00.172114 22523 tablet_copy_client.cc:523] T 5152c002eadd4d54ab10a093bce45f17 P fdc2ac372492454c85387589a91f63af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:00.172058 22541 raft_consensus.cc:348] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:00.172526 22541 raft_consensus.cc:374] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:00.172686 22541 raft_consensus.cc:729] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:54:00.173043 22541 consensus_queue.cc:260] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:00.173555 17737 consensus_queue.cc:237] T 7e9be84ef6a043818f166f6b47898855 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } }
I20240502 01:54:00.174021 22541 ts_tablet_manager.cc:1398] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:54:00.177493 20498 raft_consensus.cc:1261] T 7e9be84ef6a043818f166f6b47898855 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:00.178359 22037 consensus_queue.cc:1035] T 7e9be84ef6a043818f166f6b47898855 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:00.179919 20498 raft_consensus.cc:2932] T 7e9be84ef6a043818f166f6b47898855 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:00.187381 22037 raft_consensus.cc:2932] T 7e9be84ef6a043818f166f6b47898855 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:00.187474 20496 raft_consensus.cc:2932] T 7e9be84ef6a043818f166f6b47898855 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:00.193835 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7e9be84ef6a043818f166f6b47898855 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:00.197942 17326 catalog_manager.cc:5484] T 7e9be84ef6a043818f166f6b47898855 P dfa1152987644bd781b46c7e42a7a685 reported cstate change: config changed from index 5 to 6, VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) evicted. New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 01:54:00.201781 20300 consensus_peers.cc:485] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): Couldn't send request to peer 3fceccefba274b1dbbd8216af8b0bc29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba1e82a39c2e439889da60007372a99b. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:00.202172 20231 raft_consensus.cc:1261] T ba1e82a39c2e439889da60007372a99b P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:00.203015 22465 consensus_queue.cc:1035] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:00.203274 17327 catalog_manager.cc:5484] T 9d551360acc74a56863411fb656b6ee7 P bb31bceba76d466db24b9341c04900c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb31bceba76d466db24b9341c04900c3 (127.11.42.68). New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:00.204877 22217 raft_consensus.cc:482] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:00.205075 22217 raft_consensus.cc:504] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:00.205811 22217 leader_election.cc:288] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:54:00.205998 20629 raft_consensus.cc:1261] T ba1e82a39c2e439889da60007372a99b P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:00.206451 17605 raft_consensus.cc:1261] T 7e9be84ef6a043818f166f6b47898855 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:00.207963 17865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61ab80ce377a44ac95a9c574e9a615f2" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:54:00.208199 17865 raft_consensus.cc:2454] T 61ab80ce377a44ac95a9c574e9a615f2 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 0.
I20240502 01:54:00.208731 20168 leader_election.cc:302] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb31bceba76d466db24b9341c04900c3, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:54:00.209051 22217 raft_consensus.cc:2781] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:00.209141 22217 raft_consensus.cc:482] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:00.209365 22217 raft_consensus.cc:3037] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:00.209647 22037 consensus_queue.cc:1035] T 7e9be84ef6a043818f166f6b47898855 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:00.210343 17737 raft_consensus.cc:1261] T 0618761d30f34cfe9650d13c9275cdcb P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:54:00.211123 22169 consensus_queue.cc:1035] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:00.211931 22217 raft_consensus.cc:504] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:00.212615 22217 leader_election.cc:288] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 election: Requested vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:54:00.213750 17845 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e9be84ef6a043818f166f6b47898855 with delete_type TABLET_DATA_TOMBSTONED (TS bb31bceba76d466db24b9341c04900c3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47226
I20240502 01:54:00.213817 17865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61ab80ce377a44ac95a9c574e9a615f2" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:54:00.213984 17865 raft_consensus.cc:3037] T 61ab80ce377a44ac95a9c574e9a615f2 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:00.214793 20366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61ab80ce377a44ac95a9c574e9a615f2" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
W20240502 01:54:00.215430 20167 leader_election.cc:341] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:00.216230 22156 consensus_queue.cc:1035] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:00.216737 17865 raft_consensus.cc:2454] T 61ab80ce377a44ac95a9c574e9a615f2 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 1.
I20240502 01:54:00.217180 20168 leader_election.cc:302] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bb31bceba76d466db24b9341c04900c3, fdc2ac372492454c85387589a91f63af; no voters: 6a36c952c51542c0bf7ea3ec2857a9f9
I20240502 01:54:00.217674 22429 raft_consensus.cc:2781] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:00.217826 22429 raft_consensus.cc:686] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Becoming Leader. State: Replica: fdc2ac372492454c85387589a91f63af, State: Running, Role: LEADER
I20240502 01:54:00.218089 22429 consensus_queue.cc:237] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:00.218904 22544 tablet_bootstrap.cc:492] T f3e4c8f98b5e48e3ba6affc2df08943a P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:54:00.219098 22544 ts_tablet_manager.cc:1367] T f3e4c8f98b5e48e3ba6affc2df08943a P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.165s user 0.003s sys 0.006s
I20240502 01:54:00.220019 22544 raft_consensus.cc:348] T f3e4c8f98b5e48e3ba6affc2df08943a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:00.220271 22544 raft_consensus.cc:374] T f3e4c8f98b5e48e3ba6affc2df08943a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:00.220465 22544 raft_consensus.cc:729] T f3e4c8f98b5e48e3ba6affc2df08943a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:54:00.220757 22544 consensus_queue.cc:260] T f3e4c8f98b5e48e3ba6affc2df08943a P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:00.221252 20366 raft_consensus.cc:1261] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:54:00.221874 22544 ts_tablet_manager.cc:1398] T f3e4c8f98b5e48e3ba6affc2df08943a P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:00.222139 22544 tablet_bootstrap.cc:492] T 61ab80ce377a44ac95a9c574e9a615f2 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:54:00.223565 22487 consensus_queue.cc:1035] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:00.223788 20497 raft_consensus.cc:1261] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:54:00.223832 20365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61ab80ce377a44ac95a9c574e9a615f2" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:54:00.224349 22487 consensus_queue.cc:1035] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:54:00.224653 20167 leader_election.cc:341] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:00.224586 17605 raft_consensus.cc:2932] T 7e9be84ef6a043818f166f6b47898855 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:00.226305 22522 raft_consensus.cc:1050] T de174ba4f7d848718ab8dbeb74369055 P bb31bceba76d466db24b9341c04900c3: attempting to promote NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 to VOTER
I20240502 01:54:00.227110 22522 consensus_queue.cc:237] T de174ba4f7d848718ab8dbeb74369055 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } }
I20240502 01:54:00.241973 17605 raft_consensus.cc:1261] T de174ba4f7d848718ab8dbeb74369055 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:00.242489 22177 consensus_queue.cc:1035] T de174ba4f7d848718ab8dbeb74369055 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:00.242799 20366 raft_consensus.cc:1261] T de174ba4f7d848718ab8dbeb74369055 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:00.243393 22527 consensus_queue.cc:1035] T de174ba4f7d848718ab8dbeb74369055 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:00.247407 17734 raft_consensus.cc:1261] T de174ba4f7d848718ab8dbeb74369055 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:00.247985 21667 consensus_queue.cc:1035] T de174ba4f7d848718ab8dbeb74369055 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:00.269418 22465 raft_consensus.cc:2932] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } } }
I20240502 01:54:00.276618 22581 ts_tablet_manager.cc:902] T 7c697210a14d4982a939ea7965cb769d P f19cc6cb1e76460aa6711e49a5775091: Initiating tablet copy from peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:54:00.278898 20629 raft_consensus.cc:2932] T ba1e82a39c2e439889da60007372a99b P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } } }
I20240502 01:54:00.282992 22573 tablet_replica.cc:330] T 7e9be84ef6a043818f166f6b47898855 P bb31bceba76d466db24b9341c04900c3: stopping tablet replica
I20240502 01:54:00.284170 17326 catalog_manager.cc:5484] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af reported cstate change: term changed from 0 to 1, leader changed from <none> to fdc2ac372492454c85387589a91f63af (127.11.42.69). New cstate: current_term: 1 leader_uuid: "fdc2ac372492454c85387589a91f63af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:00.285471 22429 raft_consensus.cc:482] T 062b9d90fb0444f4be37bf09770c9caf P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:00.285135 17330 catalog_manager.cc:5484] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: config changed from index -1 to 2, NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) added. New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:00.285629 22429 raft_consensus.cc:504] T 062b9d90fb0444f4be37bf09770c9caf P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:00.286942 17864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "062b9d90fb0444f4be37bf09770c9caf" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:54:00.287204 17864 raft_consensus.cc:2454] T 062b9d90fb0444f4be37bf09770c9caf P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 0.
I20240502 01:54:00.287685 20168 leader_election.cc:302] T 062b9d90fb0444f4be37bf09770c9caf P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb31bceba76d466db24b9341c04900c3, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:54:00.287973 22440 raft_consensus.cc:2781] T 062b9d90fb0444f4be37bf09770c9caf P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:00.288067 22440 raft_consensus.cc:482] T 062b9d90fb0444f4be37bf09770c9caf P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:00.288144 22440 raft_consensus.cc:3037] T 062b9d90fb0444f4be37bf09770c9caf P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:00.290704 22440 raft_consensus.cc:504] T 062b9d90fb0444f4be37bf09770c9caf P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:00.291338 22440 leader_election.cc:288] T 062b9d90fb0444f4be37bf09770c9caf P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 election: Requested vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:54:00.291404 17474 consensus_queue.cc:237] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:00.292416 20231 raft_consensus.cc:2932] T ba1e82a39c2e439889da60007372a99b P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } } }
I20240502 01:54:00.293239 17864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "062b9d90fb0444f4be37bf09770c9caf" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:54:00.293474 17864 raft_consensus.cc:3037] T 062b9d90fb0444f4be37bf09770c9caf P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:00.296283 17864 raft_consensus.cc:2454] T 062b9d90fb0444f4be37bf09770c9caf P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 1.
I20240502 01:54:00.296666 20168 leader_election.cc:302] T 062b9d90fb0444f4be37bf09770c9caf P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb31bceba76d466db24b9341c04900c3, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:54:00.297086 22440 raft_consensus.cc:2781] T 062b9d90fb0444f4be37bf09770c9caf P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:00.297230 22440 raft_consensus.cc:686] T 062b9d90fb0444f4be37bf09770c9caf P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Becoming Leader. State: Replica: fdc2ac372492454c85387589a91f63af, State: Running, Role: LEADER
I20240502 01:54:00.297493 22440 consensus_queue.cc:237] T 062b9d90fb0444f4be37bf09770c9caf P fdc2ac372492454c85387589a91f63af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:00.285392 22573 raft_consensus.cc:2229] T 7e9be84ef6a043818f166f6b47898855 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:00.298071 22573 raft_consensus.cc:2258] T 7e9be84ef6a043818f166f6b47898855 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:00.299029 22523 tablet_bootstrap.cc:492] T 5152c002eadd4d54ab10a093bce45f17 P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:54:00.299672 22544 tablet_bootstrap.cc:654] T 61ab80ce377a44ac95a9c574e9a615f2 P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:00.300645 20497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "062b9d90fb0444f4be37bf09770c9caf" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
I20240502 01:54:00.300879 20497 raft_consensus.cc:3037] T 062b9d90fb0444f4be37bf09770c9caf P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:00.302865 22573 ts_tablet_manager.cc:1875] T 7e9be84ef6a043818f166f6b47898855 P bb31bceba76d466db24b9341c04900c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:00.304744 22429 leader_election.cc:288] T 062b9d90fb0444f4be37bf09770c9caf P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:54:00.304857 20495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "062b9d90fb0444f4be37bf09770c9caf" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
I20240502 01:54:00.318023 17737 raft_consensus.cc:1261] T 648b1cc4d8b54c5eb0948ca8c1984a1f P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:54:00.318691 21920 consensus_queue.cc:1035] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:00.319051 20365 raft_consensus.cc:1261] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:54:00.320082 21920 consensus_queue.cc:1035] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:00.321573 20497 raft_consensus.cc:2454] T 062b9d90fb0444f4be37bf09770c9caf P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 1.
I20240502 01:54:00.321576 17334 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:54:00.325485 22480 raft_consensus.cc:2932] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 4da690c0fa13407a85c0689ce8de9ade [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:00.333636 17330 catalog_manager.cc:5484] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: config changed from index 3 to 5, VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) evicted. New cstate: current_term: 2 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:00.335896 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 648b1cc4d8b54c5eb0948ca8c1984a1f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:00.336170 17735 raft_consensus.cc:2932] T 648b1cc4d8b54c5eb0948ca8c1984a1f P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:00.342378 22586 ts_tablet_manager.cc:902] T 59c288e7279f4119afb6c3bf860e6dc3 P fdc2ac372492454c85387589a91f63af: Initiating tablet copy from peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:54:00.343698 20365 raft_consensus.cc:2932] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:00.346019 20476 tablet_service.cc:1514] Processing DeleteTablet for tablet 648b1cc4d8b54c5eb0948ca8c1984a1f with delete_type TABLET_DATA_TOMBSTONED (TS 2f306549407b4bb281ccf2994035e4f1 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35018
W20240502 01:54:00.353199 17802 consensus_peers.cc:485] T 9d551360acc74a56863411fb656b6ee7 P bb31bceba76d466db24b9341c04900c3 -> Peer 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): Couldn't send request to peer 6a36c952c51542c0bf7ea3ec2857a9f9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:00.354707 20497 raft_consensus.cc:1261] T 9d551360acc74a56863411fb656b6ee7 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:00.355366 21667 consensus_queue.cc:1035] T 9d551360acc74a56863411fb656b6ee7 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:00.363678 22590 ts_tablet_manager.cc:902] T b39b88755cd74b0294d264215aea99cf P 2f306549407b4bb281ccf2994035e4f1: Initiating tablet copy from peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:54:00.364985 17330 catalog_manager.cc:5484] T 062b9d90fb0444f4be37bf09770c9caf P fdc2ac372492454c85387589a91f63af reported cstate change: term changed from 0 to 1, leader changed from <none> to fdc2ac372492454c85387589a91f63af (127.11.42.69). New cstate: current_term: 1 leader_uuid: "fdc2ac372492454c85387589a91f63af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:00.372114 22525 tablet_replica.cc:330] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1: stopping tablet replica
I20240502 01:54:00.371801 21667 raft_consensus.cc:2932] T de174ba4f7d848718ab8dbeb74369055 P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:54:00.372339 22525 raft_consensus.cc:2229] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:00.372488 22525 pending_rounds.cc:62] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1: Trying to abort 1 pending ops.
I20240502 01:54:00.372603 22525 pending_rounds.cc:68] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1: Aborting op as it isn't in flight: id { term: 2 index: 4 } timestamp: 7023062383126777856 op_type: NO_OP noop_request { }
I20240502 01:54:00.372771 22525 raft_consensus.cc:2866] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240502 01:54:00.372960 22525 raft_consensus.cc:2258] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:00.374384 22573 ts_tablet_manager.cc:1888] T 7e9be84ef6a043818f166f6b47898855 P bb31bceba76d466db24b9341c04900c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:00.376773 22581 tablet_copy_client.cc:308] T 7c697210a14d4982a939ea7965cb769d P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.68:44083
I20240502 01:54:00.377105 17605 raft_consensus.cc:2932] T de174ba4f7d848718ab8dbeb74369055 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:54:00.380407 22525 ts_tablet_manager.cc:1875] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:00.380725 22544 tablet_bootstrap.cc:492] T 61ab80ce377a44ac95a9c574e9a615f2 P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:54:00.380880 22544 ts_tablet_manager.cc:1367] T 61ab80ce377a44ac95a9c574e9a615f2 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.159s user 0.005s sys 0.005s
I20240502 01:54:00.381636 22544 raft_consensus.cc:348] T 61ab80ce377a44ac95a9c574e9a615f2 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:00.381861 22544 raft_consensus.cc:374] T 61ab80ce377a44ac95a9c574e9a615f2 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:00.381940 22544 raft_consensus.cc:729] T 61ab80ce377a44ac95a9c574e9a615f2 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:54:00.382067 20366 raft_consensus.cc:2932] T de174ba4f7d848718ab8dbeb74369055 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:54:00.378001 17887 tablet_copy_service.cc:140] P bb31bceba76d466db24b9341c04900c3: Received BeginTabletCopySession request for tablet 7c697210a14d4982a939ea7965cb769d from peer f19cc6cb1e76460aa6711e49a5775091 ({username='slave'} at 127.11.42.72:53629)
I20240502 01:54:00.383651 17887 tablet_copy_service.cc:161] P bb31bceba76d466db24b9341c04900c3: Beginning new tablet copy session on tablet 7c697210a14d4982a939ea7965cb769d from peer f19cc6cb1e76460aa6711e49a5775091 at {username='slave'} at 127.11.42.72:53629: session id = f19cc6cb1e76460aa6711e49a5775091-7c697210a14d4982a939ea7965cb769d
I20240502 01:54:00.386941 17887 tablet_copy_source_session.cc:205] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:00.388350 22581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c697210a14d4982a939ea7965cb769d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:00.388571 17326 catalog_manager.cc:5484] T de174ba4f7d848718ab8dbeb74369055 P bb31bceba76d466db24b9341c04900c3 reported cstate change: config changed from index 4 to 5, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:00.390410 22573 log.cc:1196] T 7e9be84ef6a043818f166f6b47898855 P bb31bceba76d466db24b9341c04900c3: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/wals/7e9be84ef6a043818f166f6b47898855
I20240502 01:54:00.390822 22525 ts_tablet_manager.cc:1888] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 01:54:00.390961 22525 log.cc:1196] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 2f306549407b4bb281ccf2994035e4f1: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal/wals/648b1cc4d8b54c5eb0948ca8c1984a1f
I20240502 01:54:00.391731 17322 catalog_manager.cc:4830] TS 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): tablet 648b1cc4d8b54c5eb0948ca8c1984a1f (table rebalancer_extra_table_6 [id=3e245c8534e84c55922b7cd77670a4a3]) successfully deleted
I20240502 01:54:00.393963 17322 catalog_manager.cc:4830] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): tablet 7e9be84ef6a043818f166f6b47898855 (table rebalance_test_table_7 [id=c6c3a53b24874f31aa38118254253dcb]) successfully deleted
I20240502 01:54:00.395869 17735 raft_consensus.cc:2932] T de174ba4f7d848718ab8dbeb74369055 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:54:00.403170 17330 catalog_manager.cc:5576] TS 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623) does not have the latest schema for tablet 9d551360acc74a56863411fb656b6ee7 (table rebalancer_dynamic_table [id=8af1254734c2412aa0ec7b7538b83711]). Expected version 1 got 0
W20240502 01:54:00.405395 17542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.42.72:42411, user_credentials={real_user=slave}} blocked reactor thread for 80935us
I20240502 01:54:00.411811 22581 tablet_copy_client.cc:783] T 7c697210a14d4982a939ea7965cb769d P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:00.412098 22581 tablet_copy_client.cc:647] T 7c697210a14d4982a939ea7965cb769d P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:00.414947 22581 tablet_copy_client.cc:523] T 7c697210a14d4982a939ea7965cb769d P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:00.415853 22608 tablet.cc:1722] T 9d551360acc74a56863411fb656b6ee7 P bb31bceba76d466db24b9341c04900c3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:00.418064 22608 tablet.cc:1620] T 9d551360acc74a56863411fb656b6ee7 P bb31bceba76d466db24b9341c04900c3: MemRowSet was empty: no flush needed.
I20240502 01:54:00.422518 22544 consensus_queue.cc:260] T 61ab80ce377a44ac95a9c574e9a615f2 P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:00.422854 17864 consensus_queue.cc:237] T de174ba4f7d848718ab8dbeb74369055 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } }
I20240502 01:54:00.424084 22581 tablet_bootstrap.cc:492] T 7c697210a14d4982a939ea7965cb769d P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:54:00.424822 17606 raft_consensus.cc:1261] T de174ba4f7d848718ab8dbeb74369055 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:00.424852 20366 raft_consensus.cc:1261] T de174ba4f7d848718ab8dbeb74369055 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:00.425580 22527 consensus_queue.cc:1035] T de174ba4f7d848718ab8dbeb74369055 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:00.425856 22522 consensus_queue.cc:1035] T de174ba4f7d848718ab8dbeb74369055 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:00.427743 22440 raft_consensus.cc:482] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:00.427902 22440 raft_consensus.cc:504] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:00.428369 22609 tablet.cc:1722] T 9d551360acc74a56863411fb656b6ee7 P 2f306549407b4bb281ccf2994035e4f1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:00.428565 22440 leader_election.cc:288] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:54:00.429040 20498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3e4c8f98b5e48e3ba6affc2df08943a" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
I20240502 01:54:00.429178 22609 tablet.cc:1620] T 9d551360acc74a56863411fb656b6ee7 P 2f306549407b4bb281ccf2994035e4f1: MemRowSet was empty: no flush needed.
I20240502 01:54:00.429275 20498 raft_consensus.cc:2454] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 0.
I20240502 01:54:00.429507 20365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3e4c8f98b5e48e3ba6affc2df08943a" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:54:00.429668 20168 leader_election.cc:302] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:54:00.429704 20365 raft_consensus.cc:2454] T f3e4c8f98b5e48e3ba6affc2df08943a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 0.
I20240502 01:54:00.431602 22440 raft_consensus.cc:2781] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:00.431700 22440 raft_consensus.cc:482] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:00.431792 22440 raft_consensus.cc:3037] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:00.432235 17334 catalog_manager.cc:5576] TS 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623) does not have the latest schema for tablet 9d551360acc74a56863411fb656b6ee7 (table rebalancer_dynamic_table [id=8af1254734c2412aa0ec7b7538b83711]). Expected version 1 got 0
I20240502 01:54:00.434437 22440 raft_consensus.cc:504] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:00.434612 21667 raft_consensus.cc:2932] T de174ba4f7d848718ab8dbeb74369055 P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:54:00.437140 20366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3e4c8f98b5e48e3ba6affc2df08943a" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
I20240502 01:54:00.437705 20366 raft_consensus.cc:3037] T f3e4c8f98b5e48e3ba6affc2df08943a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:00.438719 22440 leader_election.cc:288] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 election: Requested vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:54:00.439095 20498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3e4c8f98b5e48e3ba6affc2df08943a" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
I20240502 01:54:00.439316 20498 raft_consensus.cc:3037] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:00.439627 20365 raft_consensus.cc:2932] T de174ba4f7d848718ab8dbeb74369055 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:54:00.441958 20498 raft_consensus.cc:2454] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 1.
I20240502 01:54:00.442317 20168 leader_election.cc:302] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:54:00.442598 22440 raft_consensus.cc:2781] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:00.443087 22440 raft_consensus.cc:686] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Becoming Leader. State: Replica: fdc2ac372492454c85387589a91f63af, State: Running, Role: LEADER
I20240502 01:54:00.443377 22440 consensus_queue.cc:237] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:00.444123 20366 raft_consensus.cc:2454] T f3e4c8f98b5e48e3ba6affc2df08943a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 1.
I20240502 01:54:00.444334 22544 ts_tablet_manager.cc:1398] T 61ab80ce377a44ac95a9c574e9a615f2 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.063s user 0.000s sys 0.003s
I20240502 01:54:00.448180 17334 catalog_manager.cc:5484] T de174ba4f7d848718ab8dbeb74369055 P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: config changed from index 5 to 6, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:54:00.448501 17322 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet de174ba4f7d848718ab8dbeb74369055 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:00.450637 17605 raft_consensus.cc:2932] T de174ba4f7d848718ab8dbeb74369055 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:54:00.453970 17716 tablet_service.cc:1514] Processing DeleteTablet for tablet de174ba4f7d848718ab8dbeb74369055 with delete_type TABLET_DATA_TOMBSTONED (TS dfa1152987644bd781b46c7e42a7a685 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52198
I20240502 01:54:00.464556 17330 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=8af1254734c2412aa0ec7b7538b83711] alter complete (version 1)
I20240502 01:54:00.467319 22586 tablet_copy_client.cc:308] T 59c288e7279f4119afb6c3bf860e6dc3 P fdc2ac372492454c85387589a91f63af: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.68:44083
I20240502 01:54:00.467994 17887 tablet_copy_service.cc:140] P bb31bceba76d466db24b9341c04900c3: Received BeginTabletCopySession request for tablet 59c288e7279f4119afb6c3bf860e6dc3 from peer fdc2ac372492454c85387589a91f63af ({username='slave'} at 127.11.42.69:56011)
I20240502 01:54:00.468178 17887 tablet_copy_service.cc:161] P bb31bceba76d466db24b9341c04900c3: Beginning new tablet copy session on tablet 59c288e7279f4119afb6c3bf860e6dc3 from peer fdc2ac372492454c85387589a91f63af at {username='slave'} at 127.11.42.69:56011: session id = fdc2ac372492454c85387589a91f63af-59c288e7279f4119afb6c3bf860e6dc3
I20240502 01:54:00.471311 17887 tablet_copy_source_session.cc:205] T 59c288e7279f4119afb6c3bf860e6dc3 P bb31bceba76d466db24b9341c04900c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:00.472893 22586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59c288e7279f4119afb6c3bf860e6dc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:00.451333 22544 tablet_bootstrap.cc:492] T 9d551360acc74a56863411fb656b6ee7 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:54:00.480175 22590 tablet_copy_client.cc:308] T b39b88755cd74b0294d264215aea99cf P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.65:38321
I20240502 01:54:00.481094 17496 tablet_copy_service.cc:140] P 4da690c0fa13407a85c0689ce8de9ade: Received BeginTabletCopySession request for tablet b39b88755cd74b0294d264215aea99cf from peer 2f306549407b4bb281ccf2994035e4f1 ({username='slave'} at 127.11.42.71:44055)
I20240502 01:54:00.481276 17496 tablet_copy_service.cc:161] P 4da690c0fa13407a85c0689ce8de9ade: Beginning new tablet copy session on tablet b39b88755cd74b0294d264215aea99cf from peer 2f306549407b4bb281ccf2994035e4f1 at {username='slave'} at 127.11.42.71:44055: session id = 2f306549407b4bb281ccf2994035e4f1-b39b88755cd74b0294d264215aea99cf
I20240502 01:54:00.484195 17496 tablet_copy_source_session.cc:205] T b39b88755cd74b0294d264215aea99cf P 4da690c0fa13407a85c0689ce8de9ade: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:00.487174 22590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b39b88755cd74b0294d264215aea99cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:00.492689 22544 tablet_bootstrap.cc:654] T 9d551360acc74a56863411fb656b6ee7 P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:00.493253 17334 catalog_manager.cc:5484] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af reported cstate change: term changed from 0 to 1, leader changed from <none> to fdc2ac372492454c85387589a91f63af (127.11.42.69). New cstate: current_term: 1 leader_uuid: "fdc2ac372492454c85387589a91f63af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:00.499548 22590 tablet_copy_client.cc:783] T b39b88755cd74b0294d264215aea99cf P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:00.499821 22590 tablet_copy_client.cc:647] T b39b88755cd74b0294d264215aea99cf P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:00.500252 22617 tablet_replica.cc:330] T de174ba4f7d848718ab8dbeb74369055 P dfa1152987644bd781b46c7e42a7a685: stopping tablet replica
I20240502 01:54:00.501668 22590 tablet_copy_client.cc:523] T b39b88755cd74b0294d264215aea99cf P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:00.503892 22586 tablet_copy_client.cc:783] T 59c288e7279f4119afb6c3bf860e6dc3 P fdc2ac372492454c85387589a91f63af: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:00.504196 22586 tablet_copy_client.cc:647] T 59c288e7279f4119afb6c3bf860e6dc3 P fdc2ac372492454c85387589a91f63af: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:00.508066 22617 raft_consensus.cc:2229] T de174ba4f7d848718ab8dbeb74369055 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:00.508903 22586 tablet_copy_client.cc:523] T 59c288e7279f4119afb6c3bf860e6dc3 P fdc2ac372492454c85387589a91f63af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:00.509315 22617 raft_consensus.cc:2258] T de174ba4f7d848718ab8dbeb74369055 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:00.510967 22544 tablet_bootstrap.cc:492] T 9d551360acc74a56863411fb656b6ee7 P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:54:00.511773 22590 tablet_bootstrap.cc:492] T b39b88755cd74b0294d264215aea99cf P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:54:00.511967 22544 ts_tablet_manager.cc:1367] T 9d551360acc74a56863411fb656b6ee7 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.061s user 0.001s sys 0.002s
I20240502 01:54:00.513432 22617 ts_tablet_manager.cc:1875] T de174ba4f7d848718ab8dbeb74369055 P dfa1152987644bd781b46c7e42a7a685: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:00.520721 22586 tablet_bootstrap.cc:492] T 59c288e7279f4119afb6c3bf860e6dc3 P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:54:00.521287 22544 raft_consensus.cc:348] T 9d551360acc74a56863411fb656b6ee7 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:00.521569 22544 raft_consensus.cc:374] T 9d551360acc74a56863411fb656b6ee7 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:00.521699 22544 raft_consensus.cc:729] T 9d551360acc74a56863411fb656b6ee7 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:54:00.524376 22544 consensus_queue.cc:260] T 9d551360acc74a56863411fb656b6ee7 P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:00.526937 22544 ts_tablet_manager.cc:1398] T 9d551360acc74a56863411fb656b6ee7 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240502 01:54:00.530553 20496 consensus_queue.cc:237] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:54:00.534379 22581 tablet_bootstrap.cc:492] T 7c697210a14d4982a939ea7965cb769d P f19cc6cb1e76460aa6711e49a5775091: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:00.534849 22581 tablet_bootstrap.cc:492] T 7c697210a14d4982a939ea7965cb769d P f19cc6cb1e76460aa6711e49a5775091: Bootstrap complete.
I20240502 01:54:00.535171 22581 ts_tablet_manager.cc:1367] T 7c697210a14d4982a939ea7965cb769d P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.111s user 0.043s sys 0.006s
I20240502 01:54:00.536321 22581 raft_consensus.cc:348] T 7c697210a14d4982a939ea7965cb769d P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:54:00.536579 22581 raft_consensus.cc:729] T 7c697210a14d4982a939ea7965cb769d P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: LEARNER
I20240502 01:54:00.536801 22581 consensus_queue.cc:260] T 7c697210a14d4982a939ea7965cb769d P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:54:00.538398 22581 ts_tablet_manager.cc:1398] T 7c697210a14d4982a939ea7965cb769d P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240502 01:54:00.539013 17887 tablet_copy_service.cc:342] P bb31bceba76d466db24b9341c04900c3: Request end of tablet copy session f19cc6cb1e76460aa6711e49a5775091-7c697210a14d4982a939ea7965cb769d received from {username='slave'} at 127.11.42.72:53629
I20240502 01:54:00.539147 17887 tablet_copy_service.cc:433] P bb31bceba76d466db24b9341c04900c3: ending tablet copy session f19cc6cb1e76460aa6711e49a5775091-7c697210a14d4982a939ea7965cb769d on tablet 7c697210a14d4982a939ea7965cb769d with peer f19cc6cb1e76460aa6711e49a5775091
I20240502 01:54:00.541646 17334 catalog_manager.cc:5576] TS 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155) does not have the latest schema for tablet 9d551360acc74a56863411fb656b6ee7 (table rebalancer_dynamic_table [id=8af1254734c2412aa0ec7b7538b83711]). Expected version 1 got 0
I20240502 01:54:00.550762 17330 catalog_manager.cc:5576] TS 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155) does not have the latest schema for tablet 9d551360acc74a56863411fb656b6ee7 (table rebalancer_dynamic_table [id=8af1254734c2412aa0ec7b7538b83711]). Expected version 1 got 0
I20240502 01:54:00.559469 22617 ts_tablet_manager.cc:1888] T de174ba4f7d848718ab8dbeb74369055 P dfa1152987644bd781b46c7e42a7a685: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
W20240502 01:54:00.564858 20433 consensus_peers.cc:485] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aebde87a7f114537abd9c1d8b7e97004. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:00.564982 20365 raft_consensus.cc:1261] T aebde87a7f114537abd9c1d8b7e97004 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:00.565749 22393 consensus_queue.cc:1035] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:00.566236 22617 log.cc:1196] T de174ba4f7d848718ab8dbeb74369055 P dfa1152987644bd781b46c7e42a7a685: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/wals/de174ba4f7d848718ab8dbeb74369055
I20240502 01:54:00.566269 17864 raft_consensus.cc:1261] T aebde87a7f114537abd9c1d8b7e97004 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:00.567210 17321 catalog_manager.cc:4830] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): tablet de174ba4f7d848718ab8dbeb74369055 (table rebalance_test_table_9 [id=ffe6d29170f24ad48433354f548b9b2a]) successfully deleted
I20240502 01:54:00.573727 22623 consensus_queue.cc:1035] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:00.588579 17330 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:54:00.596200 22393 raft_consensus.cc:2932] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:00.600513 22523 tablet_bootstrap.cc:492] T 5152c002eadd4d54ab10a093bce45f17 P fdc2ac372492454c85387589a91f63af: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:00.601069 22523 tablet_bootstrap.cc:492] T 5152c002eadd4d54ab10a093bce45f17 P fdc2ac372492454c85387589a91f63af: Bootstrap complete.
I20240502 01:54:00.601452 22523 ts_tablet_manager.cc:1367] T 5152c002eadd4d54ab10a093bce45f17 P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.302s user 0.046s sys 0.003s
I20240502 01:54:00.602319 22523 raft_consensus.cc:348] T 5152c002eadd4d54ab10a093bce45f17 P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } }
I20240502 01:54:00.602653 22523 raft_consensus.cc:729] T 5152c002eadd4d54ab10a093bce45f17 P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: LEARNER
I20240502 01:54:00.602941 22523 consensus_queue.cc:260] T 5152c002eadd4d54ab10a093bce45f17 P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } }
I20240502 01:54:00.603340 17865 raft_consensus.cc:2932] T aebde87a7f114537abd9c1d8b7e97004 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:00.604683 22523 ts_tablet_manager.cc:1398] T 5152c002eadd4d54ab10a093bce45f17 P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:00.609378 17334 catalog_manager.cc:5484] T aebde87a7f114537abd9c1d8b7e97004 P bb31bceba76d466db24b9341c04900c3 reported cstate change: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:00.609623 22608 tablet.cc:1722] T 9d551360acc74a56863411fb656b6ee7 P bb31bceba76d466db24b9341c04900c3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:00.610123 22608 tablet.cc:1620] T 9d551360acc74a56863411fb656b6ee7 P bb31bceba76d466db24b9341c04900c3: MemRowSet was empty: no flush needed.
I20240502 01:54:00.612147 17627 tablet_copy_service.cc:342] P 3fceccefba274b1dbbd8216af8b0bc29: Request end of tablet copy session fdc2ac372492454c85387589a91f63af-5152c002eadd4d54ab10a093bce45f17 received from {username='slave'} at 127.11.42.69:59447
I20240502 01:54:00.612284 17627 tablet_copy_service.cc:433] P 3fceccefba274b1dbbd8216af8b0bc29: ending tablet copy session fdc2ac372492454c85387589a91f63af-5152c002eadd4d54ab10a093bce45f17 on tablet 5152c002eadd4d54ab10a093bce45f17 with peer fdc2ac372492454c85387589a91f63af
I20240502 01:54:00.617415 22586 tablet_bootstrap.cc:492] T 59c288e7279f4119afb6c3bf860e6dc3 P fdc2ac372492454c85387589a91f63af: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:00.617838 22586 tablet_bootstrap.cc:492] T 59c288e7279f4119afb6c3bf860e6dc3 P fdc2ac372492454c85387589a91f63af: Bootstrap complete.
I20240502 01:54:00.618108 22586 ts_tablet_manager.cc:1367] T 59c288e7279f4119afb6c3bf860e6dc3 P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.097s user 0.040s sys 0.013s
I20240502 01:54:00.619889 22586 raft_consensus.cc:348] T 59c288e7279f4119afb6c3bf860e6dc3 P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } }
I20240502 01:54:00.620170 22586 raft_consensus.cc:729] T 59c288e7279f4119afb6c3bf860e6dc3 P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: LEARNER
I20240502 01:54:00.620386 22586 consensus_queue.cc:260] T 59c288e7279f4119afb6c3bf860e6dc3 P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } }
I20240502 01:54:00.621052 22586 ts_tablet_manager.cc:1398] T 59c288e7279f4119afb6c3bf860e6dc3 P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:54:00.620941 22609 tablet.cc:1722] T 9d551360acc74a56863411fb656b6ee7 P 2f306549407b4bb281ccf2994035e4f1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:00.621441 22609 tablet.cc:1620] T 9d551360acc74a56863411fb656b6ee7 P 2f306549407b4bb281ccf2994035e4f1: MemRowSet was empty: no flush needed.
I20240502 01:54:00.621874 17887 tablet_copy_service.cc:342] P bb31bceba76d466db24b9341c04900c3: Request end of tablet copy session fdc2ac372492454c85387589a91f63af-59c288e7279f4119afb6c3bf860e6dc3 received from {username='slave'} at 127.11.42.69:56011
I20240502 01:54:00.622052 17887 tablet_copy_service.cc:433] P bb31bceba76d466db24b9341c04900c3: ending tablet copy session fdc2ac372492454c85387589a91f63af-59c288e7279f4119afb6c3bf860e6dc3 on tablet 59c288e7279f4119afb6c3bf860e6dc3 with peer fdc2ac372492454c85387589a91f63af
I20240502 01:54:00.627127 17334 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=8af1254734c2412aa0ec7b7538b83711] alter complete (version 2)
I20240502 01:54:00.633494 17865 consensus_queue.cc:237] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } }
I20240502 01:54:00.635107 17735 raft_consensus.cc:1261] T 0618761d30f34cfe9650d13c9275cdcb P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:54:00.635408 20231 raft_consensus.cc:1261] T 0618761d30f34cfe9650d13c9275cdcb P fdc2ac372492454c85387589a91f63af [term 2 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:54:00.635800 22527 consensus_queue.cc:1035] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:00.636121 22522 consensus_queue.cc:1035] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:00.638651 22169 raft_consensus.cc:2932] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:00.639087 17735 raft_consensus.cc:2932] T 0618761d30f34cfe9650d13c9275cdcb P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:00.639183 20231 raft_consensus.cc:2932] T 0618761d30f34cfe9650d13c9275cdcb P fdc2ac372492454c85387589a91f63af [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:00.642746 17322 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0618761d30f34cfe9650d13c9275cdcb with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:00.645167 17330 catalog_manager.cc:5484] T 0618761d30f34cfe9650d13c9275cdcb P dfa1152987644bd781b46c7e42a7a685 reported cstate change: config changed from index 5 to 7, VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) evicted. New cstate: current_term: 2 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:00.647534 20365 raft_consensus.cc:2932] T aebde87a7f114537abd9c1d8b7e97004 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:00.650440 17454 tablet_service.cc:1514] Processing DeleteTablet for tablet 0618761d30f34cfe9650d13c9275cdcb with delete_type TABLET_DATA_TOMBSTONED (TS 4da690c0fa13407a85c0689ce8de9ade not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:54520
W20240502 01:54:00.655987 20434 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.42.68:44083, user_credentials={real_user=slave}} blocked reactor thread for 47326.9us
I20240502 01:54:00.662827 22630 tablet_replica.cc:330] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade: stopping tablet replica
I20240502 01:54:00.663250 22630 raft_consensus.cc:2229] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:00.663666 22630 raft_consensus.cc:2258] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:00.664705 22630 ts_tablet_manager.cc:1875] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:00.671801 22630 ts_tablet_manager.cc:1888] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 01:54:00.671981 22630 log.cc:1196] T 0618761d30f34cfe9650d13c9275cdcb P 4da690c0fa13407a85c0689ce8de9ade: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/0618761d30f34cfe9650d13c9275cdcb
I20240502 01:54:00.672919 17321 catalog_manager.cc:4830] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): tablet 0618761d30f34cfe9650d13c9275cdcb (table rebalance_test_table_15 [id=21fe1df2eeed4efd801d10edf592ecce]) successfully deleted
I20240502 01:54:00.678442 20365 raft_consensus.cc:3037] T 9d551360acc74a56863411fb656b6ee7 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:00.686832 22590 tablet_bootstrap.cc:492] T b39b88755cd74b0294d264215aea99cf P 2f306549407b4bb281ccf2994035e4f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:00.687278 22590 tablet_bootstrap.cc:492] T b39b88755cd74b0294d264215aea99cf P 2f306549407b4bb281ccf2994035e4f1: Bootstrap complete.
I20240502 01:54:00.687556 22590 ts_tablet_manager.cc:1367] T b39b88755cd74b0294d264215aea99cf P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.176s user 0.062s sys 0.007s
I20240502 01:54:00.688369 22590 raft_consensus.cc:348] T b39b88755cd74b0294d264215aea99cf P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
I20240502 01:54:00.688832 22590 raft_consensus.cc:729] T b39b88755cd74b0294d264215aea99cf P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: LEARNER
I20240502 01:54:00.689118 22590 consensus_queue.cc:260] T b39b88755cd74b0294d264215aea99cf P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
I20240502 01:54:00.690961 22590 ts_tablet_manager.cc:1398] T b39b88755cd74b0294d264215aea99cf P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240502 01:54:00.691634 17496 tablet_copy_service.cc:342] P 4da690c0fa13407a85c0689ce8de9ade: Request end of tablet copy session 2f306549407b4bb281ccf2994035e4f1-b39b88755cd74b0294d264215aea99cf received from {username='slave'} at 127.11.42.71:44055
I20240502 01:54:00.691798 17496 tablet_copy_service.cc:433] P 4da690c0fa13407a85c0689ce8de9ade: ending tablet copy session 2f306549407b4bb281ccf2994035e4f1-b39b88755cd74b0294d264215aea99cf on tablet b39b88755cd74b0294d264215aea99cf with peer 2f306549407b4bb281ccf2994035e4f1
I20240502 01:54:00.695457 22429 consensus_queue.cc:1035] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:00.701808 20366 raft_consensus.cc:3037] T 61ab80ce377a44ac95a9c574e9a615f2 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:00.705103 22217 consensus_queue.cc:1035] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:00.705922 22156 raft_consensus.cc:482] T 61ab80ce377a44ac95a9c574e9a615f2 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:00.705852 22639 tablet.cc:1722] T 9d551360acc74a56863411fb656b6ee7 P 6a36c952c51542c0bf7ea3ec2857a9f9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:00.706084 22156 raft_consensus.cc:504] T 61ab80ce377a44ac95a9c574e9a615f2 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:00.706713 22156 leader_election.cc:288] T 61ab80ce377a44ac95a9c574e9a615f2 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:54:00.707902 17864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61ab80ce377a44ac95a9c574e9a615f2" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:54:00.708590 22639 tablet.cc:1620] T 9d551360acc74a56863411fb656b6ee7 P 6a36c952c51542c0bf7ea3ec2857a9f9: MemRowSet was empty: no flush needed.
I20240502 01:54:00.709059 17330 catalog_manager.cc:5576] TS 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155) does not have the latest schema for tablet 9d551360acc74a56863411fb656b6ee7 (table rebalancer_dynamic_table [id=8af1254734c2412aa0ec7b7538b83711]). Expected version 2 got 0
I20240502 01:54:00.709322 20231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61ab80ce377a44ac95a9c574e9a615f2" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:54:00.714239 20303 leader_election.cc:302] T 61ab80ce377a44ac95a9c574e9a615f2 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9; no voters: bb31bceba76d466db24b9341c04900c3, fdc2ac372492454c85387589a91f63af
I20240502 01:54:00.718750 22639 tablet.cc:1722] T 9d551360acc74a56863411fb656b6ee7 P 6a36c952c51542c0bf7ea3ec2857a9f9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:00.719161 22639 tablet.cc:1620] T 9d551360acc74a56863411fb656b6ee7 P 6a36c952c51542c0bf7ea3ec2857a9f9: MemRowSet was empty: no flush needed.
I20240502 01:54:00.719209 17735 consensus_queue.cc:237] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
W20240502 01:54:00.724499 17672 consensus_peers.cc:485] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0d9f168de0d1410da2518abf936035ff. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:00.725082 17606 raft_consensus.cc:1261] T 0d9f168de0d1410da2518abf936035ff P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:00.725802 22179 consensus_queue.cc:1035] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:00.725996 17864 raft_consensus.cc:1261] T 0d9f168de0d1410da2518abf936035ff P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:00.726936 22179 consensus_queue.cc:1035] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:00.728538 22037 raft_consensus.cc:2932] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:54:00.729243 17606 raft_consensus.cc:2932] T 0d9f168de0d1410da2518abf936035ff P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:54:00.732139 17330 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:54:00.735774 22217 consensus_queue.cc:1035] T 062b9d90fb0444f4be37bf09770c9caf P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:00.735791 17334 catalog_manager.cc:5484] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 reported cstate change: config changed from index -1 to 4, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:00.746402 22608 tablet.cc:1722] T 9d551360acc74a56863411fb656b6ee7 P bb31bceba76d466db24b9341c04900c3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:00.752674 22639 tablet.cc:1722] T 9d551360acc74a56863411fb656b6ee7 P 6a36c952c51542c0bf7ea3ec2857a9f9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:00.755419 22609 tablet.cc:1722] T 9d551360acc74a56863411fb656b6ee7 P 2f306549407b4bb281ccf2994035e4f1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:00.761353 17330 catalog_manager.cc:5576] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083) does not have the latest schema for tablet 9d551360acc74a56863411fb656b6ee7 (table rebalancer_dynamic_table_new_name [id=8af1254734c2412aa0ec7b7538b83711]). Expected version 3 got 2
I20240502 01:54:00.778558 22217 consensus_queue.cc:1035] T 062b9d90fb0444f4be37bf09770c9caf P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:00.784693 22643 ts_tablet_manager.cc:902] T ba1e82a39c2e439889da60007372a99b P 3fceccefba274b1dbbd8216af8b0bc29: Initiating tablet copy from peer 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:54:00.790105 17334 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table_new_name [id=8af1254734c2412aa0ec7b7538b83711] alter complete (version 3)
I20240502 01:54:00.790828 22156 raft_consensus.cc:2726] T 61ab80ce377a44ac95a9c574e9a615f2 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:00.801317 17864 raft_consensus.cc:2932] T 0d9f168de0d1410da2518abf936035ff P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:54:00.806730 20496 raft_consensus.cc:1203] T b39b88755cd74b0294d264215aea99cf P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:00.808780 22643 tablet_copy_client.cc:308] T ba1e82a39c2e439889da60007372a99b P 3fceccefba274b1dbbd8216af8b0bc29: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.70:39155
I20240502 01:54:00.809716 20387 tablet_copy_service.cc:140] P 6a36c952c51542c0bf7ea3ec2857a9f9: Received BeginTabletCopySession request for tablet ba1e82a39c2e439889da60007372a99b from peer 3fceccefba274b1dbbd8216af8b0bc29 ({username='slave'} at 127.11.42.66:36601)
I20240502 01:54:00.809883 20387 tablet_copy_service.cc:161] P 6a36c952c51542c0bf7ea3ec2857a9f9: Beginning new tablet copy session on tablet ba1e82a39c2e439889da60007372a99b from peer 3fceccefba274b1dbbd8216af8b0bc29 at {username='slave'} at 127.11.42.66:36601: session id = 3fceccefba274b1dbbd8216af8b0bc29-ba1e82a39c2e439889da60007372a99b
I20240502 01:54:00.812105 20387 tablet_copy_source_session.cc:205] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:00.813321 22643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba1e82a39c2e439889da60007372a99b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:00.813855 20231 raft_consensus.cc:1203] T 59c288e7279f4119afb6c3bf860e6dc3 P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:00.822568 22643 tablet_copy_client.cc:783] T ba1e82a39c2e439889da60007372a99b P 3fceccefba274b1dbbd8216af8b0bc29: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:00.829205 22643 tablet_copy_client.cc:647] T ba1e82a39c2e439889da60007372a99b P 3fceccefba274b1dbbd8216af8b0bc29: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:00.831305 22643 tablet_copy_client.cc:523] T ba1e82a39c2e439889da60007372a99b P 3fceccefba274b1dbbd8216af8b0bc29: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:00.837284 22643 tablet_bootstrap.cc:492] T ba1e82a39c2e439889da60007372a99b P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:54:00.848663 22429 consensus_queue.cc:1035] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:00.852036 22643 tablet_bootstrap.cc:492] T ba1e82a39c2e439889da60007372a99b P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:00.852474 22643 tablet_bootstrap.cc:492] T ba1e82a39c2e439889da60007372a99b P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap complete.
I20240502 01:54:00.852732 22643 ts_tablet_manager.cc:1367] T ba1e82a39c2e439889da60007372a99b P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.001s
I20240502 01:54:00.853080 20629 raft_consensus.cc:1203] T 7c697210a14d4982a939ea7965cb769d P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:00.853618 22643 raft_consensus.cc:348] T ba1e82a39c2e439889da60007372a99b P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } }
I20240502 01:54:00.853853 22643 raft_consensus.cc:729] T ba1e82a39c2e439889da60007372a99b P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: LEARNER
I20240502 01:54:00.854079 22643 consensus_queue.cc:260] T ba1e82a39c2e439889da60007372a99b P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } }
I20240502 01:54:00.855755 22643 ts_tablet_manager.cc:1398] T ba1e82a39c2e439889da60007372a99b P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 01:54:00.857828 20387 tablet_copy_service.cc:342] P 6a36c952c51542c0bf7ea3ec2857a9f9: Request end of tablet copy session 3fceccefba274b1dbbd8216af8b0bc29-ba1e82a39c2e439889da60007372a99b received from {username='slave'} at 127.11.42.66:36601
I20240502 01:54:00.857960 20387 tablet_copy_service.cc:433] P 6a36c952c51542c0bf7ea3ec2857a9f9: ending tablet copy session 3fceccefba274b1dbbd8216af8b0bc29-ba1e82a39c2e439889da60007372a99b on tablet ba1e82a39c2e439889da60007372a99b with peer 3fceccefba274b1dbbd8216af8b0bc29
I20240502 01:54:00.870766 22440 consensus_queue.cc:1035] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:00.897754 17327 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:00.898002 17327 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:00.902346 17327 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet 9d551360acc74a56863411fb656b6ee7
I20240502 01:54:00.903451 20345 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d551360acc74a56863411fb656b6ee7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:00 UTC) from {username='slave'} at 127.0.0.1:40196
I20240502 01:54:00.903898 17845 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d551360acc74a56863411fb656b6ee7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:00 UTC) from {username='slave'} at 127.0.0.1:47226
I20240502 01:54:00.904069 20476 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d551360acc74a56863411fb656b6ee7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:00 UTC) from {username='slave'} at 127.0.0.1:35018
I20240502 01:54:00.915606 22667 tablet_replica.cc:330] T 9d551360acc74a56863411fb656b6ee7 P bb31bceba76d466db24b9341c04900c3: stopping tablet replica
I20240502 01:54:00.915817 22667 raft_consensus.cc:2229] T 9d551360acc74a56863411fb656b6ee7 P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:00.916205 22667 raft_consensus.cc:2258] T 9d551360acc74a56863411fb656b6ee7 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:00.917587 22667 ts_tablet_manager.cc:1875] T 9d551360acc74a56863411fb656b6ee7 P bb31bceba76d466db24b9341c04900c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:00.922950 22667 ts_tablet_manager.cc:1888] T 9d551360acc74a56863411fb656b6ee7 P bb31bceba76d466db24b9341c04900c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:00.923424 22665 tablet_replica.cc:330] T 9d551360acc74a56863411fb656b6ee7 P 6a36c952c51542c0bf7ea3ec2857a9f9: stopping tablet replica
I20240502 01:54:00.923734 22665 raft_consensus.cc:2229] T 9d551360acc74a56863411fb656b6ee7 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:00.924072 22665 raft_consensus.cc:2258] T 9d551360acc74a56863411fb656b6ee7 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:00.924649 22667 log.cc:1196] T 9d551360acc74a56863411fb656b6ee7 P bb31bceba76d466db24b9341c04900c3: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/wals/9d551360acc74a56863411fb656b6ee7
I20240502 01:54:00.925336 22665 ts_tablet_manager.cc:1875] T 9d551360acc74a56863411fb656b6ee7 P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:00.925925 22668 tablet_replica.cc:330] T 9d551360acc74a56863411fb656b6ee7 P 2f306549407b4bb281ccf2994035e4f1: stopping tablet replica
I20240502 01:54:00.926179 22668 raft_consensus.cc:2229] T 9d551360acc74a56863411fb656b6ee7 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:00.926486 22668 raft_consensus.cc:2258] T 9d551360acc74a56863411fb656b6ee7 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:00.927425 22668 ts_tablet_manager.cc:1875] T 9d551360acc74a56863411fb656b6ee7 P 2f306549407b4bb281ccf2994035e4f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:00.931687 22665 ts_tablet_manager.cc:1888] T 9d551360acc74a56863411fb656b6ee7 P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:00.931825 22665 log.cc:1196] T 9d551360acc74a56863411fb656b6ee7 P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal/wals/9d551360acc74a56863411fb656b6ee7
I20240502 01:54:00.932220 22667 ts_tablet_manager.cc:1909] T 9d551360acc74a56863411fb656b6ee7 P bb31bceba76d466db24b9341c04900c3: Deleting consensus metadata
I20240502 01:54:00.933862 22668 ts_tablet_manager.cc:1888] T 9d551360acc74a56863411fb656b6ee7 P 2f306549407b4bb281ccf2994035e4f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:00.934141 22665 ts_tablet_manager.cc:1909] T 9d551360acc74a56863411fb656b6ee7 P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting consensus metadata
I20240502 01:54:00.934252 22668 log.cc:1196] T 9d551360acc74a56863411fb656b6ee7 P 2f306549407b4bb281ccf2994035e4f1: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal/wals/9d551360acc74a56863411fb656b6ee7
I20240502 01:54:00.934805 22668 ts_tablet_manager.cc:1909] T 9d551360acc74a56863411fb656b6ee7 P 2f306549407b4bb281ccf2994035e4f1: Deleting consensus metadata
I20240502 01:54:00.934773 17322 catalog_manager.cc:4830] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): tablet 9d551360acc74a56863411fb656b6ee7 (table rebalancer_dynamic_table_new_name [id=8af1254734c2412aa0ec7b7538b83711]) successfully deleted
I20240502 01:54:00.935711 17321 catalog_manager.cc:4830] TS 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): tablet 9d551360acc74a56863411fb656b6ee7 (table rebalancer_dynamic_table_new_name [id=8af1254734c2412aa0ec7b7538b83711]) successfully deleted
I20240502 01:54:00.936122 17322 catalog_manager.cc:4830] TS 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): tablet 9d551360acc74a56863411fb656b6ee7 (table rebalancer_dynamic_table_new_name [id=8af1254734c2412aa0ec7b7538b83711]) successfully deleted
I20240502 01:54:00.985863 20231 raft_consensus.cc:1203] T 5152c002eadd4d54ab10a093bce45f17 P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:01.005558 17327 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:54:01.013940 20212 tablet_service.cc:1467] Processing CreateTablet for tablet 3e657454f8744d5d9b9a271505acbaf0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b8199e32355e4fafa1cb6c61fa02e838]), partition=
I20240502 01:54:01.014354 20212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e657454f8744d5d9b9a271505acbaf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:01.014353 20609 tablet_service.cc:1467] Processing CreateTablet for tablet 3e657454f8744d5d9b9a271505acbaf0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b8199e32355e4fafa1cb6c61fa02e838]), partition=
I20240502 01:54:01.014494 17587 tablet_service.cc:1467] Processing CreateTablet for tablet 3e657454f8744d5d9b9a271505acbaf0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b8199e32355e4fafa1cb6c61fa02e838]), partition=
I20240502 01:54:01.014678 20609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e657454f8744d5d9b9a271505acbaf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:01.014895 17587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e657454f8744d5d9b9a271505acbaf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:01.030231 22704 tablet_bootstrap.cc:492] T 3e657454f8744d5d9b9a271505acbaf0 P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:54:01.030231 22705 tablet_bootstrap.cc:492] T 3e657454f8744d5d9b9a271505acbaf0 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:54:01.031800 22703 tablet_bootstrap.cc:492] T 3e657454f8744d5d9b9a271505acbaf0 P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:54:01.034055 22705 tablet_bootstrap.cc:654] T 3e657454f8744d5d9b9a271505acbaf0 P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:01.034346 22704 tablet_bootstrap.cc:654] T 3e657454f8744d5d9b9a271505acbaf0 P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:01.042136 22703 tablet_bootstrap.cc:654] T 3e657454f8744d5d9b9a271505acbaf0 P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:01.054108 22703 tablet_bootstrap.cc:492] T 3e657454f8744d5d9b9a271505acbaf0 P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:54:01.054278 22703 ts_tablet_manager.cc:1367] T 3e657454f8744d5d9b9a271505acbaf0 P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.000s
I20240502 01:54:01.055179 22704 tablet_bootstrap.cc:492] T 3e657454f8744d5d9b9a271505acbaf0 P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:54:01.055140 22703 raft_consensus.cc:348] T 3e657454f8744d5d9b9a271505acbaf0 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:01.055373 22703 raft_consensus.cc:374] T 3e657454f8744d5d9b9a271505acbaf0 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:01.055470 22703 raft_consensus.cc:729] T 3e657454f8744d5d9b9a271505acbaf0 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:54:01.055472 22704 ts_tablet_manager.cc:1367] T 3e657454f8744d5d9b9a271505acbaf0 P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.007s
I20240502 01:54:01.055675 22703 consensus_queue.cc:260] T 3e657454f8744d5d9b9a271505acbaf0 P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:01.056134 22703 ts_tablet_manager.cc:1398] T 3e657454f8744d5d9b9a271505acbaf0 P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:54:01.056223 22704 raft_consensus.cc:348] T 3e657454f8744d5d9b9a271505acbaf0 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:01.056423 22704 raft_consensus.cc:374] T 3e657454f8744d5d9b9a271505acbaf0 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:01.056505 22704 raft_consensus.cc:729] T 3e657454f8744d5d9b9a271505acbaf0 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:54:01.056718 22704 consensus_queue.cc:260] T 3e657454f8744d5d9b9a271505acbaf0 P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:01.057189 22705 tablet_bootstrap.cc:492] T 3e657454f8744d5d9b9a271505acbaf0 P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:54:01.057420 22705 ts_tablet_manager.cc:1367] T 3e657454f8744d5d9b9a271505acbaf0 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.006s
I20240502 01:54:01.058351 22705 raft_consensus.cc:348] T 3e657454f8744d5d9b9a271505acbaf0 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:01.058630 22705 raft_consensus.cc:374] T 3e657454f8744d5d9b9a271505acbaf0 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:01.058808 22705 raft_consensus.cc:729] T 3e657454f8744d5d9b9a271505acbaf0 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:54:01.058960 22704 ts_tablet_manager.cc:1398] T 3e657454f8744d5d9b9a271505acbaf0 P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:54:01.059098 22705 consensus_queue.cc:260] T 3e657454f8744d5d9b9a271505acbaf0 P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:01.059679 22705 ts_tablet_manager.cc:1398] T 3e657454f8744d5d9b9a271505acbaf0 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:54:01.109555 17327 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_extra_table_26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:01.118464 20609 tablet_service.cc:1467] Processing CreateTablet for tablet 2ea539ebb37f42f68c166242a06f8d2e (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=43fc79d9e659415194e6983d21d5384a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:01.118921 20609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ea539ebb37f42f68c166242a06f8d2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:01.119171 20476 tablet_service.cc:1467] Processing CreateTablet for tablet 2ea539ebb37f42f68c166242a06f8d2e (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=43fc79d9e659415194e6983d21d5384a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:01.119647 20476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ea539ebb37f42f68c166242a06f8d2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:01.121748 20212 tablet_service.cc:1467] Processing CreateTablet for tablet aefcd4cee0724eeb866d76ae95bd0080 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=43fc79d9e659415194e6983d21d5384a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:01.122153 20212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aefcd4cee0724eeb866d76ae95bd0080. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:01.122306 20477 tablet_service.cc:1467] Processing CreateTablet for tablet 3dcd890bdd2d413e95401897e9f1bcab (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=43fc79d9e659415194e6983d21d5384a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:01.122831 20477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dcd890bdd2d413e95401897e9f1bcab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:01.123500 17587 tablet_service.cc:1467] Processing CreateTablet for tablet 3dcd890bdd2d413e95401897e9f1bcab (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=43fc79d9e659415194e6983d21d5384a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:01.123937 17587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dcd890bdd2d413e95401897e9f1bcab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:01.125746 20345 tablet_service.cc:1467] Processing CreateTablet for tablet 3dcd890bdd2d413e95401897e9f1bcab (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=43fc79d9e659415194e6983d21d5384a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:01.125864 22704 tablet_bootstrap.cc:492] T 2ea539ebb37f42f68c166242a06f8d2e P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:54:01.126272 20345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dcd890bdd2d413e95401897e9f1bcab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:01.127465 22704 tablet_bootstrap.cc:654] T 2ea539ebb37f42f68c166242a06f8d2e P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:01.129359 20610 tablet_service.cc:1467] Processing CreateTablet for tablet aefcd4cee0724eeb866d76ae95bd0080 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=43fc79d9e659415194e6983d21d5384a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:01.129743 20610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aefcd4cee0724eeb866d76ae95bd0080. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:01.133384 17456 tablet_service.cc:1467] Processing CreateTablet for tablet aefcd4cee0724eeb866d76ae95bd0080 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=43fc79d9e659415194e6983d21d5384a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:01.133790 17456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aefcd4cee0724eeb866d76ae95bd0080. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:01.137136 22741 tablet_bootstrap.cc:492] T 3dcd890bdd2d413e95401897e9f1bcab P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:54:01.137964 22703 tablet_bootstrap.cc:492] T aefcd4cee0724eeb866d76ae95bd0080 P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:54:01.139524 22741 tablet_bootstrap.cc:654] T 3dcd890bdd2d413e95401897e9f1bcab P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:01.139614 22703 tablet_bootstrap.cc:654] T aefcd4cee0724eeb866d76ae95bd0080 P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:01.139981 17454 tablet_service.cc:1467] Processing CreateTablet for tablet 2ea539ebb37f42f68c166242a06f8d2e (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=43fc79d9e659415194e6983d21d5384a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:01.140357 17454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ea539ebb37f42f68c166242a06f8d2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:01.142659 22704 tablet_bootstrap.cc:492] T 2ea539ebb37f42f68c166242a06f8d2e P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:54:01.142947 22704 ts_tablet_manager.cc:1367] T 2ea539ebb37f42f68c166242a06f8d2e P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.005s
I20240502 01:54:01.143991 22704 raft_consensus.cc:348] T 2ea539ebb37f42f68c166242a06f8d2e P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:01.144322 22704 raft_consensus.cc:374] T 2ea539ebb37f42f68c166242a06f8d2e P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:01.144465 22704 raft_consensus.cc:729] T 2ea539ebb37f42f68c166242a06f8d2e P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:54:01.144788 22704 consensus_queue.cc:260] T 2ea539ebb37f42f68c166242a06f8d2e P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:01.145447 22704 ts_tablet_manager.cc:1398] T 2ea539ebb37f42f68c166242a06f8d2e P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:01.155848 22705 tablet_bootstrap.cc:492] T 3dcd890bdd2d413e95401897e9f1bcab P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:54:01.157642 22705 tablet_bootstrap.cc:654] T 3dcd890bdd2d413e95401897e9f1bcab P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:01.168143 22741 tablet_bootstrap.cc:492] T 3dcd890bdd2d413e95401897e9f1bcab P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:54:01.169070 22741 ts_tablet_manager.cc:1367] T 3dcd890bdd2d413e95401897e9f1bcab P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.032s user 0.010s sys 0.009s
I20240502 01:54:01.170457 22741 raft_consensus.cc:348] T 3dcd890bdd2d413e95401897e9f1bcab P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:01.171407 22705 tablet_bootstrap.cc:492] T 3dcd890bdd2d413e95401897e9f1bcab P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:54:01.171640 22705 ts_tablet_manager.cc:1367] T 3dcd890bdd2d413e95401897e9f1bcab P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.005s
I20240502 01:54:01.173061 22705 raft_consensus.cc:348] T 3dcd890bdd2d413e95401897e9f1bcab P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:01.173432 22705 raft_consensus.cc:374] T 3dcd890bdd2d413e95401897e9f1bcab P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:01.173558 22705 raft_consensus.cc:729] T 3dcd890bdd2d413e95401897e9f1bcab P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:54:01.173869 22705 consensus_queue.cc:260] T 3dcd890bdd2d413e95401897e9f1bcab P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:01.174630 22705 ts_tablet_manager.cc:1398] T 3dcd890bdd2d413e95401897e9f1bcab P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:01.174805 22741 raft_consensus.cc:374] T 3dcd890bdd2d413e95401897e9f1bcab P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:01.174947 22741 raft_consensus.cc:729] T 3dcd890bdd2d413e95401897e9f1bcab P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:54:01.175202 22741 consensus_queue.cc:260] T 3dcd890bdd2d413e95401897e9f1bcab P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:01.176208 22746 tablet_bootstrap.cc:492] T aefcd4cee0724eeb866d76ae95bd0080 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:54:01.176882 22704 tablet_bootstrap.cc:492] T aefcd4cee0724eeb866d76ae95bd0080 P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:54:01.177492 22748 tablet_bootstrap.cc:492] T 3dcd890bdd2d413e95401897e9f1bcab P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:54:01.177963 22746 tablet_bootstrap.cc:654] T aefcd4cee0724eeb866d76ae95bd0080 P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:01.179874 22704 tablet_bootstrap.cc:654] T aefcd4cee0724eeb866d76ae95bd0080 P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:01.184614 22741 ts_tablet_manager.cc:1398] T 3dcd890bdd2d413e95401897e9f1bcab P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240502 01:54:01.185814 22748 tablet_bootstrap.cc:654] T 3dcd890bdd2d413e95401897e9f1bcab P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:01.188226 22741 tablet_bootstrap.cc:492] T 2ea539ebb37f42f68c166242a06f8d2e P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:54:01.191646 22741 tablet_bootstrap.cc:654] T 2ea539ebb37f42f68c166242a06f8d2e P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:01.192142 22745 ts_tablet_manager.cc:902] T aebde87a7f114537abd9c1d8b7e97004 P dfa1152987644bd781b46c7e42a7a685: Initiating tablet copy from peer 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:54:01.192590 22703 tablet_bootstrap.cc:492] T aefcd4cee0724eeb866d76ae95bd0080 P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:54:01.192838 22703 ts_tablet_manager.cc:1367] T aefcd4cee0724eeb866d76ae95bd0080 P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.055s user 0.002s sys 0.019s
I20240502 01:54:01.193688 22703 raft_consensus.cc:348] T aefcd4cee0724eeb866d76ae95bd0080 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:01.193985 22703 raft_consensus.cc:374] T aefcd4cee0724eeb866d76ae95bd0080 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:01.194100 22703 raft_consensus.cc:729] T aefcd4cee0724eeb866d76ae95bd0080 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:54:01.194429 22703 consensus_queue.cc:260] T aefcd4cee0724eeb866d76ae95bd0080 P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:01.196136 22703 ts_tablet_manager.cc:1398] T aefcd4cee0724eeb866d76ae95bd0080 P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:54:01.205705 22746 tablet_bootstrap.cc:492] T aefcd4cee0724eeb866d76ae95bd0080 P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:54:01.205976 22746 ts_tablet_manager.cc:1367] T aefcd4cee0724eeb866d76ae95bd0080 P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.000s
I20240502 01:54:01.206959 22746 raft_consensus.cc:348] T aefcd4cee0724eeb866d76ae95bd0080 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:01.207283 22746 raft_consensus.cc:374] T aefcd4cee0724eeb866d76ae95bd0080 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:01.207422 22746 raft_consensus.cc:729] T aefcd4cee0724eeb866d76ae95bd0080 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:54:01.207984 22746 consensus_queue.cc:260] T aefcd4cee0724eeb866d76ae95bd0080 P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:01.211521 22746 ts_tablet_manager.cc:1398] T aefcd4cee0724eeb866d76ae95bd0080 P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:54:01.211848 22746 tablet_bootstrap.cc:492] T 2ea539ebb37f42f68c166242a06f8d2e P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:54:01.213595 22746 tablet_bootstrap.cc:654] T 2ea539ebb37f42f68c166242a06f8d2e P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:01.217952 22704 tablet_bootstrap.cc:492] T aefcd4cee0724eeb866d76ae95bd0080 P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:54:01.218159 22704 ts_tablet_manager.cc:1367] T aefcd4cee0724eeb866d76ae95bd0080 P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.003s
I20240502 01:54:01.219908 22704 raft_consensus.cc:348] T aefcd4cee0724eeb866d76ae95bd0080 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:01.220279 22704 raft_consensus.cc:374] T aefcd4cee0724eeb866d76ae95bd0080 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:01.220443 22704 raft_consensus.cc:729] T aefcd4cee0724eeb866d76ae95bd0080 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:54:01.220729 22704 consensus_queue.cc:260] T aefcd4cee0724eeb866d76ae95bd0080 P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:01.222771 22704 ts_tablet_manager.cc:1398] T aefcd4cee0724eeb866d76ae95bd0080 P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:54:01.223016 22746 tablet_bootstrap.cc:492] T 2ea539ebb37f42f68c166242a06f8d2e P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:54:01.223668 22746 ts_tablet_manager.cc:1367] T 2ea539ebb37f42f68c166242a06f8d2e P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20240502 01:54:01.224992 22746 raft_consensus.cc:348] T 2ea539ebb37f42f68c166242a06f8d2e P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:01.226763 22487 raft_consensus.cc:1050] T b39b88755cd74b0294d264215aea99cf P 4da690c0fa13407a85c0689ce8de9ade: attempting to promote NON_VOTER 2f306549407b4bb281ccf2994035e4f1 to VOTER
I20240502 01:54:01.230373 22741 tablet_bootstrap.cc:492] T 2ea539ebb37f42f68c166242a06f8d2e P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:54:01.230962 22741 ts_tablet_manager.cc:1367] T 2ea539ebb37f42f68c166242a06f8d2e P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.012s
I20240502 01:54:01.230756 22487 consensus_queue.cc:237] T b39b88755cd74b0294d264215aea99cf P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } }
I20240502 01:54:01.232442 22741 raft_consensus.cc:348] T 2ea539ebb37f42f68c166242a06f8d2e P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:01.234437 22741 raft_consensus.cc:374] T 2ea539ebb37f42f68c166242a06f8d2e P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:01.234683 22741 raft_consensus.cc:729] T 2ea539ebb37f42f68c166242a06f8d2e P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:54:01.235076 22741 consensus_queue.cc:260] T 2ea539ebb37f42f68c166242a06f8d2e P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:01.242926 22741 ts_tablet_manager.cc:1398] T 2ea539ebb37f42f68c166242a06f8d2e P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20240502 01:54:01.243646 20496 raft_consensus.cc:1261] T b39b88755cd74b0294d264215aea99cf P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:01.244149 21920 consensus_queue.cc:1035] T b39b88755cd74b0294d264215aea99cf P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:01.251715 17734 raft_consensus.cc:1261] T b39b88755cd74b0294d264215aea99cf P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:01.252291 21920 consensus_queue.cc:1035] T b39b88755cd74b0294d264215aea99cf P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:01.253563 22527 raft_consensus.cc:1050] T 59c288e7279f4119afb6c3bf860e6dc3 P bb31bceba76d466db24b9341c04900c3: attempting to promote NON_VOTER fdc2ac372492454c85387589a91f63af to VOTER
I20240502 01:54:01.254451 22527 consensus_queue.cc:237] T 59c288e7279f4119afb6c3bf860e6dc3 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } }
I20240502 01:54:01.258569 17605 raft_consensus.cc:1203] T ba1e82a39c2e439889da60007372a99b P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:01.260339 22748 tablet_bootstrap.cc:492] T 3dcd890bdd2d413e95401897e9f1bcab P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:54:01.260496 22748 ts_tablet_manager.cc:1367] T 3dcd890bdd2d413e95401897e9f1bcab P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.083s user 0.011s sys 0.022s
I20240502 01:54:01.261324 22748 raft_consensus.cc:348] T 3dcd890bdd2d413e95401897e9f1bcab P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:01.261538 22748 raft_consensus.cc:374] T 3dcd890bdd2d413e95401897e9f1bcab P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:01.261634 22748 raft_consensus.cc:729] T 3dcd890bdd2d413e95401897e9f1bcab P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:54:01.262741 22748 consensus_queue.cc:260] T 3dcd890bdd2d413e95401897e9f1bcab P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:01.267349 22748 ts_tablet_manager.cc:1398] T 3dcd890bdd2d413e95401897e9f1bcab P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240502 01:54:01.273898 20231 raft_consensus.cc:1261] T 59c288e7279f4119afb6c3bf860e6dc3 P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:01.275501 21667 consensus_queue.cc:1035] T 59c288e7279f4119afb6c3bf860e6dc3 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:01.277896 17605 raft_consensus.cc:1261] T b39b88755cd74b0294d264215aea99cf P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:01.278507 21920 consensus_queue.cc:1035] T b39b88755cd74b0294d264215aea99cf P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:01.288300 17605 raft_consensus.cc:1261] T 59c288e7279f4119afb6c3bf860e6dc3 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:01.288892 17735 raft_consensus.cc:1261] T 59c288e7279f4119afb6c3bf860e6dc3 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:01.289297 22527 consensus_queue.cc:1035] T 59c288e7279f4119afb6c3bf860e6dc3 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:01.289628 21667 consensus_queue.cc:1035] T 59c288e7279f4119afb6c3bf860e6dc3 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:01.296574 21667 raft_consensus.cc:2932] T 59c288e7279f4119afb6c3bf860e6dc3 P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:01.301597 20231 raft_consensus.cc:2932] T 59c288e7279f4119afb6c3bf860e6dc3 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:01.307677 17334 catalog_manager.cc:5484] T 59c288e7279f4119afb6c3bf860e6dc3 P fdc2ac372492454c85387589a91f63af reported cstate change: config changed from index 4 to 5, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:01.310832 22766 ts_tablet_manager.cc:902] T 0d9f168de0d1410da2518abf936035ff P f19cc6cb1e76460aa6711e49a5775091: Initiating tablet copy from peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:54:01.312160 21920 raft_consensus.cc:2932] T b39b88755cd74b0294d264215aea99cf P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:01.312559 17605 raft_consensus.cc:2932] T 59c288e7279f4119afb6c3bf860e6dc3 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:01.316983 17864 consensus_queue.cc:237] T 59c288e7279f4119afb6c3bf860e6dc3 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } }
I20240502 01:54:01.318559 17605 raft_consensus.cc:2932] T b39b88755cd74b0294d264215aea99cf P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:01.322779 17334 catalog_manager.cc:5484] T b39b88755cd74b0294d264215aea99cf P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: config changed from index 4 to 5, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:01.324368 17606 raft_consensus.cc:1261] T 59c288e7279f4119afb6c3bf860e6dc3 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:01.324764 20496 raft_consensus.cc:2932] T b39b88755cd74b0294d264215aea99cf P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:01.326912 20231 raft_consensus.cc:1261] T 59c288e7279f4119afb6c3bf860e6dc3 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:01.327463 22527 consensus_queue.cc:1035] T 59c288e7279f4119afb6c3bf860e6dc3 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:01.328327 21667 consensus_queue.cc:1035] T 59c288e7279f4119afb6c3bf860e6dc3 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:01.330574 22746 raft_consensus.cc:374] T 2ea539ebb37f42f68c166242a06f8d2e P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:01.331758 21667 raft_consensus.cc:2932] T 59c288e7279f4119afb6c3bf860e6dc3 P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:01.332396 20231 raft_consensus.cc:2932] T 59c288e7279f4119afb6c3bf860e6dc3 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:01.335683 17322 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 59c288e7279f4119afb6c3bf860e6dc3 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:01.338395 17327 catalog_manager.cc:5484] T 59c288e7279f4119afb6c3bf860e6dc3 P bb31bceba76d466db24b9341c04900c3 reported cstate change: config changed from index 5 to 6, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:01.340298 22745 tablet_copy_client.cc:308] T aebde87a7f114537abd9c1d8b7e97004 P dfa1152987644bd781b46c7e42a7a685: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.71:38623
I20240502 01:54:01.331238 22746 raft_consensus.cc:729] T 2ea539ebb37f42f68c166242a06f8d2e P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:54:01.342607 20518 tablet_copy_service.cc:140] P 2f306549407b4bb281ccf2994035e4f1: Received BeginTabletCopySession request for tablet aebde87a7f114537abd9c1d8b7e97004 from peer dfa1152987644bd781b46c7e42a7a685 ({username='slave'} at 127.11.42.67:59949)
I20240502 01:54:01.342774 20518 tablet_copy_service.cc:161] P 2f306549407b4bb281ccf2994035e4f1: Beginning new tablet copy session on tablet aebde87a7f114537abd9c1d8b7e97004 from peer dfa1152987644bd781b46c7e42a7a685 at {username='slave'} at 127.11.42.67:59949: session id = dfa1152987644bd781b46c7e42a7a685-aebde87a7f114537abd9c1d8b7e97004
I20240502 01:54:01.343349 17716 tablet_service.cc:1514] Processing DeleteTablet for tablet 59c288e7279f4119afb6c3bf860e6dc3 with delete_type TABLET_DATA_TOMBSTONED (TS dfa1152987644bd781b46c7e42a7a685 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52198
I20240502 01:54:01.343976 22775 tablet_replica.cc:330] T 59c288e7279f4119afb6c3bf860e6dc3 P dfa1152987644bd781b46c7e42a7a685: stopping tablet replica
I20240502 01:54:01.344244 22775 raft_consensus.cc:2229] T 59c288e7279f4119afb6c3bf860e6dc3 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:01.344774 22775 pending_rounds.cc:62] T 59c288e7279f4119afb6c3bf860e6dc3 P dfa1152987644bd781b46c7e42a7a685: Trying to abort 1 pending ops.
I20240502 01:54:01.345717 20518 tablet_copy_source_session.cc:205] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:01.345614 17605 raft_consensus.cc:2932] T 59c288e7279f4119afb6c3bf860e6dc3 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:01.342792 17474 consensus_queue.cc:237] T b39b88755cd74b0294d264215aea99cf P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } }
I20240502 01:54:01.344893 22775 pending_rounds.cc:68] T 59c288e7279f4119afb6c3bf860e6dc3 P dfa1152987644bd781b46c7e42a7a685: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023062389775867904 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "59c288e7279f4119afb6c3bf860e6dc3" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } } }
I20240502 01:54:01.349359 22775 raft_consensus.cc:2894] T 59c288e7279f4119afb6c3bf860e6dc3 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240502 01:54:01.349494 22775 raft_consensus.cc:2866] T 59c288e7279f4119afb6c3bf860e6dc3 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 01:54:01.349566 17737 raft_consensus.cc:1261] T b39b88755cd74b0294d264215aea99cf P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:01.349624 22775 raft_consensus.cc:2258] T 59c288e7279f4119afb6c3bf860e6dc3 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:01.350934 22487 consensus_queue.cc:1035] T b39b88755cd74b0294d264215aea99cf P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:01.351646 17734 raft_consensus.cc:2932] T b39b88755cd74b0294d264215aea99cf P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
W20240502 01:54:01.353394 22770 consensus_queue.cc:1159] T 59c288e7279f4119afb6c3bf860e6dc3 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "dfa1152987644bd781b46c7e42a7a685" status { last_received { term: 1 index: 5 } last_committed_idx: 4 last_received_current_leader { term: 1 index: 5 } }
I20240502 01:54:01.347087 22746 consensus_queue.cc:260] T 2ea539ebb37f42f68c166242a06f8d2e P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:01.355278 22746 ts_tablet_manager.cc:1398] T 2ea539ebb37f42f68c166242a06f8d2e P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.131s user 0.000s sys 0.002s
I20240502 01:54:01.354519 22745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aebde87a7f114537abd9c1d8b7e97004. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:01.357308 22775 ts_tablet_manager.cc:1875] T 59c288e7279f4119afb6c3bf860e6dc3 P dfa1152987644bd781b46c7e42a7a685: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:01.362223 22583 raft_consensus.cc:482] T 3e657454f8744d5d9b9a271505acbaf0 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:01.361961 22487 raft_consensus.cc:2932] T b39b88755cd74b0294d264215aea99cf P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:01.362383 22583 raft_consensus.cc:504] T 3e657454f8744d5d9b9a271505acbaf0 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:01.362643 17737 raft_consensus.cc:2932] T b39b88755cd74b0294d264215aea99cf P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:01.363029 22583 leader_election.cc:288] T 3e657454f8744d5d9b9a271505acbaf0 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:54:01.364045 20231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e657454f8744d5d9b9a271505acbaf0" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:54:01.364076 20629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e657454f8744d5d9b9a271505acbaf0" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091" is_pre_election: true
I20240502 01:54:01.364266 20231 raft_consensus.cc:2454] T 3e657454f8744d5d9b9a271505acbaf0 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:54:01.364681 17543 leader_election.cc:302] T 3e657454f8744d5d9b9a271505acbaf0 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:54:01.365697 22583 raft_consensus.cc:2781] T 3e657454f8744d5d9b9a271505acbaf0 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:01.365911 22583 raft_consensus.cc:482] T 3e657454f8744d5d9b9a271505acbaf0 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:01.366070 22583 raft_consensus.cc:3037] T 3e657454f8744d5d9b9a271505acbaf0 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:01.368119 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b39b88755cd74b0294d264215aea99cf with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:01.368268 17334 catalog_manager.cc:5484] T b39b88755cd74b0294d264215aea99cf P dfa1152987644bd781b46c7e42a7a685 reported cstate change: config changed from index 5 to 6, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New cstate: current_term: 1 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:01.369761 22583 raft_consensus.cc:504] T 3e657454f8744d5d9b9a271505acbaf0 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:01.370610 22583 leader_election.cc:288] T 3e657454f8744d5d9b9a271505acbaf0 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Requested vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:54:01.370759 20231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e657454f8744d5d9b9a271505acbaf0" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:54:01.370940 20231 raft_consensus.cc:3037] T 3e657454f8744d5d9b9a271505acbaf0 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:01.371273 20630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e657454f8744d5d9b9a271505acbaf0" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091"
I20240502 01:54:01.370823 20629 raft_consensus.cc:2454] T 3e657454f8744d5d9b9a271505acbaf0 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:54:01.373489 22745 tablet_copy_client.cc:783] T aebde87a7f114537abd9c1d8b7e97004 P dfa1152987644bd781b46c7e42a7a685: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:01.373512 20231 raft_consensus.cc:2454] T 3e657454f8744d5d9b9a271505acbaf0 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:54:01.373852 20494 raft_consensus.cc:1261] T b39b88755cd74b0294d264215aea99cf P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:01.374751 17543 leader_election.cc:302] T 3e657454f8744d5d9b9a271505acbaf0 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, fdc2ac372492454c85387589a91f63af; no voters: f19cc6cb1e76460aa6711e49a5775091
I20240502 01:54:01.374845 22745 tablet_copy_client.cc:647] T aebde87a7f114537abd9c1d8b7e97004 P dfa1152987644bd781b46c7e42a7a685: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:01.375208 17587 tablet_service.cc:1514] Processing DeleteTablet for tablet b39b88755cd74b0294d264215aea99cf with delete_type TABLET_DATA_TOMBSTONED (TS 3fceccefba274b1dbbd8216af8b0bc29 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55232
I20240502 01:54:01.375270 22583 raft_consensus.cc:2781] T 3e657454f8744d5d9b9a271505acbaf0 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:01.376365 22583 raft_consensus.cc:686] T 3e657454f8744d5d9b9a271505acbaf0 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Becoming Leader. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Running, Role: LEADER
I20240502 01:54:01.376761 22583 consensus_queue.cc:237] T 3e657454f8744d5d9b9a271505acbaf0 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:01.376907 22487 consensus_queue.cc:1035] T b39b88755cd74b0294d264215aea99cf P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:01.379802 22766 tablet_copy_client.cc:308] T 0d9f168de0d1410da2518abf936035ff P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.67:36115
I20240502 01:54:01.380088 22781 tablet_replica.cc:330] T b39b88755cd74b0294d264215aea99cf P 3fceccefba274b1dbbd8216af8b0bc29: stopping tablet replica
I20240502 01:54:01.380331 22781 raft_consensus.cc:2229] T b39b88755cd74b0294d264215aea99cf P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:01.380722 22781 raft_consensus.cc:2258] T b39b88755cd74b0294d264215aea99cf P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:01.381700 22781 ts_tablet_manager.cc:1875] T b39b88755cd74b0294d264215aea99cf P 3fceccefba274b1dbbd8216af8b0bc29: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:01.381937 17757 tablet_copy_service.cc:140] P dfa1152987644bd781b46c7e42a7a685: Received BeginTabletCopySession request for tablet 0d9f168de0d1410da2518abf936035ff from peer f19cc6cb1e76460aa6711e49a5775091 ({username='slave'} at 127.11.42.72:53665)
I20240502 01:54:01.382109 17757 tablet_copy_service.cc:161] P dfa1152987644bd781b46c7e42a7a685: Beginning new tablet copy session on tablet 0d9f168de0d1410da2518abf936035ff from peer f19cc6cb1e76460aa6711e49a5775091 at {username='slave'} at 127.11.42.72:53665: session id = f19cc6cb1e76460aa6711e49a5775091-0d9f168de0d1410da2518abf936035ff
I20240502 01:54:01.383116 22745 tablet_copy_client.cc:523] T aebde87a7f114537abd9c1d8b7e97004 P dfa1152987644bd781b46c7e42a7a685: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:01.383926 22626 raft_consensus.cc:482] T 2ea539ebb37f42f68c166242a06f8d2e P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:01.384052 22626 raft_consensus.cc:504] T 2ea539ebb37f42f68c166242a06f8d2e P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:01.384613 17757 tablet_copy_source_session.cc:205] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:01.384673 22626 leader_election.cc:288] T 2ea539ebb37f42f68c166242a06f8d2e P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:54:01.385104 17474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea539ebb37f42f68c166242a06f8d2e" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:54:01.385319 17474 raft_consensus.cc:2454] T 2ea539ebb37f42f68c166242a06f8d2e P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 0.
I20240502 01:54:01.385895 20433 leader_election.cc:302] T 2ea539ebb37f42f68c166242a06f8d2e P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 4da690c0fa13407a85c0689ce8de9ade; no voters:
I20240502 01:54:01.386246 22626 raft_consensus.cc:2781] T 2ea539ebb37f42f68c166242a06f8d2e P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:01.386358 22626 raft_consensus.cc:482] T 2ea539ebb37f42f68c166242a06f8d2e P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:01.386445 22626 raft_consensus.cc:3037] T 2ea539ebb37f42f68c166242a06f8d2e P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:01.388844 22626 raft_consensus.cc:504] T 2ea539ebb37f42f68c166242a06f8d2e P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:01.389573 22781 ts_tablet_manager.cc:1888] T b39b88755cd74b0294d264215aea99cf P 3fceccefba274b1dbbd8216af8b0bc29: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:01.389729 22626 leader_election.cc:288] T 2ea539ebb37f42f68c166242a06f8d2e P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 election: Requested vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:54:01.390192 17474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea539ebb37f42f68c166242a06f8d2e" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:54:01.390480 17474 raft_consensus.cc:3037] T 2ea539ebb37f42f68c166242a06f8d2e P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:01.391346 20630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea539ebb37f42f68c166242a06f8d2e" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091"
I20240502 01:54:01.391655 20630 raft_consensus.cc:3037] T 2ea539ebb37f42f68c166242a06f8d2e P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:01.392727 22775 ts_tablet_manager.cc:1888] T 59c288e7279f4119afb6c3bf860e6dc3 P dfa1152987644bd781b46c7e42a7a685: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:01.392850 22775 log.cc:1196] T 59c288e7279f4119afb6c3bf860e6dc3 P dfa1152987644bd781b46c7e42a7a685: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/wals/59c288e7279f4119afb6c3bf860e6dc3
I20240502 01:54:01.393695 22766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d9f168de0d1410da2518abf936035ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:01.394556 20630 raft_consensus.cc:2454] T 2ea539ebb37f42f68c166242a06f8d2e P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 1.
I20240502 01:54:01.398957 22781 log.cc:1196] T b39b88755cd74b0294d264215aea99cf P 3fceccefba274b1dbbd8216af8b0bc29: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/wals/b39b88755cd74b0294d264215aea99cf
I20240502 01:54:01.400115 17474 raft_consensus.cc:2454] T 2ea539ebb37f42f68c166242a06f8d2e P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 1.
I20240502 01:54:01.385923 20496 raft_consensus.cc:2932] T b39b88755cd74b0294d264215aea99cf P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:01.396919 20433 leader_election.cc:302] T 2ea539ebb37f42f68c166242a06f8d2e P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:54:01.401141 22626 raft_consensus.cc:2781] T 2ea539ebb37f42f68c166242a06f8d2e P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:01.401306 22626 raft_consensus.cc:686] T 2ea539ebb37f42f68c166242a06f8d2e P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Running, Role: LEADER
I20240502 01:54:01.401569 22626 consensus_queue.cc:237] T 2ea539ebb37f42f68c166242a06f8d2e P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:01.402556 17605 consensus_queue.cc:237] T 30f43088bd0f48ee86907df16194b9be P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } }
I20240502 01:54:01.404927 17737 raft_consensus.cc:1261] T 30f43088bd0f48ee86907df16194b9be P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:54:01.405084 17543 consensus_peers.cc:485] T 30f43088bd0f48ee86907df16194b9be P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 30f43088bd0f48ee86907df16194b9be. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:01.405612 22592 consensus_queue.cc:1035] T 30f43088bd0f48ee86907df16194b9be P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:01.407289 17474 raft_consensus.cc:1261] T 30f43088bd0f48ee86907df16194b9be P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:01.407583 17319 catalog_manager.cc:4830] TS 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): tablet b39b88755cd74b0294d264215aea99cf (table rebalance_test_table_9 [id=ffe6d29170f24ad48433354f548b9b2a]) successfully deleted
I20240502 01:54:01.408185 22592 consensus_queue.cc:1035] T 30f43088bd0f48ee86907df16194b9be P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:01.410051 22692 raft_consensus.cc:1050] T 5152c002eadd4d54ab10a093bce45f17 P 3fceccefba274b1dbbd8216af8b0bc29: attempting to promote NON_VOTER fdc2ac372492454c85387589a91f63af to VOTER
I20240502 01:54:01.410924 22692 consensus_queue.cc:237] T 5152c002eadd4d54ab10a093bce45f17 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } }
I20240502 01:54:01.411612 17321 catalog_manager.cc:4830] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): tablet 59c288e7279f4119afb6c3bf860e6dc3 (table rebalance_test_table_19 [id=b69eb497d10340fea8df98ad03a1e465]) successfully deleted
I20240502 01:54:01.417093 17864 raft_consensus.cc:1261] T 5152c002eadd4d54ab10a093bce45f17 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:01.417654 22766 tablet_copy_client.cc:783] T 0d9f168de0d1410da2518abf936035ff P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:01.417925 22766 tablet_copy_client.cc:647] T 0d9f168de0d1410da2518abf936035ff P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:01.417963 20231 raft_consensus.cc:1261] T 5152c002eadd4d54ab10a093bce45f17 P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:01.418921 17475 raft_consensus.cc:1261] T 5152c002eadd4d54ab10a093bce45f17 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:01.420432 22766 tablet_copy_client.cc:523] T 0d9f168de0d1410da2518abf936035ff P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:01.421396 22592 consensus_queue.cc:1035] T 5152c002eadd4d54ab10a093bce45f17 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:01.421797 21920 raft_consensus.cc:482] T aefcd4cee0724eeb866d76ae95bd0080 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:01.421931 22592 consensus_queue.cc:1035] T 5152c002eadd4d54ab10a093bce45f17 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:01.421957 21920 raft_consensus.cc:504] T aefcd4cee0724eeb866d76ae95bd0080 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:01.422376 22592 consensus_queue.cc:1035] T 5152c002eadd4d54ab10a093bce45f17 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:01.422634 21920 leader_election.cc:288] T aefcd4cee0724eeb866d76ae95bd0080 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:54:01.424357 20233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefcd4cee0724eeb866d76ae95bd0080" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:54:01.424587 20233 raft_consensus.cc:2454] T aefcd4cee0724eeb866d76ae95bd0080 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 0.
I20240502 01:54:01.425057 17412 leader_election.cc:302] T aefcd4cee0724eeb866d76ae95bd0080 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:54:01.425392 22487 raft_consensus.cc:2781] T aefcd4cee0724eeb866d76ae95bd0080 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:01.425472 22487 raft_consensus.cc:482] T aefcd4cee0724eeb866d76ae95bd0080 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:01.425562 22487 raft_consensus.cc:3037] T aefcd4cee0724eeb866d76ae95bd0080 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:01.428517 22487 raft_consensus.cc:504] T aefcd4cee0724eeb866d76ae95bd0080 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:01.429003 20630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefcd4cee0724eeb866d76ae95bd0080" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091" is_pre_election: true
I20240502 01:54:01.430366 22487 leader_election.cc:288] T aefcd4cee0724eeb866d76ae95bd0080 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Requested vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:54:01.429725 20628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefcd4cee0724eeb866d76ae95bd0080" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091"
I20240502 01:54:01.430449 20233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefcd4cee0724eeb866d76ae95bd0080" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:54:01.430624 20233 raft_consensus.cc:3037] T aefcd4cee0724eeb866d76ae95bd0080 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:01.430652 20628 raft_consensus.cc:3037] T aefcd4cee0724eeb866d76ae95bd0080 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:01.432816 22592 raft_consensus.cc:482] T 3dcd890bdd2d413e95401897e9f1bcab P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:01.432973 22592 raft_consensus.cc:504] T 3dcd890bdd2d413e95401897e9f1bcab P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:01.433236 20233 raft_consensus.cc:2454] T aefcd4cee0724eeb866d76ae95bd0080 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
I20240502 01:54:01.433692 17412 leader_election.cc:302] T aefcd4cee0724eeb866d76ae95bd0080 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:54:01.433816 20628 raft_consensus.cc:2454] T aefcd4cee0724eeb866d76ae95bd0080 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
I20240502 01:54:01.433967 22487 raft_consensus.cc:2781] T aefcd4cee0724eeb866d76ae95bd0080 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:01.434108 22487 raft_consensus.cc:686] T aefcd4cee0724eeb866d76ae95bd0080 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Becoming Leader. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Running, Role: LEADER
I20240502 01:54:01.434233 20494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcd890bdd2d413e95401897e9f1bcab" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
I20240502 01:54:01.434437 20494 raft_consensus.cc:2454] T 3dcd890bdd2d413e95401897e9f1bcab P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:54:01.434527 22487 consensus_queue.cc:237] T aefcd4cee0724eeb866d76ae95bd0080 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:01.434697 20630 raft_consensus.cc:2362] T aefcd4cee0724eeb866d76ae95bd0080 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1. Re-sending same reply.
I20240502 01:54:01.434908 22592 leader_election.cc:288] T 3dcd890bdd2d413e95401897e9f1bcab P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:54:01.435405 17543 leader_election.cc:302] T 3dcd890bdd2d413e95401897e9f1bcab P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 3fceccefba274b1dbbd8216af8b0bc29; no voters:
I20240502 01:54:01.435461 20366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcd890bdd2d413e95401897e9f1bcab" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:54:01.435693 20366 raft_consensus.cc:2454] T 3dcd890bdd2d413e95401897e9f1bcab P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:54:01.436033 22592 raft_consensus.cc:2781] T 3dcd890bdd2d413e95401897e9f1bcab P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:01.436120 22592 raft_consensus.cc:482] T 3dcd890bdd2d413e95401897e9f1bcab P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:01.436237 22592 raft_consensus.cc:3037] T 3dcd890bdd2d413e95401897e9f1bcab P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:01.438938 22592 raft_consensus.cc:504] T 3dcd890bdd2d413e95401897e9f1bcab P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:01.441211 20494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcd890bdd2d413e95401897e9f1bcab" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
I20240502 01:54:01.441407 20494 raft_consensus.cc:3037] T 3dcd890bdd2d413e95401897e9f1bcab P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:01.442958 22766 tablet_bootstrap.cc:492] T 0d9f168de0d1410da2518abf936035ff P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:54:01.443929 20494 raft_consensus.cc:2454] T 3dcd890bdd2d413e95401897e9f1bcab P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:54:01.444655 17543 leader_election.cc:302] T 3dcd890bdd2d413e95401897e9f1bcab P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 3fceccefba274b1dbbd8216af8b0bc29; no voters:
I20240502 01:54:01.446523 22527 raft_consensus.cc:1050] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3: attempting to promote NON_VOTER f19cc6cb1e76460aa6711e49a5775091 to VOTER
I20240502 01:54:01.445284 17474 raft_consensus.cc:2932] T 30f43088bd0f48ee86907df16194b9be P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:54:01.447324 20629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea539ebb37f42f68c166242a06f8d2e" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091" is_pre_election: true
I20240502 01:54:01.447415 22527 consensus_queue.cc:237] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } }
I20240502 01:54:01.447695 20629 raft_consensus.cc:2362] T 2ea539ebb37f42f68c166242a06f8d2e P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 1. Re-sending same reply.
I20240502 01:54:01.455299 22592 leader_election.cc:288] T 3dcd890bdd2d413e95401897e9f1bcab P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Requested vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:54:01.455756 20366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcd890bdd2d413e95401897e9f1bcab" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
I20240502 01:54:01.455978 20366 raft_consensus.cc:3037] T 3dcd890bdd2d413e95401897e9f1bcab P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:01.455540 22592 raft_consensus.cc:2932] T 30f43088bd0f48ee86907df16194b9be P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:54:01.458624 20366 raft_consensus.cc:2454] T 3dcd890bdd2d413e95401897e9f1bcab P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:54:01.461920 20629 raft_consensus.cc:1261] T 7c697210a14d4982a939ea7965cb769d P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:01.461850 17334 catalog_manager.cc:5484] T aefcd4cee0724eeb866d76ae95bd0080 P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: term changed from 0 to 1, leader changed from <none> to 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65). New cstate: current_term: 1 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:01.462432 22177 consensus_queue.cc:1035] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:01.464032 17605 raft_consensus.cc:1261] T 7c697210a14d4982a939ea7965cb769d P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:01.464572 22177 consensus_queue.cc:1035] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:01.468863 22799 raft_consensus.cc:2781] T 3dcd890bdd2d413e95401897e9f1bcab P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:01.472527 22745 tablet_bootstrap.cc:492] T aebde87a7f114537abd9c1d8b7e97004 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:54:01.477353 22799 raft_consensus.cc:686] T 3dcd890bdd2d413e95401897e9f1bcab P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Becoming Leader. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Running, Role: LEADER
I20240502 01:54:01.478245 22799 consensus_queue.cc:237] T 3dcd890bdd2d413e95401897e9f1bcab P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:01.484869 22177 raft_consensus.cc:2932] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:01.493093 17334 catalog_manager.cc:5484] T 2ea539ebb37f42f68c166242a06f8d2e P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71). New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:01.497061 22802 raft_consensus.cc:2932] T 5152c002eadd4d54ab10a093bce45f17 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:01.500735 17604 raft_consensus.cc:2932] T 7c697210a14d4982a939ea7965cb769d P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:01.501801 20233 raft_consensus.cc:2932] T 5152c002eadd4d54ab10a093bce45f17 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:01.508015 17330 catalog_manager.cc:5484] T 3e657454f8744d5d9b9a271505acbaf0 P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66). New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:01.509212 17735 raft_consensus.cc:1261] T 7c697210a14d4982a939ea7965cb769d P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:01.509300 22745 tablet_bootstrap.cc:492] T aebde87a7f114537abd9c1d8b7e97004 P dfa1152987644bd781b46c7e42a7a685: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:01.510736 22177 consensus_queue.cc:1035] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:01.511482 22745 tablet_bootstrap.cc:492] T aebde87a7f114537abd9c1d8b7e97004 P dfa1152987644bd781b46c7e42a7a685: Bootstrap complete.
I20240502 01:54:01.512672 22745 ts_tablet_manager.cc:1367] T aebde87a7f114537abd9c1d8b7e97004 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.040s user 0.007s sys 0.002s
I20240502 01:54:01.514431 17333 catalog_manager.cc:5484] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3 reported cstate change: config changed from index 4 to 5, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:01.515594 22745 raft_consensus.cc:348] T aebde87a7f114537abd9c1d8b7e97004 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:54:01.516407 22745 raft_consensus.cc:729] T aebde87a7f114537abd9c1d8b7e97004 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: LEARNER
I20240502 01:54:01.517036 22745 consensus_queue.cc:260] T aebde87a7f114537abd9c1d8b7e97004 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:54:01.518221 20629 raft_consensus.cc:2932] T 7c697210a14d4982a939ea7965cb769d P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:01.518777 17330 catalog_manager.cc:5484] T 5152c002eadd4d54ab10a093bce45f17 P fdc2ac372492454c85387589a91f63af reported cstate change: config changed from index 4 to 5, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:01.522109 22745 ts_tablet_manager.cc:1398] T aebde87a7f114537abd9c1d8b7e97004 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240502 01:54:01.524876 20518 tablet_copy_service.cc:342] P 2f306549407b4bb281ccf2994035e4f1: Request end of tablet copy session dfa1152987644bd781b46c7e42a7a685-aebde87a7f114537abd9c1d8b7e97004 received from {username='slave'} at 127.11.42.67:59949
I20240502 01:54:01.525094 20518 tablet_copy_service.cc:433] P 2f306549407b4bb281ccf2994035e4f1: ending tablet copy session dfa1152987644bd781b46c7e42a7a685-aebde87a7f114537abd9c1d8b7e97004 on tablet aebde87a7f114537abd9c1d8b7e97004 with peer dfa1152987644bd781b46c7e42a7a685
I20240502 01:54:01.526536 22766 tablet_bootstrap.cc:492] T 0d9f168de0d1410da2518abf936035ff P f19cc6cb1e76460aa6711e49a5775091: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:01.526451 17735 raft_consensus.cc:2932] T 7c697210a14d4982a939ea7965cb769d P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:01.527006 22766 tablet_bootstrap.cc:492] T 0d9f168de0d1410da2518abf936035ff P f19cc6cb1e76460aa6711e49a5775091: Bootstrap complete.
I20240502 01:54:01.527315 22766 ts_tablet_manager.cc:1367] T 0d9f168de0d1410da2518abf936035ff P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.084s user 0.043s sys 0.006s
I20240502 01:54:01.528133 22766 raft_consensus.cc:348] T 0d9f168de0d1410da2518abf936035ff P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:54:01.528395 22766 raft_consensus.cc:729] T 0d9f168de0d1410da2518abf936035ff P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: LEARNER
I20240502 01:54:01.528599 22766 consensus_queue.cc:260] T 0d9f168de0d1410da2518abf936035ff P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:54:01.529392 22766 ts_tablet_manager.cc:1398] T 0d9f168de0d1410da2518abf936035ff P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:01.530068 17757 tablet_copy_service.cc:342] P dfa1152987644bd781b46c7e42a7a685: Request end of tablet copy session f19cc6cb1e76460aa6711e49a5775091-0d9f168de0d1410da2518abf936035ff received from {username='slave'} at 127.11.42.72:53665
I20240502 01:54:01.530190 17757 tablet_copy_service.cc:433] P dfa1152987644bd781b46c7e42a7a685: ending tablet copy session f19cc6cb1e76460aa6711e49a5775091-0d9f168de0d1410da2518abf936035ff on tablet 0d9f168de0d1410da2518abf936035ff with peer f19cc6cb1e76460aa6711e49a5775091
I20240502 01:54:01.532735 17737 raft_consensus.cc:2932] T 30f43088bd0f48ee86907df16194b9be P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:54:01.533370 17864 raft_consensus.cc:2932] T 5152c002eadd4d54ab10a093bce45f17 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:01.543022 17330 catalog_manager.cc:5484] T 30f43088bd0f48ee86907df16194b9be P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: config changed from index -1 to 4, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:54:01.544659 17474 raft_consensus.cc:2932] T 5152c002eadd4d54ab10a093bce45f17 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:01.546216 17333 catalog_manager.cc:5484] T 3dcd890bdd2d413e95401897e9f1bcab P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66). New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:01.552333 17604 consensus_queue.cc:237] T 5152c002eadd4d54ab10a093bce45f17 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } }
I20240502 01:54:01.555313 20231 raft_consensus.cc:1261] T 5152c002eadd4d54ab10a093bce45f17 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:01.557080 22792 consensus_queue.cc:1035] T 5152c002eadd4d54ab10a093bce45f17 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:01.563361 22692 raft_consensus.cc:2932] T 5152c002eadd4d54ab10a093bce45f17 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:01.568130 17475 raft_consensus.cc:1261] T 5152c002eadd4d54ab10a093bce45f17 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:01.568058 20231 raft_consensus.cc:2932] T 5152c002eadd4d54ab10a093bce45f17 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:01.568962 22788 consensus_queue.cc:1035] T 5152c002eadd4d54ab10a093bce45f17 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:01.569980 17319 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5152c002eadd4d54ab10a093bce45f17 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:01.571146 17475 raft_consensus.cc:2932] T 5152c002eadd4d54ab10a093bce45f17 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:01.572139 17330 catalog_manager.cc:5484] T 5152c002eadd4d54ab10a093bce45f17 P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: config changed from index 5 to 6, VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) evicted. New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:01.576105 17845 tablet_service.cc:1514] Processing DeleteTablet for tablet 5152c002eadd4d54ab10a093bce45f17 with delete_type TABLET_DATA_TOMBSTONED (TS bb31bceba76d466db24b9341c04900c3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47226
I20240502 01:54:01.576938 22813 tablet_replica.cc:330] T 5152c002eadd4d54ab10a093bce45f17 P bb31bceba76d466db24b9341c04900c3: stopping tablet replica
I20240502 01:54:01.577271 22813 raft_consensus.cc:2229] T 5152c002eadd4d54ab10a093bce45f17 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:01.577702 22813 raft_consensus.cc:2258] T 5152c002eadd4d54ab10a093bce45f17 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:01.588424 17735 raft_consensus.cc:1203] T aebde87a7f114537abd9c1d8b7e97004 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:01.595826 22813 ts_tablet_manager.cc:1875] T 5152c002eadd4d54ab10a093bce45f17 P bb31bceba76d466db24b9341c04900c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:01.595955 17735 consensus_queue.cc:237] T 4099f4b2ae164020839faab53957750f P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
W20240502 01:54:01.601090 17672 consensus_peers.cc:485] T 4099f4b2ae164020839faab53957750f P dfa1152987644bd781b46c7e42a7a685 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4099f4b2ae164020839faab53957750f. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:01.601603 17474 raft_consensus.cc:1261] T 4099f4b2ae164020839faab53957750f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:01.601686 17604 raft_consensus.cc:1261] T 4099f4b2ae164020839faab53957750f P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:01.603318 22785 consensus_queue.cc:1035] T 4099f4b2ae164020839faab53957750f P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:01.605628 22568 consensus_queue.cc:1035] T 4099f4b2ae164020839faab53957750f P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:01.606207 22813 ts_tablet_manager.cc:1888] T 5152c002eadd4d54ab10a093bce45f17 P bb31bceba76d466db24b9341c04900c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:01.606401 22813 log.cc:1196] T 5152c002eadd4d54ab10a093bce45f17 P bb31bceba76d466db24b9341c04900c3: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/wals/5152c002eadd4d54ab10a093bce45f17
I20240502 01:54:01.613341 17322 catalog_manager.cc:4830] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): tablet 5152c002eadd4d54ab10a093bce45f17 (table rebalance_test_table_5 [id=1567095c8de34d6a98c56d3bb926b668]) successfully deleted
I20240502 01:54:01.616859 22785 raft_consensus.cc:2932] T 4099f4b2ae164020839faab53957750f P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:54:01.618067 17604 raft_consensus.cc:2932] T 4099f4b2ae164020839faab53957750f P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:54:01.618059 17473 raft_consensus.cc:2932] T 4099f4b2ae164020839faab53957750f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:54:01.622229 17330 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:54:01.627478 17333 catalog_manager.cc:5484] T 4099f4b2ae164020839faab53957750f P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: config changed from index -1 to 4, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:54:01.647517 20629 raft_consensus.cc:3037] T 3e657454f8744d5d9b9a271505acbaf0 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:01.648300 20231 raft_consensus.cc:1261] T 3e657454f8744d5d9b9a271505acbaf0 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:01.648864 22792 consensus_queue.cc:1035] T 3e657454f8744d5d9b9a271505acbaf0 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:01.650173 20629 raft_consensus.cc:1261] T 3e657454f8744d5d9b9a271505acbaf0 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:01.651300 22792 consensus_queue.cc:1035] T 3e657454f8744d5d9b9a271505acbaf0 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:01.656494 17334 catalog_manager.cc:5576] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411) does not have the latest schema for tablet 3e657454f8744d5d9b9a271505acbaf0 (table rebalancer_dynamic_table [id=b8199e32355e4fafa1cb6c61fa02e838]). Expected version 1 got 0
I20240502 01:54:01.656414 22828 tablet.cc:1722] T 3e657454f8744d5d9b9a271505acbaf0 P 3fceccefba274b1dbbd8216af8b0bc29: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:01.657470 22828 tablet.cc:1620] T 3e657454f8744d5d9b9a271505acbaf0 P 3fceccefba274b1dbbd8216af8b0bc29: MemRowSet was empty: no flush needed.
I20240502 01:54:01.661981 17864 consensus_queue.cc:237] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
I20240502 01:54:01.664531 17735 raft_consensus.cc:1261] T 377f8b554ad1448d981d2283b865f8fe P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:01.664531 17334 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=b8199e32355e4fafa1cb6c61fa02e838] alter complete (version 1)
I20240502 01:54:01.665411 17604 raft_consensus.cc:1261] T 377f8b554ad1448d981d2283b865f8fe P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:01.665481 21667 consensus_queue.cc:1035] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:54:01.666610 17803 consensus_peers.cc:485] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 -> Peer 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): Couldn't send request to peer 2f306549407b4bb281ccf2994035e4f1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 377f8b554ad1448d981d2283b865f8fe. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:01.667061 21667 consensus_queue.cc:1035] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:01.675508 22829 tablet.cc:1722] T 3e657454f8744d5d9b9a271505acbaf0 P f19cc6cb1e76460aa6711e49a5775091: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:01.678491 17334 catalog_manager.cc:5576] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411) does not have the latest schema for tablet 3e657454f8744d5d9b9a271505acbaf0 (table rebalancer_dynamic_table [id=b8199e32355e4fafa1cb6c61fa02e838]). Expected version 1 got 0
I20240502 01:54:01.679162 22829 tablet.cc:1620] T 3e657454f8744d5d9b9a271505acbaf0 P f19cc6cb1e76460aa6711e49a5775091: MemRowSet was empty: no flush needed.
I20240502 01:54:01.681293 21667 raft_consensus.cc:2932] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:54:01.685501 17737 raft_consensus.cc:2932] T 377f8b554ad1448d981d2283b865f8fe P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:54:01.686482 17605 raft_consensus.cc:2932] T 377f8b554ad1448d981d2283b865f8fe P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:54:01.691645 17333 catalog_manager.cc:5484] T 377f8b554ad1448d981d2283b865f8fe P dfa1152987644bd781b46c7e42a7a685 reported cstate change: config changed from index -1 to 4, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:54:01.693106 17327 catalog_manager.cc:5576] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339) does not have the latest schema for tablet 3e657454f8744d5d9b9a271505acbaf0 (table rebalancer_dynamic_table [id=b8199e32355e4fafa1cb6c61fa02e838]). Expected version 1 got 0
I20240502 01:54:01.698654 22834 tablet.cc:1722] T 3e657454f8744d5d9b9a271505acbaf0 P fdc2ac372492454c85387589a91f63af: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:01.701189 17327 catalog_manager.cc:5576] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339) does not have the latest schema for tablet 3e657454f8744d5d9b9a271505acbaf0 (table rebalancer_dynamic_table [id=b8199e32355e4fafa1cb6c61fa02e838]). Expected version 1 got 0
I20240502 01:54:01.701537 22834 tablet.cc:1620] T 3e657454f8744d5d9b9a271505acbaf0 P fdc2ac372492454c85387589a91f63af: MemRowSet was empty: no flush needed.
I20240502 01:54:01.720621 17864 consensus_queue.cc:237] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } }
W20240502 01:54:01.730403 17803 consensus_peers.cc:485] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ed1522ec150a46f28fd553f9b563651f. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:01.731212 17473 raft_consensus.cc:1261] T ed1522ec150a46f28fd553f9b563651f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:01.731900 22527 consensus_queue.cc:1035] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:01.732107 17735 raft_consensus.cc:1261] T ed1522ec150a46f28fd553f9b563651f P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:01.733738 22527 consensus_queue.cc:1035] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:01.735142 22177 raft_consensus.cc:2932] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:54:01.735811 17473 raft_consensus.cc:2932] T ed1522ec150a46f28fd553f9b563651f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:54:01.742300 17327 catalog_manager.cc:5484] T ed1522ec150a46f28fd553f9b563651f P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: config changed from index -1 to 4, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:54:01.745782 20630 raft_consensus.cc:1203] T 0d9f168de0d1410da2518abf936035ff P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:01.759690 17735 raft_consensus.cc:2932] T ed1522ec150a46f28fd553f9b563651f P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:54:01.771366 17327 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:54:01.782403 22828 tablet.cc:1722] T 3e657454f8744d5d9b9a271505acbaf0 P 3fceccefba274b1dbbd8216af8b0bc29: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:01.782876 22828 tablet.cc:1620] T 3e657454f8744d5d9b9a271505acbaf0 P 3fceccefba274b1dbbd8216af8b0bc29: MemRowSet was empty: no flush needed.
I20240502 01:54:01.785390 22834 tablet.cc:1722] T 3e657454f8744d5d9b9a271505acbaf0 P fdc2ac372492454c85387589a91f63af: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:01.785868 22834 tablet.cc:1620] T 3e657454f8744d5d9b9a271505acbaf0 P fdc2ac372492454c85387589a91f63af: MemRowSet was empty: no flush needed.
I20240502 01:54:01.786612 22829 tablet.cc:1722] T 3e657454f8744d5d9b9a271505acbaf0 P f19cc6cb1e76460aa6711e49a5775091: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:01.787034 22829 tablet.cc:1620] T 3e657454f8744d5d9b9a271505acbaf0 P f19cc6cb1e76460aa6711e49a5775091: MemRowSet was empty: no flush needed.
I20240502 01:54:01.798401 17334 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=b8199e32355e4fafa1cb6c61fa02e838] alter complete (version 2)
I20240502 01:54:01.813086 22814 consensus_queue.cc:1035] T 2ea539ebb37f42f68c166242a06f8d2e P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:01.834326 22393 consensus_queue.cc:1035] T 2ea539ebb37f42f68c166242a06f8d2e P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:01.840822 22465 raft_consensus.cc:1050] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9: attempting to promote NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 to VOTER
I20240502 01:54:01.841640 22465 consensus_queue.cc:237] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } }
I20240502 01:54:01.844082 20231 raft_consensus.cc:1261] T ba1e82a39c2e439889da60007372a99b P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:01.844271 17604 raft_consensus.cc:1261] T ba1e82a39c2e439889da60007372a99b P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEARNER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:01.844718 22863 consensus_queue.cc:1035] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:01.845958 22156 consensus_queue.cc:1035] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:01.847965 20630 raft_consensus.cc:1261] T ba1e82a39c2e439889da60007372a99b P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:01.848696 22465 consensus_queue.cc:1035] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:01.859663 22465 raft_consensus.cc:2932] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:01.861168 17605 raft_consensus.cc:2932] T ba1e82a39c2e439889da60007372a99b P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:01.864285 20231 raft_consensus.cc:2932] T ba1e82a39c2e439889da60007372a99b P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:01.866266 20630 raft_consensus.cc:2932] T ba1e82a39c2e439889da60007372a99b P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:01.867055 17327 catalog_manager.cc:5484] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:01.875329 20366 consensus_queue.cc:237] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } }
I20240502 01:54:01.877188 17605 raft_consensus.cc:1261] T ba1e82a39c2e439889da60007372a99b P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:01.877259 20231 raft_consensus.cc:1261] T ba1e82a39c2e439889da60007372a99b P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:01.878172 22465 consensus_queue.cc:1035] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:01.879092 22156 consensus_queue.cc:1035] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:01.881042 22465 raft_consensus.cc:2932] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:01.881223 17604 raft_consensus.cc:2932] T ba1e82a39c2e439889da60007372a99b P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:01.885756 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ba1e82a39c2e439889da60007372a99b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:01.889334 17327 catalog_manager.cc:5484] T ba1e82a39c2e439889da60007372a99b P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: config changed from index 3 to 4, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:01.893249 20610 tablet_service.cc:1514] Processing DeleteTablet for tablet ba1e82a39c2e439889da60007372a99b with delete_type TABLET_DATA_TOMBSTONED (TS f19cc6cb1e76460aa6711e49a5775091 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33698
I20240502 01:54:01.894065 20231 raft_consensus.cc:2932] T ba1e82a39c2e439889da60007372a99b P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:01.893929 22877 tablet_replica.cc:330] T ba1e82a39c2e439889da60007372a99b P f19cc6cb1e76460aa6711e49a5775091: stopping tablet replica
I20240502 01:54:01.896660 22877 raft_consensus.cc:2229] T ba1e82a39c2e439889da60007372a99b P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:01.897071 22877 raft_consensus.cc:2258] T ba1e82a39c2e439889da60007372a99b P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:01.898155 22877 ts_tablet_manager.cc:1875] T ba1e82a39c2e439889da60007372a99b P f19cc6cb1e76460aa6711e49a5775091: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:01.901585 17327 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:54:01.909047 22877 ts_tablet_manager.cc:1888] T ba1e82a39c2e439889da60007372a99b P f19cc6cb1e76460aa6711e49a5775091: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:54:01.909235 22877 log.cc:1196] T ba1e82a39c2e439889da60007372a99b P f19cc6cb1e76460aa6711e49a5775091: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal/wals/ba1e82a39c2e439889da60007372a99b
I20240502 01:54:01.910624 22828 tablet.cc:1722] T 3e657454f8744d5d9b9a271505acbaf0 P 3fceccefba274b1dbbd8216af8b0bc29: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:01.912921 22829 tablet.cc:1722] T 3e657454f8744d5d9b9a271505acbaf0 P f19cc6cb1e76460aa6711e49a5775091: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:01.915102 17321 catalog_manager.cc:4830] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): tablet ba1e82a39c2e439889da60007372a99b (table rebalancer_extra_table_2 [id=2444c341261d4eb29c484622291d0517]) successfully deleted
I20240502 01:54:01.918871 22834 tablet.cc:1722] T 3e657454f8744d5d9b9a271505acbaf0 P fdc2ac372492454c85387589a91f63af: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:01.927155 22692 consensus_queue.cc:1035] T 3dcd890bdd2d413e95401897e9f1bcab P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:01.927407 17327 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table_new_name [id=b8199e32355e4fafa1cb6c61fa02e838] alter complete (version 3)
I20240502 01:54:01.933286 22788 consensus_queue.cc:1035] T 3dcd890bdd2d413e95401897e9f1bcab P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:01.961891 21920 consensus_queue.cc:1035] T aefcd4cee0724eeb866d76ae95bd0080 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:01.966701 22487 consensus_queue.cc:1035] T aefcd4cee0724eeb866d76ae95bd0080 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:01.989481 22906 ts_tablet_manager.cc:902] T 30f43088bd0f48ee86907df16194b9be P fdc2ac372492454c85387589a91f63af: Initiating tablet copy from peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:54:02.010187 22906 tablet_copy_client.cc:308] T 30f43088bd0f48ee86907df16194b9be P fdc2ac372492454c85387589a91f63af: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.66:38531
I20240502 01:54:02.010840 17627 tablet_copy_service.cc:140] P 3fceccefba274b1dbbd8216af8b0bc29: Received BeginTabletCopySession request for tablet 30f43088bd0f48ee86907df16194b9be from peer fdc2ac372492454c85387589a91f63af ({username='slave'} at 127.11.42.69:59447)
I20240502 01:54:02.010993 17627 tablet_copy_service.cc:161] P 3fceccefba274b1dbbd8216af8b0bc29: Beginning new tablet copy session on tablet 30f43088bd0f48ee86907df16194b9be from peer fdc2ac372492454c85387589a91f63af at {username='slave'} at 127.11.42.69:59447: session id = fdc2ac372492454c85387589a91f63af-30f43088bd0f48ee86907df16194b9be
I20240502 01:54:02.012985 17627 tablet_copy_source_session.cc:205] T 30f43088bd0f48ee86907df16194b9be P 3fceccefba274b1dbbd8216af8b0bc29: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:02.015187 22906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30f43088bd0f48ee86907df16194b9be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:02.021888 22906 tablet_copy_client.cc:783] T 30f43088bd0f48ee86907df16194b9be P fdc2ac372492454c85387589a91f63af: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:02.022240 22906 tablet_copy_client.cc:647] T 30f43088bd0f48ee86907df16194b9be P fdc2ac372492454c85387589a91f63af: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:02.025981 22906 tablet_copy_client.cc:523] T 30f43088bd0f48ee86907df16194b9be P fdc2ac372492454c85387589a91f63af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:02.029103 22906 tablet_bootstrap.cc:492] T 30f43088bd0f48ee86907df16194b9be P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:54:02.031932 17334 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:02.032191 17334 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:02.035784 17334 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet 3e657454f8744d5d9b9a271505acbaf0
I20240502 01:54:02.036669 20610 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e657454f8744d5d9b9a271505acbaf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:02 UTC) from {username='slave'} at 127.0.0.1:33698
I20240502 01:54:02.037426 22877 tablet_replica.cc:330] T 3e657454f8744d5d9b9a271505acbaf0 P f19cc6cb1e76460aa6711e49a5775091: stopping tablet replica
I20240502 01:54:02.037762 20212 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e657454f8744d5d9b9a271505acbaf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:02 UTC) from {username='slave'} at 127.0.0.1:44624
I20240502 01:54:02.037860 17587 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e657454f8744d5d9b9a271505acbaf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:02 UTC) from {username='slave'} at 127.0.0.1:55232
I20240502 01:54:02.038257 22877 raft_consensus.cc:2229] T 3e657454f8744d5d9b9a271505acbaf0 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:02.038492 22877 raft_consensus.cc:2258] T 3e657454f8744d5d9b9a271505acbaf0 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:02.039515 22877 ts_tablet_manager.cc:1875] T 3e657454f8744d5d9b9a271505acbaf0 P f19cc6cb1e76460aa6711e49a5775091: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:02.047369 22877 ts_tablet_manager.cc:1888] T 3e657454f8744d5d9b9a271505acbaf0 P f19cc6cb1e76460aa6711e49a5775091: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:02.049674 22877 log.cc:1196] T 3e657454f8744d5d9b9a271505acbaf0 P f19cc6cb1e76460aa6711e49a5775091: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal/wals/3e657454f8744d5d9b9a271505acbaf0
I20240502 01:54:02.049728 22926 tablet_replica.cc:330] T 3e657454f8744d5d9b9a271505acbaf0 P 3fceccefba274b1dbbd8216af8b0bc29: stopping tablet replica
I20240502 01:54:02.050069 22926 raft_consensus.cc:2229] T 3e657454f8744d5d9b9a271505acbaf0 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:02.050190 22877 ts_tablet_manager.cc:1909] T 3e657454f8744d5d9b9a271505acbaf0 P f19cc6cb1e76460aa6711e49a5775091: Deleting consensus metadata
I20240502 01:54:02.050289 22925 tablet_replica.cc:330] T 3e657454f8744d5d9b9a271505acbaf0 P fdc2ac372492454c85387589a91f63af: stopping tablet replica
I20240502 01:54:02.051530 22926 raft_consensus.cc:2258] T 3e657454f8744d5d9b9a271505acbaf0 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:02.051667 17321 catalog_manager.cc:4830] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): tablet 3e657454f8744d5d9b9a271505acbaf0 (table rebalancer_dynamic_table_new_name [id=b8199e32355e4fafa1cb6c61fa02e838]) successfully deleted
I20240502 01:54:02.051769 22925 raft_consensus.cc:2229] T 3e657454f8744d5d9b9a271505acbaf0 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:02.051986 22925 raft_consensus.cc:2258] T 3e657454f8744d5d9b9a271505acbaf0 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:02.053066 22926 ts_tablet_manager.cc:1875] T 3e657454f8744d5d9b9a271505acbaf0 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:02.053921 22925 ts_tablet_manager.cc:1875] T 3e657454f8744d5d9b9a271505acbaf0 P fdc2ac372492454c85387589a91f63af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:02.059170 22925 ts_tablet_manager.cc:1888] T 3e657454f8744d5d9b9a271505acbaf0 P fdc2ac372492454c85387589a91f63af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:02.063607 22926 ts_tablet_manager.cc:1888] T 3e657454f8744d5d9b9a271505acbaf0 P 3fceccefba274b1dbbd8216af8b0bc29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:02.063722 22926 log.cc:1196] T 3e657454f8744d5d9b9a271505acbaf0 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/wals/3e657454f8744d5d9b9a271505acbaf0
I20240502 01:54:02.063961 22925 log.cc:1196] T 3e657454f8744d5d9b9a271505acbaf0 P fdc2ac372492454c85387589a91f63af: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal/wals/3e657454f8744d5d9b9a271505acbaf0
I20240502 01:54:02.064203 22926 ts_tablet_manager.cc:1909] T 3e657454f8744d5d9b9a271505acbaf0 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting consensus metadata
I20240502 01:54:02.064488 22925 ts_tablet_manager.cc:1909] T 3e657454f8744d5d9b9a271505acbaf0 P fdc2ac372492454c85387589a91f63af: Deleting consensus metadata
I20240502 01:54:02.065508 17319 catalog_manager.cc:4830] TS 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): tablet 3e657454f8744d5d9b9a271505acbaf0 (table rebalancer_dynamic_table_new_name [id=b8199e32355e4fafa1cb6c61fa02e838]) successfully deleted
I20240502 01:54:02.065688 17322 catalog_manager.cc:4830] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): tablet 3e657454f8744d5d9b9a271505acbaf0 (table rebalancer_dynamic_table_new_name [id=b8199e32355e4fafa1cb6c61fa02e838]) successfully deleted
I20240502 01:54:02.076350 22906 tablet_bootstrap.cc:492] T 30f43088bd0f48ee86907df16194b9be P fdc2ac372492454c85387589a91f63af: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:02.076797 22906 tablet_bootstrap.cc:492] T 30f43088bd0f48ee86907df16194b9be P fdc2ac372492454c85387589a91f63af: Bootstrap complete.
I20240502 01:54:02.077100 22906 ts_tablet_manager.cc:1367] T 30f43088bd0f48ee86907df16194b9be P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.048s user 0.044s sys 0.004s
I20240502 01:54:02.078084 22906 raft_consensus.cc:348] T 30f43088bd0f48ee86907df16194b9be P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } }
I20240502 01:54:02.078505 22906 raft_consensus.cc:729] T 30f43088bd0f48ee86907df16194b9be P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: LEARNER
I20240502 01:54:02.078799 22906 consensus_queue.cc:260] T 30f43088bd0f48ee86907df16194b9be P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } }
I20240502 01:54:02.079349 22906 ts_tablet_manager.cc:1398] T 30f43088bd0f48ee86907df16194b9be P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:54:02.080806 17627 tablet_copy_service.cc:342] P 3fceccefba274b1dbbd8216af8b0bc29: Request end of tablet copy session fdc2ac372492454c85387589a91f63af-30f43088bd0f48ee86907df16194b9be received from {username='slave'} at 127.11.42.69:59447
I20240502 01:54:02.080955 17627 tablet_copy_service.cc:433] P 3fceccefba274b1dbbd8216af8b0bc29: ending tablet copy session fdc2ac372492454c85387589a91f63af-30f43088bd0f48ee86907df16194b9be on tablet 30f43088bd0f48ee86907df16194b9be with peer fdc2ac372492454c85387589a91f63af
I20240502 01:54:02.086817 17734 consensus_queue.cc:237] T b29218a879e74895bfe00021ad358aa4 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
W20240502 01:54:02.089694 17673 consensus_peers.cc:485] T b29218a879e74895bfe00021ad358aa4 P dfa1152987644bd781b46c7e42a7a685 -> Peer 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): Couldn't send request to peer 2f306549407b4bb281ccf2994035e4f1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b29218a879e74895bfe00021ad358aa4. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:02.089676 17473 raft_consensus.cc:1261] T b29218a879e74895bfe00021ad358aa4 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:02.090269 17604 raft_consensus.cc:1261] T b29218a879e74895bfe00021ad358aa4 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:02.090255 22785 consensus_queue.cc:1035] T b29218a879e74895bfe00021ad358aa4 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:02.090855 22785 consensus_queue.cc:1035] T b29218a879e74895bfe00021ad358aa4 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:02.093924 22816 raft_consensus.cc:2932] T b29218a879e74895bfe00021ad358aa4 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:54:02.099967 17604 raft_consensus.cc:2932] T b29218a879e74895bfe00021ad358aa4 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:54:02.100844 17333 catalog_manager.cc:5484] T b29218a879e74895bfe00021ad358aa4 P dfa1152987644bd781b46c7e42a7a685 reported cstate change: config changed from index -1 to 4, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:02.101349 17473 raft_consensus.cc:2932] T b29218a879e74895bfe00021ad358aa4 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:54:02.103600 22932 ts_tablet_manager.cc:902] T 377f8b554ad1448d981d2283b865f8fe P 2f306549407b4bb281ccf2994035e4f1: Initiating tablet copy from peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:54:02.109278 17864 consensus_queue.cc:237] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } }
I20240502 01:54:02.111923 20630 raft_consensus.cc:1261] T 7c697210a14d4982a939ea7965cb769d P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:02.113421 17734 raft_consensus.cc:1261] T 7c697210a14d4982a939ea7965cb769d P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:02.113457 22177 consensus_queue.cc:1035] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:02.114353 22177 consensus_queue.cc:1035] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:02.116746 22527 raft_consensus.cc:2932] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:02.117633 20630 raft_consensus.cc:2932] T 7c697210a14d4982a939ea7965cb769d P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:02.119704 17322 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7c697210a14d4982a939ea7965cb769d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:02.121568 17327 catalog_manager.cc:5484] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3 reported cstate change: config changed from index 5 to 6, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:02.124295 17734 raft_consensus.cc:2932] T 7c697210a14d4982a939ea7965cb769d P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:02.125845 17587 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c697210a14d4982a939ea7965cb769d with delete_type TABLET_DATA_TOMBSTONED (TS 3fceccefba274b1dbbd8216af8b0bc29 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55232
I20240502 01:54:02.126173 22926 tablet_replica.cc:330] T 7c697210a14d4982a939ea7965cb769d P 3fceccefba274b1dbbd8216af8b0bc29: stopping tablet replica
I20240502 01:54:02.126401 22926 raft_consensus.cc:2229] T 7c697210a14d4982a939ea7965cb769d P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:02.126750 22926 raft_consensus.cc:2258] T 7c697210a14d4982a939ea7965cb769d P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:02.128172 22926 ts_tablet_manager.cc:1875] T 7c697210a14d4982a939ea7965cb769d P 3fceccefba274b1dbbd8216af8b0bc29: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:02.134959 22926 ts_tablet_manager.cc:1888] T 7c697210a14d4982a939ea7965cb769d P 3fceccefba274b1dbbd8216af8b0bc29: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:02.135166 22926 log.cc:1196] T 7c697210a14d4982a939ea7965cb769d P 3fceccefba274b1dbbd8216af8b0bc29: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/wals/7c697210a14d4982a939ea7965cb769d
I20240502 01:54:02.137075 17319 catalog_manager.cc:4830] TS 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): tablet 7c697210a14d4982a939ea7965cb769d (table rebalance_test_table_5 [id=1567095c8de34d6a98c56d3bb926b668]) successfully deleted
I20240502 01:54:02.139487 17327 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_extra_table_28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:02.141978 17333 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:54:02.142848 22943 ts_tablet_manager.cc:902] T 4099f4b2ae164020839faab53957750f P f19cc6cb1e76460aa6711e49a5775091: Initiating tablet copy from peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:54:02.151351 20212 tablet_service.cc:1467] Processing CreateTablet for tablet 7544a9b7ee6f4d2091b0b711076cf8d2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2fedd81ef2f345f8b9e5896b6b6bb642]), partition=
I20240502 01:54:02.151746 20212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7544a9b7ee6f4d2091b0b711076cf8d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:02.151901 17716 tablet_service.cc:1467] Processing CreateTablet for tablet 7544a9b7ee6f4d2091b0b711076cf8d2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2fedd81ef2f345f8b9e5896b6b6bb642]), partition=
I20240502 01:54:02.152067 20476 tablet_service.cc:1467] Processing CreateTablet for tablet 7544a9b7ee6f4d2091b0b711076cf8d2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2fedd81ef2f345f8b9e5896b6b6bb642]), partition=
I20240502 01:54:02.152272 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7544a9b7ee6f4d2091b0b711076cf8d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:02.152419 20476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7544a9b7ee6f4d2091b0b711076cf8d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:02.167007 22947 tablet_bootstrap.cc:492] T 7544a9b7ee6f4d2091b0b711076cf8d2 P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:54:02.169857 22947 tablet_bootstrap.cc:654] T 7544a9b7ee6f4d2091b0b711076cf8d2 P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:02.171337 22948 tablet_bootstrap.cc:492] T 7544a9b7ee6f4d2091b0b711076cf8d2 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:54:02.174472 22948 tablet_bootstrap.cc:654] T 7544a9b7ee6f4d2091b0b711076cf8d2 P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:02.182619 20212 tablet_service.cc:1467] Processing CreateTablet for tablet 68bc23294ac64c9486bffebbdb16b508 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=736004bf70524edd905a6bb90bb25cc0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:02.182684 20345 tablet_service.cc:1467] Processing CreateTablet for tablet 88e68acb7de346f0ad165cadabd077be (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=736004bf70524edd905a6bb90bb25cc0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:02.183106 20345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88e68acb7de346f0ad165cadabd077be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:02.183439 20610 tablet_service.cc:1467] Processing CreateTablet for tablet 68bc23294ac64c9486bffebbdb16b508 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=736004bf70524edd905a6bb90bb25cc0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:02.183684 20609 tablet_service.cc:1467] Processing CreateTablet for tablet 495700f8ed5e41228f2e597465dad126 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=736004bf70524edd905a6bb90bb25cc0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:02.183856 20610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68bc23294ac64c9486bffebbdb16b508. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:02.184152 20609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 495700f8ed5e41228f2e597465dad126. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:02.185810 17456 tablet_service.cc:1467] Processing CreateTablet for tablet 495700f8ed5e41228f2e597465dad126 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=736004bf70524edd905a6bb90bb25cc0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:02.186237 17456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 495700f8ed5e41228f2e597465dad126. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:02.189133 20212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68bc23294ac64c9486bffebbdb16b508. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:02.190111 20347 tablet_service.cc:1467] Processing CreateTablet for tablet 495700f8ed5e41228f2e597465dad126 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=736004bf70524edd905a6bb90bb25cc0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:02.190524 20347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 495700f8ed5e41228f2e597465dad126. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:02.198031 20478 tablet_service.cc:1467] Processing CreateTablet for tablet 88e68acb7de346f0ad165cadabd077be (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=736004bf70524edd905a6bb90bb25cc0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:02.198031 20477 tablet_service.cc:1467] Processing CreateTablet for tablet 68bc23294ac64c9486bffebbdb16b508 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=736004bf70524edd905a6bb90bb25cc0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:02.198545 20478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88e68acb7de346f0ad165cadabd077be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:02.198838 20477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68bc23294ac64c9486bffebbdb16b508. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:02.199213 20211 tablet_service.cc:1467] Processing CreateTablet for tablet 88e68acb7de346f0ad165cadabd077be (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=736004bf70524edd905a6bb90bb25cc0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:02.205341 22814 raft_consensus.cc:1050] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1: attempting to promote NON_VOTER dfa1152987644bd781b46c7e42a7a685 to VOTER
I20240502 01:54:02.206265 22814 consensus_queue.cc:237] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:02.211236 20211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88e68acb7de346f0ad165cadabd077be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:02.220628 22947 tablet_bootstrap.cc:492] T 7544a9b7ee6f4d2091b0b711076cf8d2 P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:54:02.220932 22947 ts_tablet_manager.cc:1367] T 7544a9b7ee6f4d2091b0b711076cf8d2 P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.054s user 0.002s sys 0.004s
I20240502 01:54:02.221772 22947 raft_consensus.cc:348] T 7544a9b7ee6f4d2091b0b711076cf8d2 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:02.222220 22947 raft_consensus.cc:374] T 7544a9b7ee6f4d2091b0b711076cf8d2 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:02.222388 22947 raft_consensus.cc:729] T 7544a9b7ee6f4d2091b0b711076cf8d2 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:54:02.222752 22947 consensus_queue.cc:260] T 7544a9b7ee6f4d2091b0b711076cf8d2 P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:02.224537 22947 ts_tablet_manager.cc:1398] T 7544a9b7ee6f4d2091b0b711076cf8d2 P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:54:02.224836 22947 tablet_bootstrap.cc:492] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:54:02.227895 22906 ts_tablet_manager.cc:902] T ed1522ec150a46f28fd553f9b563651f P fdc2ac372492454c85387589a91f63af: Initiating tablet copy from peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:54:02.229570 22948 tablet_bootstrap.cc:492] T 7544a9b7ee6f4d2091b0b711076cf8d2 P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:54:02.231256 22948 ts_tablet_manager.cc:1367] T 7544a9b7ee6f4d2091b0b711076cf8d2 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.060s user 0.003s sys 0.003s
I20240502 01:54:02.234834 22948 raft_consensus.cc:348] T 7544a9b7ee6f4d2091b0b711076cf8d2 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:02.236860 22948 raft_consensus.cc:374] T 7544a9b7ee6f4d2091b0b711076cf8d2 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:02.237371 22948 raft_consensus.cc:729] T 7544a9b7ee6f4d2091b0b711076cf8d2 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:54:02.239521 22948 consensus_queue.cc:260] T 7544a9b7ee6f4d2091b0b711076cf8d2 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:02.241403 22948 ts_tablet_manager.cc:1398] T 7544a9b7ee6f4d2091b0b711076cf8d2 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 01:54:02.246083 22947 tablet_bootstrap.cc:654] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:02.250771 17864 raft_consensus.cc:1261] T aebde87a7f114537abd9c1d8b7e97004 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:02.251413 22626 consensus_queue.cc:1035] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:02.254122 17734 raft_consensus.cc:1261] T aebde87a7f114537abd9c1d8b7e97004 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:02.254954 22626 consensus_queue.cc:1035] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:02.257645 20366 raft_consensus.cc:1261] T aebde87a7f114537abd9c1d8b7e97004 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:02.258569 22393 consensus_queue.cc:1035] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:02.261307 22626 raft_consensus.cc:2932] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:02.265062 22947 tablet_bootstrap.cc:492] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:54:02.266403 22947 ts_tablet_manager.cc:1367] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.042s user 0.005s sys 0.001s
I20240502 01:54:02.267838 22947 raft_consensus.cc:348] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:02.267627 17734 raft_consensus.cc:2932] T aebde87a7f114537abd9c1d8b7e97004 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:02.267004 22906 tablet_copy_client.cc:308] T ed1522ec150a46f28fd553f9b563651f P fdc2ac372492454c85387589a91f63af: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.68:44083
I20240502 01:54:02.271111 22947 raft_consensus.cc:374] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:02.271203 22947 raft_consensus.cc:729] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:54:02.271430 17887 tablet_copy_service.cc:140] P bb31bceba76d466db24b9341c04900c3: Received BeginTabletCopySession request for tablet ed1522ec150a46f28fd553f9b563651f from peer fdc2ac372492454c85387589a91f63af ({username='slave'} at 127.11.42.69:56011)
I20240502 01:54:02.271584 17887 tablet_copy_service.cc:161] P bb31bceba76d466db24b9341c04900c3: Beginning new tablet copy session on tablet ed1522ec150a46f28fd553f9b563651f from peer fdc2ac372492454c85387589a91f63af at {username='slave'} at 127.11.42.69:56011: session id = fdc2ac372492454c85387589a91f63af-ed1522ec150a46f28fd553f9b563651f
I20240502 01:54:02.271672 22955 tablet_bootstrap.cc:492] T 495700f8ed5e41228f2e597465dad126 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:54:02.272210 22947 consensus_queue.cc:260] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:02.272773 22947 ts_tablet_manager.cc:1398] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 01:54:02.274863 17887 tablet_copy_source_session.cc:205] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:02.274950 22955 tablet_bootstrap.cc:654] T 495700f8ed5e41228f2e597465dad126 P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:02.278154 22906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed1522ec150a46f28fd553f9b563651f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:02.281579 22956 tablet_bootstrap.cc:492] T 495700f8ed5e41228f2e597465dad126 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:54:02.284044 22956 tablet_bootstrap.cc:654] T 495700f8ed5e41228f2e597465dad126 P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:02.285868 22949 tablet_bootstrap.cc:492] T 7544a9b7ee6f4d2091b0b711076cf8d2 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:54:02.288272 17864 raft_consensus.cc:2932] T aebde87a7f114537abd9c1d8b7e97004 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:02.289036 22949 tablet_bootstrap.cc:654] T 7544a9b7ee6f4d2091b0b711076cf8d2 P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:02.290316 22952 tablet_bootstrap.cc:492] T 68bc23294ac64c9486bffebbdb16b508 P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:54:02.290499 17330 catalog_manager.cc:5484] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:02.292259 22952 tablet_bootstrap.cc:654] T 68bc23294ac64c9486bffebbdb16b508 P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:02.301775 22947 tablet_bootstrap.cc:492] T 68bc23294ac64c9486bffebbdb16b508 P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:54:02.304172 22947 tablet_bootstrap.cc:654] T 68bc23294ac64c9486bffebbdb16b508 P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:02.311106 22816 raft_consensus.cc:1050] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685: attempting to promote NON_VOTER f19cc6cb1e76460aa6711e49a5775091 to VOTER
I20240502 01:54:02.312000 22816 consensus_queue.cc:237] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } }
I20240502 01:54:02.322052 22952 tablet_bootstrap.cc:492] T 68bc23294ac64c9486bffebbdb16b508 P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:54:02.322185 22956 tablet_bootstrap.cc:492] T 495700f8ed5e41228f2e597465dad126 P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:54:02.322388 22956 ts_tablet_manager.cc:1367] T 495700f8ed5e41228f2e597465dad126 P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.041s user 0.002s sys 0.002s
I20240502 01:54:02.323248 22956 raft_consensus.cc:348] T 495700f8ed5e41228f2e597465dad126 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:02.323539 22956 raft_consensus.cc:374] T 495700f8ed5e41228f2e597465dad126 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:02.323669 22952 ts_tablet_manager.cc:1367] T 68bc23294ac64c9486bffebbdb16b508 P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.000s
I20240502 01:54:02.323772 22956 raft_consensus.cc:729] T 495700f8ed5e41228f2e597465dad126 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:54:02.324117 22956 consensus_queue.cc:260] T 495700f8ed5e41228f2e597465dad126 P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:02.324817 22952 raft_consensus.cc:348] T 68bc23294ac64c9486bffebbdb16b508 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:02.326994 22952 raft_consensus.cc:374] T 68bc23294ac64c9486bffebbdb16b508 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:02.327410 22952 raft_consensus.cc:729] T 68bc23294ac64c9486bffebbdb16b508 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:54:02.327983 22952 consensus_queue.cc:260] T 68bc23294ac64c9486bffebbdb16b508 P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:02.328596 22947 tablet_bootstrap.cc:492] T 68bc23294ac64c9486bffebbdb16b508 P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:54:02.328790 22947 ts_tablet_manager.cc:1367] T 68bc23294ac64c9486bffebbdb16b508 P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20240502 01:54:02.329550 22952 ts_tablet_manager.cc:1398] T 68bc23294ac64c9486bffebbdb16b508 P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:54:02.329572 22947 raft_consensus.cc:348] T 68bc23294ac64c9486bffebbdb16b508 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:02.329094 22943 tablet_copy_client.cc:308] T 4099f4b2ae164020839faab53957750f P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.67:36115
I20240502 01:54:02.330135 22952 tablet_bootstrap.cc:492] T 495700f8ed5e41228f2e597465dad126 P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:54:02.331861 22952 tablet_bootstrap.cc:654] T 495700f8ed5e41228f2e597465dad126 P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:02.340204 22947 raft_consensus.cc:374] T 68bc23294ac64c9486bffebbdb16b508 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:02.340449 22947 raft_consensus.cc:729] T 68bc23294ac64c9486bffebbdb16b508 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:54:02.340757 22947 consensus_queue.cc:260] T 68bc23294ac64c9486bffebbdb16b508 P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:02.343035 22947 ts_tablet_manager.cc:1398] T 68bc23294ac64c9486bffebbdb16b508 P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.014s user 0.000s sys 0.001s
I20240502 01:54:02.348266 22949 tablet_bootstrap.cc:492] T 7544a9b7ee6f4d2091b0b711076cf8d2 P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:54:02.348438 22949 ts_tablet_manager.cc:1367] T 7544a9b7ee6f4d2091b0b711076cf8d2 P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.063s user 0.002s sys 0.001s
I20240502 01:54:02.349315 22949 raft_consensus.cc:348] T 7544a9b7ee6f4d2091b0b711076cf8d2 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:02.349615 22949 raft_consensus.cc:374] T 7544a9b7ee6f4d2091b0b711076cf8d2 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:02.349783 22949 raft_consensus.cc:729] T 7544a9b7ee6f4d2091b0b711076cf8d2 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:54:02.350138 22949 consensus_queue.cc:260] T 7544a9b7ee6f4d2091b0b711076cf8d2 P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:02.351289 17757 tablet_copy_service.cc:140] P dfa1152987644bd781b46c7e42a7a685: Received BeginTabletCopySession request for tablet 4099f4b2ae164020839faab53957750f from peer f19cc6cb1e76460aa6711e49a5775091 ({username='slave'} at 127.11.42.72:53665)
I20240502 01:54:02.351447 17757 tablet_copy_service.cc:161] P dfa1152987644bd781b46c7e42a7a685: Beginning new tablet copy session on tablet 4099f4b2ae164020839faab53957750f from peer f19cc6cb1e76460aa6711e49a5775091 at {username='slave'} at 127.11.42.72:53665: session id = f19cc6cb1e76460aa6711e49a5775091-4099f4b2ae164020839faab53957750f
I20240502 01:54:02.351562 22949 ts_tablet_manager.cc:1398] T 7544a9b7ee6f4d2091b0b711076cf8d2 P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:54:02.353483 22956 ts_tablet_manager.cc:1398] T 495700f8ed5e41228f2e597465dad126 P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.031s user 0.000s sys 0.002s
I20240502 01:54:02.354262 17757 tablet_copy_source_session.cc:205] T 4099f4b2ae164020839faab53957750f P dfa1152987644bd781b46c7e42a7a685: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:02.358912 22949 tablet_bootstrap.cc:492] T 88e68acb7de346f0ad165cadabd077be P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:54:02.360868 22940 raft_consensus.cc:482] T 68bc23294ac64c9486bffebbdb16b508 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:02.361140 22940 raft_consensus.cc:504] T 68bc23294ac64c9486bffebbdb16b508 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:02.361883 22940 leader_election.cc:288] T 68bc23294ac64c9486bffebbdb16b508 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:54:02.362084 22949 tablet_bootstrap.cc:654] T 88e68acb7de346f0ad165cadabd077be P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:02.362660 20233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68bc23294ac64c9486bffebbdb16b508" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:54:02.362911 20233 raft_consensus.cc:2454] T 68bc23294ac64c9486bffebbdb16b508 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 0.
I20240502 01:54:02.362476 22932 tablet_copy_client.cc:308] T 377f8b554ad1448d981d2283b865f8fe P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.68:44083
I20240502 01:54:02.364460 20496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68bc23294ac64c9486bffebbdb16b508" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
W20240502 01:54:02.365057 20565 leader_election.cc:341] T 68bc23294ac64c9486bffebbdb16b508 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:02.365340 20565 leader_election.cc:302] T 68bc23294ac64c9486bffebbdb16b508 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f19cc6cb1e76460aa6711e49a5775091, fdc2ac372492454c85387589a91f63af; no voters: 2f306549407b4bb281ccf2994035e4f1
I20240502 01:54:02.365808 22940 raft_consensus.cc:2781] T 68bc23294ac64c9486bffebbdb16b508 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:02.365926 22940 raft_consensus.cc:482] T 68bc23294ac64c9486bffebbdb16b508 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:02.366020 22940 raft_consensus.cc:3037] T 68bc23294ac64c9486bffebbdb16b508 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:02.368860 22949 tablet_bootstrap.cc:492] T 88e68acb7de346f0ad165cadabd077be P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:54:02.368997 22949 ts_tablet_manager.cc:1367] T 88e68acb7de346f0ad165cadabd077be P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240502 01:54:02.369701 22949 raft_consensus.cc:348] T 88e68acb7de346f0ad165cadabd077be P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:02.369966 22949 raft_consensus.cc:374] T 88e68acb7de346f0ad165cadabd077be P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:02.370106 22949 raft_consensus.cc:729] T 88e68acb7de346f0ad165cadabd077be P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:54:02.370213 17887 tablet_copy_service.cc:140] P bb31bceba76d466db24b9341c04900c3: Received BeginTabletCopySession request for tablet 377f8b554ad1448d981d2283b865f8fe from peer 2f306549407b4bb281ccf2994035e4f1 ({username='slave'} at 127.11.42.71:48285)
I20240502 01:54:02.370398 22949 consensus_queue.cc:260] T 88e68acb7de346f0ad165cadabd077be P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:02.370606 17887 tablet_copy_service.cc:161] P bb31bceba76d466db24b9341c04900c3: Beginning new tablet copy session on tablet 377f8b554ad1448d981d2283b865f8fe from peer 2f306549407b4bb281ccf2994035e4f1 at {username='slave'} at 127.11.42.71:48285: session id = 2f306549407b4bb281ccf2994035e4f1-377f8b554ad1448d981d2283b865f8fe
I20240502 01:54:02.372150 22949 ts_tablet_manager.cc:1398] T 88e68acb7de346f0ad165cadabd077be P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 01:54:02.372467 22949 tablet_bootstrap.cc:492] T 68bc23294ac64c9486bffebbdb16b508 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:54:02.373958 17887 tablet_copy_source_session.cc:205] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:02.374074 22949 tablet_bootstrap.cc:654] T 68bc23294ac64c9486bffebbdb16b508 P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:02.374233 22952 tablet_bootstrap.cc:492] T 495700f8ed5e41228f2e597465dad126 P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:54:02.374411 22952 ts_tablet_manager.cc:1367] T 495700f8ed5e41228f2e597465dad126 P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.044s user 0.013s sys 0.006s
I20240502 01:54:02.375185 22952 raft_consensus.cc:348] T 495700f8ed5e41228f2e597465dad126 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:02.375422 22952 raft_consensus.cc:374] T 495700f8ed5e41228f2e597465dad126 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:02.375500 22952 raft_consensus.cc:729] T 495700f8ed5e41228f2e597465dad126 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:54:02.375725 22952 consensus_queue.cc:260] T 495700f8ed5e41228f2e597465dad126 P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:02.376240 22952 ts_tablet_manager.cc:1398] T 495700f8ed5e41228f2e597465dad126 P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 01:54:02.376267 22940 raft_consensus.cc:504] T 68bc23294ac64c9486bffebbdb16b508 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:02.377080 22940 leader_election.cc:288] T 68bc23294ac64c9486bffebbdb16b508 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 election: Requested vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:54:02.380808 20233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68bc23294ac64c9486bffebbdb16b508" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:54:02.381067 20233 raft_consensus.cc:3037] T 68bc23294ac64c9486bffebbdb16b508 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:02.382582 22949 tablet_bootstrap.cc:492] T 68bc23294ac64c9486bffebbdb16b508 P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:54:02.382712 22949 ts_tablet_manager.cc:1367] T 68bc23294ac64c9486bffebbdb16b508 P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20240502 01:54:02.383457 22949 raft_consensus.cc:348] T 68bc23294ac64c9486bffebbdb16b508 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:02.383713 22949 raft_consensus.cc:374] T 68bc23294ac64c9486bffebbdb16b508 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:02.383854 22949 raft_consensus.cc:729] T 68bc23294ac64c9486bffebbdb16b508 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:54:02.384104 22949 consensus_queue.cc:260] T 68bc23294ac64c9486bffebbdb16b508 P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:02.386653 20496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68bc23294ac64c9486bffebbdb16b508" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
I20240502 01:54:02.386893 20496 raft_consensus.cc:3037] T 68bc23294ac64c9486bffebbdb16b508 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:02.388012 22949 ts_tablet_manager.cc:1398] T 68bc23294ac64c9486bffebbdb16b508 P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 01:54:02.390491 20496 raft_consensus.cc:2454] T 68bc23294ac64c9486bffebbdb16b508 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 1.
I20240502 01:54:02.391745 22568 raft_consensus.cc:482] T 7544a9b7ee6f4d2091b0b711076cf8d2 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:02.391958 22568 raft_consensus.cc:504] T 7544a9b7ee6f4d2091b0b711076cf8d2 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:02.392724 22568 leader_election.cc:288] T 7544a9b7ee6f4d2091b0b711076cf8d2 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:54:02.393206 20565 leader_election.cc:302] T 68bc23294ac64c9486bffebbdb16b508 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:54:02.393546 22940 raft_consensus.cc:2781] T 68bc23294ac64c9486bffebbdb16b508 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:02.393751 20231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7544a9b7ee6f4d2091b0b711076cf8d2" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:54:02.393977 20231 raft_consensus.cc:2454] T 7544a9b7ee6f4d2091b0b711076cf8d2 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:54:02.394119 22940 raft_consensus.cc:686] T 68bc23294ac64c9486bffebbdb16b508 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Becoming Leader. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Running, Role: LEADER
I20240502 01:54:02.394456 22940 consensus_queue.cc:237] T 68bc23294ac64c9486bffebbdb16b508 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:02.394605 20496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7544a9b7ee6f4d2091b0b711076cf8d2" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
I20240502 01:54:02.394835 20496 raft_consensus.cc:2454] T 7544a9b7ee6f4d2091b0b711076cf8d2 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:54:02.395058 20233 raft_consensus.cc:2454] T 68bc23294ac64c9486bffebbdb16b508 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 1.
I20240502 01:54:02.396263 17673 leader_election.cc:302] T 7544a9b7ee6f4d2091b0b711076cf8d2 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:54:02.397063 22568 raft_consensus.cc:2781] T 7544a9b7ee6f4d2091b0b711076cf8d2 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:02.397215 22568 raft_consensus.cc:482] T 7544a9b7ee6f4d2091b0b711076cf8d2 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:02.397372 22568 raft_consensus.cc:3037] T 7544a9b7ee6f4d2091b0b711076cf8d2 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:02.399199 17327 catalog_manager.cc:5484] T 68bc23294ac64c9486bffebbdb16b508 P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: term changed from 0 to 1, leader changed from <none> to f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72). New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:02.402753 22568 raft_consensus.cc:504] T 7544a9b7ee6f4d2091b0b711076cf8d2 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:02.403448 22568 leader_election.cc:288] T 7544a9b7ee6f4d2091b0b711076cf8d2 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Requested vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:54:02.404192 20496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7544a9b7ee6f4d2091b0b711076cf8d2" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
I20240502 01:54:02.404381 20496 raft_consensus.cc:3037] T 7544a9b7ee6f4d2091b0b711076cf8d2 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:02.404335 20233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7544a9b7ee6f4d2091b0b711076cf8d2" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:54:02.404529 20233 raft_consensus.cc:3037] T 7544a9b7ee6f4d2091b0b711076cf8d2 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:02.407173 20496 raft_consensus.cc:2454] T 7544a9b7ee6f4d2091b0b711076cf8d2 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:54:02.407570 20233 raft_consensus.cc:2454] T 7544a9b7ee6f4d2091b0b711076cf8d2 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:54:02.407948 17673 leader_election.cc:302] T 7544a9b7ee6f4d2091b0b711076cf8d2 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:54:02.408676 22785 raft_consensus.cc:2781] T 7544a9b7ee6f4d2091b0b711076cf8d2 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:02.408823 22785 raft_consensus.cc:686] T 7544a9b7ee6f4d2091b0b711076cf8d2 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Becoming Leader. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Running, Role: LEADER
I20240502 01:54:02.412302 22785 consensus_queue.cc:237] T 7544a9b7ee6f4d2091b0b711076cf8d2 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:02.419551 17327 catalog_manager.cc:5484] T 7544a9b7ee6f4d2091b0b711076cf8d2 P dfa1152987644bd781b46c7e42a7a685 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfa1152987644bd781b46c7e42a7a685 (127.11.42.67). New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:02.438645 22932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 377f8b554ad1448d981d2283b865f8fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:02.440137 20233 raft_consensus.cc:1203] T 30f43088bd0f48ee86907df16194b9be P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:02.445695 22943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4099f4b2ae164020839faab53957750f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:02.447782 20630 raft_consensus.cc:1261] T 0d9f168de0d1410da2518abf936035ff P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:02.448397 22785 consensus_queue.cc:1035] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:02.454417 17864 raft_consensus.cc:1261] T 0d9f168de0d1410da2518abf936035ff P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:02.455142 22785 consensus_queue.cc:1035] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:02.456025 22440 raft_consensus.cc:482] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:02.456213 22440 raft_consensus.cc:504] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:02.456898 22440 leader_election.cc:288] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:54:02.457765 20365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e68acb7de346f0ad165cadabd077be" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
W20240502 01:54:02.458510 20167 leader_election.cc:341] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:02.459120 20496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e68acb7de346f0ad165cadabd077be" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
I20240502 01:54:02.459354 20496 raft_consensus.cc:2454] T 88e68acb7de346f0ad165cadabd077be P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 0.
I20240502 01:54:02.459774 20168 leader_election.cc:302] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, fdc2ac372492454c85387589a91f63af; no voters: 6a36c952c51542c0bf7ea3ec2857a9f9
I20240502 01:54:02.460458 22440 raft_consensus.cc:2781] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:02.460564 22440 raft_consensus.cc:482] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:02.460673 22440 raft_consensus.cc:3037] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:02.463269 22932 tablet_copy_client.cc:783] T 377f8b554ad1448d981d2283b865f8fe P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:02.463632 22932 tablet_copy_client.cc:647] T 377f8b554ad1448d981d2283b865f8fe P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:02.463884 22440 raft_consensus.cc:504] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:02.464982 22943 tablet_copy_client.cc:783] T 4099f4b2ae164020839faab53957750f P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:02.465301 22943 tablet_copy_client.cc:647] T 4099f4b2ae164020839faab53957750f P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:02.472677 20365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e68acb7de346f0ad165cadabd077be" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
W20240502 01:54:02.473296 20167 leader_election.cc:341] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:02.473775 22440 leader_election.cc:288] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 election: Requested vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:54:02.474277 20496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e68acb7de346f0ad165cadabd077be" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
I20240502 01:54:02.474483 20496 raft_consensus.cc:3037] T 88e68acb7de346f0ad165cadabd077be P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:02.479686 20496 raft_consensus.cc:2454] T 88e68acb7de346f0ad165cadabd077be P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 1.
I20240502 01:54:02.480115 20168 leader_election.cc:302] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, fdc2ac372492454c85387589a91f63af; no voters: 6a36c952c51542c0bf7ea3ec2857a9f9
I20240502 01:54:02.484014 22440 raft_consensus.cc:2781] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:02.489980 22440 raft_consensus.cc:686] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Becoming Leader. State: Replica: fdc2ac372492454c85387589a91f63af, State: Running, Role: LEADER
I20240502 01:54:02.490202 22981 ts_tablet_manager.cc:902] T b29218a879e74895bfe00021ad358aa4 P 2f306549407b4bb281ccf2994035e4f1: Initiating tablet copy from peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:54:02.490288 22440 consensus_queue.cc:237] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:02.493683 22932 tablet_copy_client.cc:523] T 377f8b554ad1448d981d2283b865f8fe P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:02.498795 22932 tablet_bootstrap.cc:492] T 377f8b554ad1448d981d2283b865f8fe P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:54:02.515524 17604 raft_consensus.cc:1261] T 0d9f168de0d1410da2518abf936035ff P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:02.516201 22981 tablet_copy_client.cc:308] T b29218a879e74895bfe00021ad358aa4 P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.67:36115
I20240502 01:54:02.516769 22785 consensus_queue.cc:1035] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:02.525714 17327 catalog_manager.cc:5484] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af reported cstate change: term changed from 0 to 1, leader changed from <none> to fdc2ac372492454c85387589a91f63af (127.11.42.69). New cstate: current_term: 1 leader_uuid: "fdc2ac372492454c85387589a91f63af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:02.528193 17757 tablet_copy_service.cc:140] P dfa1152987644bd781b46c7e42a7a685: Received BeginTabletCopySession request for tablet b29218a879e74895bfe00021ad358aa4 from peer 2f306549407b4bb281ccf2994035e4f1 ({username='slave'} at 127.11.42.71:49495)
I20240502 01:54:02.528481 17757 tablet_copy_service.cc:161] P dfa1152987644bd781b46c7e42a7a685: Beginning new tablet copy session on tablet b29218a879e74895bfe00021ad358aa4 from peer 2f306549407b4bb281ccf2994035e4f1 at {username='slave'} at 127.11.42.71:49495: session id = 2f306549407b4bb281ccf2994035e4f1-b29218a879e74895bfe00021ad358aa4
I20240502 01:54:02.534768 17757 tablet_copy_source_session.cc:205] T b29218a879e74895bfe00021ad358aa4 P dfa1152987644bd781b46c7e42a7a685: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:02.540037 22981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b29218a879e74895bfe00021ad358aa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:02.541178 17327 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:54:02.547681 22981 tablet_copy_client.cc:783] T b29218a879e74895bfe00021ad358aa4 P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:02.547915 22981 tablet_copy_client.cc:647] T b29218a879e74895bfe00021ad358aa4 P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:02.567101 22981 tablet_copy_client.cc:523] T b29218a879e74895bfe00021ad358aa4 P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:02.570751 22981 tablet_bootstrap.cc:492] T b29218a879e74895bfe00021ad358aa4 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
W20240502 01:54:02.580067 22964 log.cc:924] Time spent T aebde87a7f114537abd9c1d8b7e97004 P 6a36c952c51542c0bf7ea3ec2857a9f9: Append to log took a long time: real 0.134s user 0.000s sys 0.000s
W20240502 01:54:02.580885 22969 log.cc:924] Time spent T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685: Append to log took a long time: real 0.135s user 0.000s sys 0.000s
I20240502 01:54:02.582351 22906 tablet_copy_client.cc:783] T ed1522ec150a46f28fd553f9b563651f P fdc2ac372492454c85387589a91f63af: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:02.582370 22943 tablet_copy_client.cc:523] T 4099f4b2ae164020839faab53957750f P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:02.582583 22906 tablet_copy_client.cc:647] T ed1522ec150a46f28fd553f9b563651f P fdc2ac372492454c85387589a91f63af: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:02.585721 22943 tablet_bootstrap.cc:492] T 4099f4b2ae164020839faab53957750f P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:54:02.583969 22568 raft_consensus.cc:2932] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:02.591710 20629 raft_consensus.cc:2932] T 0d9f168de0d1410da2518abf936035ff P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:02.593384 20494 raft_consensus.cc:1261] T 7544a9b7ee6f4d2091b0b711076cf8d2 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:02.594599 20231 raft_consensus.cc:1261] T 7544a9b7ee6f4d2091b0b711076cf8d2 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:02.595629 22816 consensus_queue.cc:1035] T 7544a9b7ee6f4d2091b0b711076cf8d2 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:02.596212 22816 consensus_queue.cc:1035] T 7544a9b7ee6f4d2091b0b711076cf8d2 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:02.597436 17864 raft_consensus.cc:2932] T 0d9f168de0d1410da2518abf936035ff P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:02.598699 17333 catalog_manager.cc:5484] T 0d9f168de0d1410da2518abf936035ff P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: config changed from index 4 to 5, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:02.605477 22955 tablet_bootstrap.cc:492] T 495700f8ed5e41228f2e597465dad126 P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:54:02.605621 22955 ts_tablet_manager.cc:1367] T 495700f8ed5e41228f2e597465dad126 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.334s user 0.085s sys 0.067s
I20240502 01:54:02.606400 22955 raft_consensus.cc:348] T 495700f8ed5e41228f2e597465dad126 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:02.606618 22955 raft_consensus.cc:374] T 495700f8ed5e41228f2e597465dad126 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:02.606709 22955 raft_consensus.cc:729] T 495700f8ed5e41228f2e597465dad126 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:54:02.606930 22955 consensus_queue.cc:260] T 495700f8ed5e41228f2e597465dad126 P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:02.607429 22955 ts_tablet_manager.cc:1398] T 495700f8ed5e41228f2e597465dad126 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:02.607719 22955 tablet_bootstrap.cc:492] T 88e68acb7de346f0ad165cadabd077be P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:54:02.609750 22955 tablet_bootstrap.cc:654] T 88e68acb7de346f0ad165cadabd077be P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:02.610543 17605 raft_consensus.cc:2932] T 0d9f168de0d1410da2518abf936035ff P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:02.614347 17734 consensus_queue.cc:237] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } }
I20240502 01:54:02.622853 20629 raft_consensus.cc:1261] T 0d9f168de0d1410da2518abf936035ff P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:02.623567 22977 consensus_queue.cc:1035] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:02.631932 22977 raft_consensus.cc:2932] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:02.635128 22955 tablet_bootstrap.cc:492] T 88e68acb7de346f0ad165cadabd077be P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:54:02.635294 22955 ts_tablet_manager.cc:1367] T 88e68acb7de346f0ad165cadabd077be P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.002s
I20240502 01:54:02.635615 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0d9f168de0d1410da2518abf936035ff with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:02.637753 22955 raft_consensus.cc:348] T 88e68acb7de346f0ad165cadabd077be P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:02.637985 22955 raft_consensus.cc:374] T 88e68acb7de346f0ad165cadabd077be P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:02.638048 22955 raft_consensus.cc:729] T 88e68acb7de346f0ad165cadabd077be P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:54:02.638782 22955 consensus_queue.cc:260] T 88e68acb7de346f0ad165cadabd077be P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:02.639911 22955 ts_tablet_manager.cc:1398] T 88e68acb7de346f0ad165cadabd077be P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 01:54:02.643537 22906 tablet_copy_client.cc:523] T ed1522ec150a46f28fd553f9b563651f P fdc2ac372492454c85387589a91f63af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:02.652187 22932 tablet_bootstrap.cc:492] T 377f8b554ad1448d981d2283b865f8fe P 2f306549407b4bb281ccf2994035e4f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:02.652743 22932 tablet_bootstrap.cc:492] T 377f8b554ad1448d981d2283b865f8fe P 2f306549407b4bb281ccf2994035e4f1: Bootstrap complete.
I20240502 01:54:02.653131 22932 ts_tablet_manager.cc:1367] T 377f8b554ad1448d981d2283b865f8fe P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.154s user 0.039s sys 0.007s
I20240502 01:54:02.653777 17864 raft_consensus.cc:1261] T 0d9f168de0d1410da2518abf936035ff P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:02.654119 22932 raft_consensus.cc:348] T 377f8b554ad1448d981d2283b865f8fe P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
I20240502 01:54:02.654449 22932 raft_consensus.cc:729] T 377f8b554ad1448d981d2283b865f8fe P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: LEARNER
W20240502 01:54:02.654696 20564 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.42.65:38321, user_credentials={real_user=slave}} blocked reactor thread for 72501.3us
I20240502 01:54:02.654721 22932 consensus_queue.cc:260] T 377f8b554ad1448d981d2283b865f8fe P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
I20240502 01:54:02.655649 22932 ts_tablet_manager.cc:1398] T 377f8b554ad1448d981d2283b865f8fe P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:54:02.656345 17887 tablet_copy_service.cc:342] P bb31bceba76d466db24b9341c04900c3: Request end of tablet copy session 2f306549407b4bb281ccf2994035e4f1-377f8b554ad1448d981d2283b865f8fe received from {username='slave'} at 127.11.42.71:48285
I20240502 01:54:02.656549 17887 tablet_copy_service.cc:433] P bb31bceba76d466db24b9341c04900c3: ending tablet copy session 2f306549407b4bb281ccf2994035e4f1-377f8b554ad1448d981d2283b865f8fe on tablet 377f8b554ad1448d981d2283b865f8fe with peer 2f306549407b4bb281ccf2994035e4f1
I20240502 01:54:02.656780 22906 tablet_bootstrap.cc:492] T ed1522ec150a46f28fd553f9b563651f P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:54:02.657521 20366 raft_consensus.cc:2932] T aebde87a7f114537abd9c1d8b7e97004 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:02.661967 22568 consensus_queue.cc:1035] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:02.663405 17864 raft_consensus.cc:2932] T 0d9f168de0d1410da2518abf936035ff P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:02.671912 17330 catalog_manager.cc:5484] T 0d9f168de0d1410da2518abf936035ff P bb31bceba76d466db24b9341c04900c3 reported cstate change: config changed from index 5 to 6, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:02.676498 17587 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d9f168de0d1410da2518abf936035ff with delete_type TABLET_DATA_TOMBSTONED (TS 3fceccefba274b1dbbd8216af8b0bc29 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55232
I20240502 01:54:02.686190 20629 raft_consensus.cc:2932] T 0d9f168de0d1410da2518abf936035ff P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:02.690104 23004 tablet.cc:1722] T 7544a9b7ee6f4d2091b0b711076cf8d2 P fdc2ac372492454c85387589a91f63af: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:02.690666 23004 tablet.cc:1620] T 7544a9b7ee6f4d2091b0b711076cf8d2 P fdc2ac372492454c85387589a91f63af: MemRowSet was empty: no flush needed.
I20240502 01:54:02.691619 23005 tablet.cc:1722] T 7544a9b7ee6f4d2091b0b711076cf8d2 P 2f306549407b4bb281ccf2994035e4f1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:02.692951 23005 tablet.cc:1620] T 7544a9b7ee6f4d2091b0b711076cf8d2 P 2f306549407b4bb281ccf2994035e4f1: MemRowSet was empty: no flush needed.
I20240502 01:54:02.696391 17327 catalog_manager.cc:5576] TS 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623) does not have the latest schema for tablet 7544a9b7ee6f4d2091b0b711076cf8d2 (table rebalancer_dynamic_table [id=2fedd81ef2f345f8b9e5896b6b6bb642]). Expected version 1 got 0
I20240502 01:54:02.698474 17334 catalog_manager.cc:5576] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115) does not have the latest schema for tablet 7544a9b7ee6f4d2091b0b711076cf8d2 (table rebalancer_dynamic_table [id=2fedd81ef2f345f8b9e5896b6b6bb642]). Expected version 1 got 0
I20240502 01:54:02.684877 23003 tablet.cc:1722] T 7544a9b7ee6f4d2091b0b711076cf8d2 P dfa1152987644bd781b46c7e42a7a685: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:02.699214 23003 tablet.cc:1620] T 7544a9b7ee6f4d2091b0b711076cf8d2 P dfa1152987644bd781b46c7e42a7a685: MemRowSet was empty: no flush needed.
I20240502 01:54:02.710675 17334 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=2fedd81ef2f345f8b9e5896b6b6bb642] alter complete (version 1)
I20240502 01:54:02.712460 17333 catalog_manager.cc:5576] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339) does not have the latest schema for tablet 7544a9b7ee6f4d2091b0b711076cf8d2 (table rebalancer_dynamic_table [id=2fedd81ef2f345f8b9e5896b6b6bb642]). Expected version 1 got 0
I20240502 01:54:02.722715 22999 tablet_replica.cc:330] T 0d9f168de0d1410da2518abf936035ff P 3fceccefba274b1dbbd8216af8b0bc29: stopping tablet replica
I20240502 01:54:02.723313 22999 raft_consensus.cc:2229] T 0d9f168de0d1410da2518abf936035ff P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:02.723780 22999 raft_consensus.cc:2258] T 0d9f168de0d1410da2518abf936035ff P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:02.735399 22999 ts_tablet_manager.cc:1875] T 0d9f168de0d1410da2518abf936035ff P 3fceccefba274b1dbbd8216af8b0bc29: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:02.736423 22943 tablet_bootstrap.cc:492] T 4099f4b2ae164020839faab53957750f P f19cc6cb1e76460aa6711e49a5775091: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:02.736878 22943 tablet_bootstrap.cc:492] T 4099f4b2ae164020839faab53957750f P f19cc6cb1e76460aa6711e49a5775091: Bootstrap complete.
I20240502 01:54:02.737172 22943 ts_tablet_manager.cc:1367] T 4099f4b2ae164020839faab53957750f P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.152s user 0.050s sys 0.014s
I20240502 01:54:02.737998 22943 raft_consensus.cc:348] T 4099f4b2ae164020839faab53957750f P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:54:02.738241 22943 raft_consensus.cc:729] T 4099f4b2ae164020839faab53957750f P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: LEARNER
I20240502 01:54:02.738456 22943 consensus_queue.cc:260] T 4099f4b2ae164020839faab53957750f P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:54:02.742497 22943 ts_tablet_manager.cc:1398] T 4099f4b2ae164020839faab53957750f P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:54:02.743131 17757 tablet_copy_service.cc:342] P dfa1152987644bd781b46c7e42a7a685: Request end of tablet copy session f19cc6cb1e76460aa6711e49a5775091-4099f4b2ae164020839faab53957750f received from {username='slave'} at 127.11.42.72:53665
I20240502 01:54:02.743253 17757 tablet_copy_service.cc:433] P dfa1152987644bd781b46c7e42a7a685: ending tablet copy session f19cc6cb1e76460aa6711e49a5775091-4099f4b2ae164020839faab53957750f on tablet 4099f4b2ae164020839faab53957750f with peer f19cc6cb1e76460aa6711e49a5775091
I20240502 01:54:02.766393 22999 ts_tablet_manager.cc:1888] T 0d9f168de0d1410da2518abf936035ff P 3fceccefba274b1dbbd8216af8b0bc29: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:02.767763 22999 log.cc:1196] T 0d9f168de0d1410da2518abf936035ff P 3fceccefba274b1dbbd8216af8b0bc29: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/wals/0d9f168de0d1410da2518abf936035ff
I20240502 01:54:02.768790 17319 catalog_manager.cc:4830] TS 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): tablet 0d9f168de0d1410da2518abf936035ff (table rebalance_test_table_9 [id=ffe6d29170f24ad48433354f548b9b2a]) successfully deleted
I20240502 01:54:02.810983 22487 raft_consensus.cc:482] T 495700f8ed5e41228f2e597465dad126 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:02.811203 22487 raft_consensus.cc:504] T 495700f8ed5e41228f2e597465dad126 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:02.811890 22487 leader_election.cc:288] T 495700f8ed5e41228f2e597465dad126 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:54:02.813099 20629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "495700f8ed5e41228f2e597465dad126" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091" is_pre_election: true
I20240502 01:54:02.813344 20629 raft_consensus.cc:2454] T 495700f8ed5e41228f2e597465dad126 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 0.
I20240502 01:54:02.814121 17333 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:54:02.817175 17411 leader_election.cc:302] T 495700f8ed5e41228f2e597465dad126 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:54:02.817545 20366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "495700f8ed5e41228f2e597465dad126" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:54:02.817799 20366 raft_consensus.cc:2454] T 495700f8ed5e41228f2e597465dad126 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 0.
I20240502 01:54:02.823779 23003 tablet.cc:1722] T 7544a9b7ee6f4d2091b0b711076cf8d2 P dfa1152987644bd781b46c7e42a7a685: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:02.824331 23003 tablet.cc:1620] T 7544a9b7ee6f4d2091b0b711076cf8d2 P dfa1152987644bd781b46c7e42a7a685: MemRowSet was empty: no flush needed.
I20240502 01:54:02.824406 23004 tablet.cc:1722] T 7544a9b7ee6f4d2091b0b711076cf8d2 P fdc2ac372492454c85387589a91f63af: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:02.824867 23004 tablet.cc:1620] T 7544a9b7ee6f4d2091b0b711076cf8d2 P fdc2ac372492454c85387589a91f63af: MemRowSet was empty: no flush needed.
I20240502 01:54:02.827306 22487 raft_consensus.cc:2781] T 495700f8ed5e41228f2e597465dad126 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:02.827404 22487 raft_consensus.cc:482] T 495700f8ed5e41228f2e597465dad126 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:02.827515 22487 raft_consensus.cc:3037] T 495700f8ed5e41228f2e597465dad126 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:02.830812 23005 tablet.cc:1722] T 7544a9b7ee6f4d2091b0b711076cf8d2 P 2f306549407b4bb281ccf2994035e4f1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:02.831231 23005 tablet.cc:1620] T 7544a9b7ee6f4d2091b0b711076cf8d2 P 2f306549407b4bb281ccf2994035e4f1: MemRowSet was empty: no flush needed.
I20240502 01:54:02.836182 17333 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=2fedd81ef2f345f8b9e5896b6b6bb642] alter complete (version 2)
I20240502 01:54:02.862234 22487 raft_consensus.cc:504] T 495700f8ed5e41228f2e597465dad126 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:02.862906 22487 leader_election.cc:288] T 495700f8ed5e41228f2e597465dad126 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Requested vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:54:02.864470 20630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "495700f8ed5e41228f2e597465dad126" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091"
I20240502 01:54:02.864670 20630 raft_consensus.cc:3037] T 495700f8ed5e41228f2e597465dad126 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:02.866432 22981 tablet_bootstrap.cc:492] T b29218a879e74895bfe00021ad358aa4 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:02.866912 22981 tablet_bootstrap.cc:492] T b29218a879e74895bfe00021ad358aa4 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap complete.
I20240502 01:54:02.867165 22981 ts_tablet_manager.cc:1367] T b29218a879e74895bfe00021ad358aa4 P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.297s user 0.045s sys 0.008s
I20240502 01:54:02.867792 20630 raft_consensus.cc:2454] T 495700f8ed5e41228f2e597465dad126 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
I20240502 01:54:02.867918 22981 raft_consensus.cc:348] T b29218a879e74895bfe00021ad358aa4 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
I20240502 01:54:02.868145 22981 raft_consensus.cc:729] T b29218a879e74895bfe00021ad358aa4 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: LEARNER
I20240502 01:54:02.868343 22981 consensus_queue.cc:260] T b29218a879e74895bfe00021ad358aa4 P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
I20240502 01:54:02.868686 20366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "495700f8ed5e41228f2e597465dad126" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
I20240502 01:54:02.868877 20366 raft_consensus.cc:3037] T 495700f8ed5e41228f2e597465dad126 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:02.869370 22981 ts_tablet_manager.cc:1398] T b29218a879e74895bfe00021ad358aa4 P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:02.871879 17757 tablet_copy_service.cc:342] P dfa1152987644bd781b46c7e42a7a685: Request end of tablet copy session 2f306549407b4bb281ccf2994035e4f1-b29218a879e74895bfe00021ad358aa4 received from {username='slave'} at 127.11.42.71:49495
I20240502 01:54:02.872001 17757 tablet_copy_service.cc:433] P dfa1152987644bd781b46c7e42a7a685: ending tablet copy session 2f306549407b4bb281ccf2994035e4f1-b29218a879e74895bfe00021ad358aa4 on tablet b29218a879e74895bfe00021ad358aa4 with peer 2f306549407b4bb281ccf2994035e4f1
I20240502 01:54:02.872606 17411 leader_election.cc:302] T 495700f8ed5e41228f2e597465dad126 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:54:02.873257 22487 raft_consensus.cc:2781] T 495700f8ed5e41228f2e597465dad126 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:02.876219 22906 tablet_bootstrap.cc:492] T ed1522ec150a46f28fd553f9b563651f P fdc2ac372492454c85387589a91f63af: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:02.876658 22906 tablet_bootstrap.cc:492] T ed1522ec150a46f28fd553f9b563651f P fdc2ac372492454c85387589a91f63af: Bootstrap complete.
I20240502 01:54:02.876919 22906 ts_tablet_manager.cc:1367] T ed1522ec150a46f28fd553f9b563651f P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.220s user 0.045s sys 0.005s
I20240502 01:54:02.877666 22906 raft_consensus.cc:348] T ed1522ec150a46f28fd553f9b563651f P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } }
I20240502 01:54:02.877928 22906 raft_consensus.cc:729] T ed1522ec150a46f28fd553f9b563651f P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: LEARNER
I20240502 01:54:02.878157 22906 consensus_queue.cc:260] T ed1522ec150a46f28fd553f9b563651f P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } }
I20240502 01:54:02.879125 22906 ts_tablet_manager.cc:1398] T ed1522ec150a46f28fd553f9b563651f P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 01:54:02.878577 20366 raft_consensus.cc:2454] T 495700f8ed5e41228f2e597465dad126 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
I20240502 01:54:02.882014 17887 tablet_copy_service.cc:342] P bb31bceba76d466db24b9341c04900c3: Request end of tablet copy session fdc2ac372492454c85387589a91f63af-ed1522ec150a46f28fd553f9b563651f received from {username='slave'} at 127.11.42.69:56011
I20240502 01:54:02.882148 17887 tablet_copy_service.cc:433] P bb31bceba76d466db24b9341c04900c3: ending tablet copy session fdc2ac372492454c85387589a91f63af-ed1522ec150a46f28fd553f9b563651f on tablet ed1522ec150a46f28fd553f9b563651f with peer fdc2ac372492454c85387589a91f63af
I20240502 01:54:02.884588 22756 consensus_queue.cc:1035] T 68bc23294ac64c9486bffebbdb16b508 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:02.890033 22487 raft_consensus.cc:686] T 495700f8ed5e41228f2e597465dad126 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Becoming Leader. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Running, Role: LEADER
I20240502 01:54:02.890597 22487 consensus_queue.cc:237] T 495700f8ed5e41228f2e597465dad126 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:02.902726 22940 consensus_queue.cc:1035] T 68bc23294ac64c9486bffebbdb16b508 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:02.905140 17333 catalog_manager.cc:5484] T 495700f8ed5e41228f2e597465dad126 P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: term changed from 0 to 1, leader changed from <none> to 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65). New cstate: current_term: 1 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:02.911489 22440 consensus_queue.cc:1035] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:02.924875 20366 raft_consensus.cc:3037] T 88e68acb7de346f0ad165cadabd077be P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:02.928406 22440 consensus_queue.cc:1035] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:02.955602 17334 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:54:02.963403 20496 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aebde87a7f114537abd9c1d8b7e97004"
dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33236
I20240502 01:54:02.963603 20496 raft_consensus.cc:595] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:54:02.963722 23003 tablet.cc:1722] T 7544a9b7ee6f4d2091b0b711076cf8d2 P dfa1152987644bd781b46c7e42a7a685: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:02.965353 23005 tablet.cc:1722] T 7544a9b7ee6f4d2091b0b711076cf8d2 P 2f306549407b4bb281ccf2994035e4f1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:02.967562 23004 tablet.cc:1722] T 7544a9b7ee6f4d2091b0b711076cf8d2 P fdc2ac372492454c85387589a91f63af: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:02.976356 17334 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table_new_name [id=2fedd81ef2f345f8b9e5896b6b6bb642] alter complete (version 3)
I20240502 01:54:02.977917 22792 raft_consensus.cc:1050] T 30f43088bd0f48ee86907df16194b9be P 3fceccefba274b1dbbd8216af8b0bc29: attempting to promote NON_VOTER fdc2ac372492454c85387589a91f63af to VOTER
I20240502 01:54:02.978729 22792 consensus_queue.cc:237] T 30f43088bd0f48ee86907df16194b9be P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } }
I20240502 01:54:02.986377 20233 raft_consensus.cc:1261] T 30f43088bd0f48ee86907df16194b9be P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:02.986908 17473 raft_consensus.cc:1261] T 30f43088bd0f48ee86907df16194b9be P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:02.987737 22692 consensus_queue.cc:1035] T 30f43088bd0f48ee86907df16194b9be P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:02.988070 22792 consensus_queue.cc:1035] T 30f43088bd0f48ee86907df16194b9be P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:02.988139 17734 raft_consensus.cc:1261] T 30f43088bd0f48ee86907df16194b9be P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:02.988926 23052 consensus_queue.cc:1035] T 30f43088bd0f48ee86907df16194b9be P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:03.001250 22792 raft_consensus.cc:2932] T 30f43088bd0f48ee86907df16194b9be P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:03.006372 17734 raft_consensus.cc:2932] T 30f43088bd0f48ee86907df16194b9be P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:03.007048 20233 raft_consensus.cc:2932] T 30f43088bd0f48ee86907df16194b9be P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:03.010991 17334 catalog_manager.cc:5484] T 30f43088bd0f48ee86907df16194b9be P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: config changed from index 4 to 5, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:03.018625 17473 raft_consensus.cc:2932] T 30f43088bd0f48ee86907df16194b9be P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:03.023609 17604 consensus_queue.cc:237] T 30f43088bd0f48ee86907df16194b9be P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } }
I20240502 01:54:03.032052 20233 raft_consensus.cc:1261] T 30f43088bd0f48ee86907df16194b9be P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:03.033329 17474 raft_consensus.cc:1261] T 30f43088bd0f48ee86907df16194b9be P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:03.034107 23052 consensus_queue.cc:1035] T 30f43088bd0f48ee86907df16194b9be P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:03.039059 23052 consensus_queue.cc:1035] T 30f43088bd0f48ee86907df16194b9be P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:03.041589 17474 raft_consensus.cc:2932] T 30f43088bd0f48ee86907df16194b9be P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:03.045440 20231 raft_consensus.cc:2932] T 30f43088bd0f48ee86907df16194b9be P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:03.046181 22692 raft_consensus.cc:2932] T 30f43088bd0f48ee86907df16194b9be P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:03.049201 17334 catalog_manager.cc:5484] T 30f43088bd0f48ee86907df16194b9be P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: config changed from index 5 to 6, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:03.049422 17319 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 30f43088bd0f48ee86907df16194b9be with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:03.053676 17716 tablet_service.cc:1514] Processing DeleteTablet for tablet 30f43088bd0f48ee86907df16194b9be with delete_type TABLET_DATA_TOMBSTONED (TS dfa1152987644bd781b46c7e42a7a685 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52198
I20240502 01:54:03.055130 23061 tablet_replica.cc:330] T 30f43088bd0f48ee86907df16194b9be P dfa1152987644bd781b46c7e42a7a685: stopping tablet replica
I20240502 01:54:03.055394 23061 raft_consensus.cc:2229] T 30f43088bd0f48ee86907df16194b9be P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:03.056342 23061 raft_consensus.cc:2258] T 30f43088bd0f48ee86907df16194b9be P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:03.058390 23061 ts_tablet_manager.cc:1875] T 30f43088bd0f48ee86907df16194b9be P dfa1152987644bd781b46c7e42a7a685: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:03.064850 23061 ts_tablet_manager.cc:1888] T 30f43088bd0f48ee86907df16194b9be P dfa1152987644bd781b46c7e42a7a685: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:03.064946 23061 log.cc:1196] T 30f43088bd0f48ee86907df16194b9be P dfa1152987644bd781b46c7e42a7a685: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/wals/30f43088bd0f48ee86907df16194b9be
I20240502 01:54:03.065663 17321 catalog_manager.cc:4830] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): tablet 30f43088bd0f48ee86907df16194b9be (table rebalance_test_table_17 [id=c6474d0c5256497eb6fa5844f3530b62]) successfully deleted
I20240502 01:54:03.068066 20496 raft_consensus.cc:1203] T 377f8b554ad1448d981d2283b865f8fe P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:03.082962 17334 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:03.083191 17334 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:03.086416 17334 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet 7544a9b7ee6f4d2091b0b711076cf8d2
I20240502 01:54:03.087316 20476 tablet_service.cc:1514] Processing DeleteTablet for tablet 7544a9b7ee6f4d2091b0b711076cf8d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:03 UTC) from {username='slave'} at 127.0.0.1:35018
I20240502 01:54:03.087338 20212 tablet_service.cc:1514] Processing DeleteTablet for tablet 7544a9b7ee6f4d2091b0b711076cf8d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:03 UTC) from {username='slave'} at 127.0.0.1:44624
I20240502 01:54:03.087733 17716 tablet_service.cc:1514] Processing DeleteTablet for tablet 7544a9b7ee6f4d2091b0b711076cf8d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:03 UTC) from {username='slave'} at 127.0.0.1:52198
I20240502 01:54:03.087975 23072 tablet_replica.cc:330] T 7544a9b7ee6f4d2091b0b711076cf8d2 P fdc2ac372492454c85387589a91f63af: stopping tablet replica
I20240502 01:54:03.088171 23072 raft_consensus.cc:2229] T 7544a9b7ee6f4d2091b0b711076cf8d2 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:03.088413 23072 raft_consensus.cc:2258] T 7544a9b7ee6f4d2091b0b711076cf8d2 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:03.088946 23073 tablet_replica.cc:330] T 7544a9b7ee6f4d2091b0b711076cf8d2 P 2f306549407b4bb281ccf2994035e4f1: stopping tablet replica
I20240502 01:54:03.089236 23073 raft_consensus.cc:2229] T 7544a9b7ee6f4d2091b0b711076cf8d2 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:03.089671 23073 raft_consensus.cc:2258] T 7544a9b7ee6f4d2091b0b711076cf8d2 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:03.089694 23061 tablet_replica.cc:330] T 7544a9b7ee6f4d2091b0b711076cf8d2 P dfa1152987644bd781b46c7e42a7a685: stopping tablet replica
I20240502 01:54:03.089736 23072 ts_tablet_manager.cc:1875] T 7544a9b7ee6f4d2091b0b711076cf8d2 P fdc2ac372492454c85387589a91f63af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:03.089898 23061 raft_consensus.cc:2229] T 7544a9b7ee6f4d2091b0b711076cf8d2 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:03.090327 23061 raft_consensus.cc:2258] T 7544a9b7ee6f4d2091b0b711076cf8d2 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:03.091363 23061 ts_tablet_manager.cc:1875] T 7544a9b7ee6f4d2091b0b711076cf8d2 P dfa1152987644bd781b46c7e42a7a685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:03.091387 23073 ts_tablet_manager.cc:1875] T 7544a9b7ee6f4d2091b0b711076cf8d2 P 2f306549407b4bb281ccf2994035e4f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:03.096295 23072 ts_tablet_manager.cc:1888] T 7544a9b7ee6f4d2091b0b711076cf8d2 P fdc2ac372492454c85387589a91f63af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:03.096505 23072 log.cc:1196] T 7544a9b7ee6f4d2091b0b711076cf8d2 P fdc2ac372492454c85387589a91f63af: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal/wals/7544a9b7ee6f4d2091b0b711076cf8d2
I20240502 01:54:03.097045 23072 ts_tablet_manager.cc:1909] T 7544a9b7ee6f4d2091b0b711076cf8d2 P fdc2ac372492454c85387589a91f63af: Deleting consensus metadata
I20240502 01:54:03.098111 17322 catalog_manager.cc:4830] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): tablet 7544a9b7ee6f4d2091b0b711076cf8d2 (table rebalancer_dynamic_table_new_name [id=2fedd81ef2f345f8b9e5896b6b6bb642]) successfully deleted
I20240502 01:54:03.099040 23061 ts_tablet_manager.cc:1888] T 7544a9b7ee6f4d2091b0b711076cf8d2 P dfa1152987644bd781b46c7e42a7a685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:03.099169 23061 log.cc:1196] T 7544a9b7ee6f4d2091b0b711076cf8d2 P dfa1152987644bd781b46c7e42a7a685: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/wals/7544a9b7ee6f4d2091b0b711076cf8d2
I20240502 01:54:03.099336 23073 ts_tablet_manager.cc:1888] T 7544a9b7ee6f4d2091b0b711076cf8d2 P 2f306549407b4bb281ccf2994035e4f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:03.099593 23061 ts_tablet_manager.cc:1909] T 7544a9b7ee6f4d2091b0b711076cf8d2 P dfa1152987644bd781b46c7e42a7a685: Deleting consensus metadata
I20240502 01:54:03.099649 23073 log.cc:1196] T 7544a9b7ee6f4d2091b0b711076cf8d2 P 2f306549407b4bb281ccf2994035e4f1: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal/wals/7544a9b7ee6f4d2091b0b711076cf8d2
I20240502 01:54:03.100126 23073 ts_tablet_manager.cc:1909] T 7544a9b7ee6f4d2091b0b711076cf8d2 P 2f306549407b4bb281ccf2994035e4f1: Deleting consensus metadata
I20240502 01:54:03.101205 17322 catalog_manager.cc:4830] TS 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): tablet 7544a9b7ee6f4d2091b0b711076cf8d2 (table rebalancer_dynamic_table_new_name [id=2fedd81ef2f345f8b9e5896b6b6bb642]) successfully deleted
I20240502 01:54:03.102219 17321 catalog_manager.cc:4830] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): tablet 7544a9b7ee6f4d2091b0b711076cf8d2 (table rebalancer_dynamic_table_new_name [id=2fedd81ef2f345f8b9e5896b6b6bb642]) successfully deleted
I20240502 01:54:03.106637 20496 raft_consensus.cc:1203] T b29218a879e74895bfe00021ad358aa4 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:03.151557 20496 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aebde87a7f114537abd9c1d8b7e97004"
dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33252
I20240502 01:54:03.151746 20496 raft_consensus.cc:595] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:54:03.189877 17334 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:54:03.198683 20347 tablet_service.cc:1467] Processing CreateTablet for tablet 23d171efbc844165944135fc1be2a099 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=56fc003514884047a0449e009700503d]), partition=
I20240502 01:54:03.198734 20610 tablet_service.cc:1467] Processing CreateTablet for tablet 23d171efbc844165944135fc1be2a099 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=56fc003514884047a0449e009700503d]), partition=
I20240502 01:54:03.198853 17716 tablet_service.cc:1467] Processing CreateTablet for tablet 23d171efbc844165944135fc1be2a099 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=56fc003514884047a0449e009700503d]), partition=
I20240502 01:54:03.199083 20347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23d171efbc844165944135fc1be2a099. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:03.199195 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23d171efbc844165944135fc1be2a099. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:03.199219 20610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23d171efbc844165944135fc1be2a099. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:03.217377 23120 tablet_bootstrap.cc:492] T 23d171efbc844165944135fc1be2a099 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:54:03.218699 23119 tablet_bootstrap.cc:492] T 23d171efbc844165944135fc1be2a099 P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:54:03.220938 23119 tablet_bootstrap.cc:654] T 23d171efbc844165944135fc1be2a099 P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:03.222774 23121 tablet_bootstrap.cc:492] T 23d171efbc844165944135fc1be2a099 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:54:03.223855 23119 tablet_bootstrap.cc:492] T 23d171efbc844165944135fc1be2a099 P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:54:03.224041 23119 ts_tablet_manager.cc:1367] T 23d171efbc844165944135fc1be2a099 P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 01:54:03.224982 23119 raft_consensus.cc:348] T 23d171efbc844165944135fc1be2a099 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:03.225270 23119 raft_consensus.cc:374] T 23d171efbc844165944135fc1be2a099 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:03.225394 23119 raft_consensus.cc:729] T 23d171efbc844165944135fc1be2a099 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:54:03.225642 23119 consensus_queue.cc:260] T 23d171efbc844165944135fc1be2a099 P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:03.226346 23119 ts_tablet_manager.cc:1398] T 23d171efbc844165944135fc1be2a099 P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:03.227780 23121 tablet_bootstrap.cc:654] T 23d171efbc844165944135fc1be2a099 P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:03.228485 23120 tablet_bootstrap.cc:654] T 23d171efbc844165944135fc1be2a099 P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:03.234355 23121 tablet_bootstrap.cc:492] T 23d171efbc844165944135fc1be2a099 P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:54:03.234529 23121 ts_tablet_manager.cc:1367] T 23d171efbc844165944135fc1be2a099 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240502 01:54:03.235031 23120 tablet_bootstrap.cc:492] T 23d171efbc844165944135fc1be2a099 P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:54:03.235195 23120 ts_tablet_manager.cc:1367] T 23d171efbc844165944135fc1be2a099 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.002s
I20240502 01:54:03.235405 23121 raft_consensus.cc:348] T 23d171efbc844165944135fc1be2a099 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:03.235668 23121 raft_consensus.cc:374] T 23d171efbc844165944135fc1be2a099 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:03.235805 23121 raft_consensus.cc:729] T 23d171efbc844165944135fc1be2a099 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:54:03.236088 23121 consensus_queue.cc:260] T 23d171efbc844165944135fc1be2a099 P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:03.236188 23120 raft_consensus.cc:348] T 23d171efbc844165944135fc1be2a099 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:03.236385 23120 raft_consensus.cc:374] T 23d171efbc844165944135fc1be2a099 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:03.236516 23120 raft_consensus.cc:729] T 23d171efbc844165944135fc1be2a099 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:54:03.236785 23120 consensus_queue.cc:260] T 23d171efbc844165944135fc1be2a099 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:03.237404 23120 ts_tablet_manager.cc:1398] T 23d171efbc844165944135fc1be2a099 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:03.238737 22393 raft_consensus.cc:979] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1: : Instructing follower 6a36c952c51542c0bf7ea3ec2857a9f9 to start an election
I20240502 01:54:03.238883 22393 raft_consensus.cc:1067] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Signalling peer 6a36c952c51542c0bf7ea3ec2857a9f9 to start an election
I20240502 01:54:03.239730 20365 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aebde87a7f114537abd9c1d8b7e97004"
dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
from {username='slave'} at 127.11.42.71:39255
I20240502 01:54:03.239967 20365 raft_consensus.cc:482] T aebde87a7f114537abd9c1d8b7e97004 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:03.240120 20365 raft_consensus.cc:3037] T aebde87a7f114537abd9c1d8b7e97004 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:03.242805 20365 raft_consensus.cc:504] T aebde87a7f114537abd9c1d8b7e97004 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:03.244315 23121 ts_tablet_manager.cc:1398] T 23d171efbc844165944135fc1be2a099 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:54:03.244422 20496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aebde87a7f114537abd9c1d8b7e97004" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
I20240502 01:54:03.244621 20496 raft_consensus.cc:3032] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:03.244688 20496 raft_consensus.cc:729] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Running, Role: LEADER
I20240502 01:54:03.244951 20496 consensus_queue.cc:260] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:03.245347 20496 raft_consensus.cc:3037] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:03.250089 20496 raft_consensus.cc:2454] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 2.
I20240502 01:54:03.251585 17864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aebde87a7f114537abd9c1d8b7e97004" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:54:03.251781 17864 raft_consensus.cc:3037] T aebde87a7f114537abd9c1d8b7e97004 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:03.254482 17864 raft_consensus.cc:2454] T aebde87a7f114537abd9c1d8b7e97004 P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 2.
I20240502 01:54:03.257267 20303 leader_election.cc:302] T aebde87a7f114537abd9c1d8b7e97004 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 6a36c952c51542c0bf7ea3ec2857a9f9, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:54:03.257561 22876 raft_consensus.cc:2781] T aebde87a7f114537abd9c1d8b7e97004 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:03.257680 22876 raft_consensus.cc:686] T aebde87a7f114537abd9c1d8b7e97004 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 LEADER]: Becoming Leader. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Running, Role: LEADER
I20240502 01:54:03.263495 22876 consensus_queue.cc:237] T aebde87a7f114537abd9c1d8b7e97004 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:03.265949 17734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aebde87a7f114537abd9c1d8b7e97004" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:54:03.266173 17734 raft_consensus.cc:3037] T aebde87a7f114537abd9c1d8b7e97004 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:03.267205 17333 catalog_manager.cc:5484] T aebde87a7f114537abd9c1d8b7e97004 P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: term changed from 1 to 2, leader changed from 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) to 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70). New cstate: current_term: 2 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:03.270193 17734 raft_consensus.cc:2454] T aebde87a7f114537abd9c1d8b7e97004 P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 2.
I20240502 01:54:03.274814 20365 leader_election.cc:288] T aebde87a7f114537abd9c1d8b7e97004 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 election: Requested vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:54:03.295145 22816 raft_consensus.cc:482] T 23d171efbc844165944135fc1be2a099 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:03.295346 22816 raft_consensus.cc:504] T 23d171efbc844165944135fc1be2a099 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:03.295950 23016 consensus_queue.cc:1035] T 495700f8ed5e41228f2e597465dad126 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:03.296090 22816 leader_election.cc:288] T 23d171efbc844165944135fc1be2a099 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:54:03.296669 20365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23d171efbc844165944135fc1be2a099" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:54:03.296670 20629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23d171efbc844165944135fc1be2a099" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091" is_pre_election: true
I20240502 01:54:03.296903 20629 raft_consensus.cc:2454] T 23d171efbc844165944135fc1be2a099 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:54:03.296916 20365 raft_consensus.cc:2454] T 23d171efbc844165944135fc1be2a099 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:54:03.297356 17672 leader_election.cc:302] T 23d171efbc844165944135fc1be2a099 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dfa1152987644bd781b46c7e42a7a685, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:54:03.297648 22816 raft_consensus.cc:2781] T 23d171efbc844165944135fc1be2a099 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:03.297760 22816 raft_consensus.cc:482] T 23d171efbc844165944135fc1be2a099 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:03.297858 22816 raft_consensus.cc:3037] T 23d171efbc844165944135fc1be2a099 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:03.299945 22816 raft_consensus.cc:504] T 23d171efbc844165944135fc1be2a099 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:03.300499 22816 leader_election.cc:288] T 23d171efbc844165944135fc1be2a099 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Requested vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:54:03.300911 20365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23d171efbc844165944135fc1be2a099" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
I20240502 01:54:03.301115 20365 raft_consensus.cc:3037] T 23d171efbc844165944135fc1be2a099 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:03.303395 20365 raft_consensus.cc:2454] T 23d171efbc844165944135fc1be2a099 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:54:03.303727 17672 leader_election.cc:302] T 23d171efbc844165944135fc1be2a099 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:54:03.304308 22816 raft_consensus.cc:2781] T 23d171efbc844165944135fc1be2a099 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:03.304802 22816 raft_consensus.cc:686] T 23d171efbc844165944135fc1be2a099 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Becoming Leader. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Running, Role: LEADER
I20240502 01:54:03.304826 20629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23d171efbc844165944135fc1be2a099" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091"
I20240502 01:54:03.305037 20629 raft_consensus.cc:3037] T 23d171efbc844165944135fc1be2a099 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:03.305115 22988 consensus_queue.cc:1035] T 495700f8ed5e41228f2e597465dad126 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:03.305120 22816 consensus_queue.cc:237] T 23d171efbc844165944135fc1be2a099 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:03.307960 20629 raft_consensus.cc:2454] T 23d171efbc844165944135fc1be2a099 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:54:03.313097 17334 catalog_manager.cc:5484] T 23d171efbc844165944135fc1be2a099 P dfa1152987644bd781b46c7e42a7a685 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfa1152987644bd781b46c7e42a7a685 (127.11.42.67). New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:03.328912 20231 raft_consensus.cc:1203] T ed1522ec150a46f28fd553f9b563651f P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:03.335662 20629 raft_consensus.cc:1203] T 4099f4b2ae164020839faab53957750f P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:03.426306 17333 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:54:03.431773 20629 raft_consensus.cc:1261] T 23d171efbc844165944135fc1be2a099 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:03.431923 20366 raft_consensus.cc:1261] T 23d171efbc844165944135fc1be2a099 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:03.432336 22568 consensus_queue.cc:1035] T 23d171efbc844165944135fc1be2a099 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:03.432636 22816 consensus_queue.cc:1035] T 23d171efbc844165944135fc1be2a099 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:03.437057 23003 tablet.cc:1722] T 23d171efbc844165944135fc1be2a099 P dfa1152987644bd781b46c7e42a7a685: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:03.437572 23003 tablet.cc:1620] T 23d171efbc844165944135fc1be2a099 P dfa1152987644bd781b46c7e42a7a685: MemRowSet was empty: no flush needed.
I20240502 01:54:03.439774 17334 catalog_manager.cc:5576] TS 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155) does not have the latest schema for tablet 23d171efbc844165944135fc1be2a099 (table rebalancer_dynamic_table [id=56fc003514884047a0449e009700503d]). Expected version 1 got 0
I20240502 01:54:03.442957 23142 tablet.cc:1722] T 23d171efbc844165944135fc1be2a099 P 6a36c952c51542c0bf7ea3ec2857a9f9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:03.445408 23142 tablet.cc:1620] T 23d171efbc844165944135fc1be2a099 P 6a36c952c51542c0bf7ea3ec2857a9f9: MemRowSet was empty: no flush needed.
I20240502 01:54:03.445758 17334 catalog_manager.cc:5576] TS 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155) does not have the latest schema for tablet 23d171efbc844165944135fc1be2a099 (table rebalancer_dynamic_table [id=56fc003514884047a0449e009700503d]). Expected version 1 got 0
I20240502 01:54:03.445999 17333 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=56fc003514884047a0449e009700503d] alter complete (version 1)
I20240502 01:54:03.450578 17333 catalog_manager.cc:5576] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411) does not have the latest schema for tablet 23d171efbc844165944135fc1be2a099 (table rebalancer_dynamic_table [id=56fc003514884047a0449e009700503d]). Expected version 1 got 0
I20240502 01:54:03.453294 23143 tablet.cc:1722] T 23d171efbc844165944135fc1be2a099 P f19cc6cb1e76460aa6711e49a5775091: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:03.454540 23143 tablet.cc:1620] T 23d171efbc844165944135fc1be2a099 P f19cc6cb1e76460aa6711e49a5775091: MemRowSet was empty: no flush needed.
I20240502 01:54:03.455602 17333 catalog_manager.cc:5576] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411) does not have the latest schema for tablet 23d171efbc844165944135fc1be2a099 (table rebalancer_dynamic_table [id=56fc003514884047a0449e009700503d]). Expected version 1 got 0
I20240502 01:54:03.476373 17333 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_extra_table_30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:03.485193 20347 tablet_service.cc:1467] Processing CreateTablet for tablet bfd7b1d1002c4b57a14d43ed9c6d9f50 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=9c8c7af6d77740e1a130a1e68baee62d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:03.485657 20347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfd7b1d1002c4b57a14d43ed9c6d9f50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:03.486991 17845 tablet_service.cc:1467] Processing CreateTablet for tablet bfd7b1d1002c4b57a14d43ed9c6d9f50 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=9c8c7af6d77740e1a130a1e68baee62d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:03.487399 17845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfd7b1d1002c4b57a14d43ed9c6d9f50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:03.489183 20476 tablet_service.cc:1467] Processing CreateTablet for tablet b671a9ceacea4a819f750fd37130be0f (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=9c8c7af6d77740e1a130a1e68baee62d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:03.489606 20476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b671a9ceacea4a819f750fd37130be0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:03.489989 17846 tablet_service.cc:1467] Processing CreateTablet for tablet 28463d45bfde44429d56179c1eec56b0 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=9c8c7af6d77740e1a130a1e68baee62d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:03.490317 17846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28463d45bfde44429d56179c1eec56b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:03.490288 20345 tablet_service.cc:1467] Processing CreateTablet for tablet 28463d45bfde44429d56179c1eec56b0 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=9c8c7af6d77740e1a130a1e68baee62d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:03.490698 20345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28463d45bfde44429d56179c1eec56b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:03.491165 20608 tablet_service.cc:1467] Processing CreateTablet for tablet b671a9ceacea4a819f750fd37130be0f (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=9c8c7af6d77740e1a130a1e68baee62d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:03.491581 20608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b671a9ceacea4a819f750fd37130be0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:03.493399 17587 tablet_service.cc:1467] Processing CreateTablet for tablet b671a9ceacea4a819f750fd37130be0f (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=9c8c7af6d77740e1a130a1e68baee62d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:03.493463 20610 tablet_service.cc:1467] Processing CreateTablet for tablet bfd7b1d1002c4b57a14d43ed9c6d9f50 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=9c8c7af6d77740e1a130a1e68baee62d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:03.493839 17587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b671a9ceacea4a819f750fd37130be0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:03.493870 20610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfd7b1d1002c4b57a14d43ed9c6d9f50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:03.496426 20609 tablet_service.cc:1467] Processing CreateTablet for tablet 28463d45bfde44429d56179c1eec56b0 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=9c8c7af6d77740e1a130a1e68baee62d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:03.496788 20609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28463d45bfde44429d56179c1eec56b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:03.503190 23119 tablet_bootstrap.cc:492] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:54:03.504664 23121 tablet_bootstrap.cc:492] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:54:03.504855 23119 tablet_bootstrap.cc:654] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:03.506203 23121 tablet_bootstrap.cc:654] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:03.506548 23144 tablet_bootstrap.cc:492] T b671a9ceacea4a819f750fd37130be0f P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:54:03.508507 23144 tablet_bootstrap.cc:654] T b671a9ceacea4a819f750fd37130be0f P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:03.508690 23119 tablet_bootstrap.cc:492] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:54:03.508847 23119 ts_tablet_manager.cc:1367] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240502 01:54:03.509630 23119 raft_consensus.cc:348] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:03.509955 23119 raft_consensus.cc:374] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:03.510077 23119 raft_consensus.cc:729] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:54:03.510382 23119 consensus_queue.cc:260] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:03.513833 23119 ts_tablet_manager.cc:1398] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 01:54:03.514201 23119 tablet_bootstrap.cc:492] T 28463d45bfde44429d56179c1eec56b0 P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:54:03.517648 23119 tablet_bootstrap.cc:654] T 28463d45bfde44429d56179c1eec56b0 P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:03.531461 23144 tablet_bootstrap.cc:492] T b671a9ceacea4a819f750fd37130be0f P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:54:03.531890 23144 ts_tablet_manager.cc:1367] T b671a9ceacea4a819f750fd37130be0f P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.011s
I20240502 01:54:03.532872 23144 raft_consensus.cc:348] T b671a9ceacea4a819f750fd37130be0f P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:03.533602 23144 raft_consensus.cc:374] T b671a9ceacea4a819f750fd37130be0f P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:03.533751 23144 raft_consensus.cc:729] T b671a9ceacea4a819f750fd37130be0f P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:54:03.534071 23144 consensus_queue.cc:260] T b671a9ceacea4a819f750fd37130be0f P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:03.536880 23121 tablet_bootstrap.cc:492] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:54:03.537138 23121 ts_tablet_manager.cc:1367] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.023s
I20240502 01:54:03.537925 23121 raft_consensus.cc:348] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:03.538213 23121 raft_consensus.cc:374] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:03.538216 23150 tablet_bootstrap.cc:492] T b671a9ceacea4a819f750fd37130be0f P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:54:03.538349 23121 raft_consensus.cc:729] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:54:03.538652 23121 consensus_queue.cc:260] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:03.539657 23144 ts_tablet_manager.cc:1398] T b671a9ceacea4a819f750fd37130be0f P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 01:54:03.542833 23149 tablet_bootstrap.cc:492] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:54:03.543227 22626 raft_consensus.cc:482] T b671a9ceacea4a819f750fd37130be0f P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:03.543365 22626 raft_consensus.cc:504] T b671a9ceacea4a819f750fd37130be0f P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:03.543535 23121 ts_tablet_manager.cc:1398] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20240502 01:54:03.544001 22626 leader_election.cc:288] T b671a9ceacea4a819f750fd37130be0f P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:54:03.544210 23149 tablet_bootstrap.cc:654] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:03.544649 20629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b671a9ceacea4a819f750fd37130be0f" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091" is_pre_election: true
I20240502 01:54:03.545382 23150 tablet_bootstrap.cc:654] T b671a9ceacea4a819f750fd37130be0f P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
W20240502 01:54:03.545696 20433 leader_election.cc:341] T b671a9ceacea4a819f750fd37130be0f P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:03.545676 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b671a9ceacea4a819f750fd37130be0f" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:54:03.546193 23119 tablet_bootstrap.cc:492] T 28463d45bfde44429d56179c1eec56b0 P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:54:03.546334 23119 ts_tablet_manager.cc:1367] T 28463d45bfde44429d56179c1eec56b0 P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.019s
W20240502 01:54:03.546666 20431 leader_election.cc:341] T b671a9ceacea4a819f750fd37130be0f P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:03.546847 20431 leader_election.cc:302] T b671a9ceacea4a819f750fd37130be0f P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1; no voters: 3fceccefba274b1dbbd8216af8b0bc29, f19cc6cb1e76460aa6711e49a5775091
I20240502 01:54:03.547215 23119 raft_consensus.cc:348] T 28463d45bfde44429d56179c1eec56b0 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:03.547309 22626 raft_consensus.cc:2726] T b671a9ceacea4a819f750fd37130be0f P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:54:03.547492 23119 raft_consensus.cc:374] T 28463d45bfde44429d56179c1eec56b0 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:03.547632 23119 raft_consensus.cc:729] T 28463d45bfde44429d56179c1eec56b0 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:54:03.547891 23119 consensus_queue.cc:260] T 28463d45bfde44429d56179c1eec56b0 P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:03.549463 23121 tablet_bootstrap.cc:492] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:54:03.549724 23119 ts_tablet_manager.cc:1398] T 28463d45bfde44429d56179c1eec56b0 P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:54:03.550072 23119 tablet_bootstrap.cc:492] T b671a9ceacea4a819f750fd37130be0f P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:54:03.551929 17330 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:54:03.552436 23149 tablet_bootstrap.cc:492] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:54:03.552551 23149 ts_tablet_manager.cc:1367] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 01:54:03.551648 23119 tablet_bootstrap.cc:654] T b671a9ceacea4a819f750fd37130be0f P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:03.554984 23121 tablet_bootstrap.cc:654] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:03.556260 23149 raft_consensus.cc:348] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:03.556528 23149 raft_consensus.cc:374] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:03.556623 23149 raft_consensus.cc:729] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:54:03.556972 23149 consensus_queue.cc:260] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:03.558687 23149 ts_tablet_manager.cc:1398] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:54:03.559019 23149 tablet_bootstrap.cc:492] T 28463d45bfde44429d56179c1eec56b0 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:54:03.560369 23149 tablet_bootstrap.cc:654] T 28463d45bfde44429d56179c1eec56b0 P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:03.569324 23003 tablet.cc:1722] T 23d171efbc844165944135fc1be2a099 P dfa1152987644bd781b46c7e42a7a685: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:03.569568 23121 tablet_bootstrap.cc:492] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:54:03.569710 23121 ts_tablet_manager.cc:1367] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.008s
I20240502 01:54:03.569875 23003 tablet.cc:1620] T 23d171efbc844165944135fc1be2a099 P dfa1152987644bd781b46c7e42a7a685: MemRowSet was empty: no flush needed.
I20240502 01:54:03.569945 23142 tablet.cc:1722] T 23d171efbc844165944135fc1be2a099 P 6a36c952c51542c0bf7ea3ec2857a9f9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:03.570398 23142 tablet.cc:1620] T 23d171efbc844165944135fc1be2a099 P 6a36c952c51542c0bf7ea3ec2857a9f9: MemRowSet was empty: no flush needed.
I20240502 01:54:03.572674 23143 tablet.cc:1722] T 23d171efbc844165944135fc1be2a099 P f19cc6cb1e76460aa6711e49a5775091: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:03.573122 23143 tablet.cc:1620] T 23d171efbc844165944135fc1be2a099 P f19cc6cb1e76460aa6711e49a5775091: MemRowSet was empty: no flush needed.
I20240502 01:54:03.577127 23121 raft_consensus.cc:348] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:03.577320 23121 raft_consensus.cc:374] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:03.577394 23121 raft_consensus.cc:729] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:54:03.577574 23121 consensus_queue.cc:260] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:03.580001 23121 ts_tablet_manager.cc:1398] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240502 01:54:03.584734 17330 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=56fc003514884047a0449e009700503d] alter complete (version 2)
I20240502 01:54:03.586232 23150 tablet_bootstrap.cc:492] T b671a9ceacea4a819f750fd37130be0f P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:54:03.586447 23150 ts_tablet_manager.cc:1367] T b671a9ceacea4a819f750fd37130be0f P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.048s user 0.005s sys 0.000s
I20240502 01:54:03.587184 23150 raft_consensus.cc:348] T b671a9ceacea4a819f750fd37130be0f P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:03.587481 23150 raft_consensus.cc:374] T b671a9ceacea4a819f750fd37130be0f P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:03.587592 23150 raft_consensus.cc:729] T b671a9ceacea4a819f750fd37130be0f P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:54:03.588068 23150 consensus_queue.cc:260] T b671a9ceacea4a819f750fd37130be0f P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:03.588732 23150 ts_tablet_manager.cc:1398] T b671a9ceacea4a819f750fd37130be0f P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:54:03.592604 23149 tablet_bootstrap.cc:492] T 28463d45bfde44429d56179c1eec56b0 P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:54:03.592770 23149 ts_tablet_manager.cc:1367] T 28463d45bfde44429d56179c1eec56b0 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.002s
I20240502 01:54:03.593466 23149 raft_consensus.cc:348] T 28463d45bfde44429d56179c1eec56b0 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:03.593654 23149 raft_consensus.cc:374] T 28463d45bfde44429d56179c1eec56b0 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:03.593758 23149 raft_consensus.cc:729] T 28463d45bfde44429d56179c1eec56b0 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:54:03.593986 23149 consensus_queue.cc:260] T 28463d45bfde44429d56179c1eec56b0 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:03.594625 23149 ts_tablet_manager.cc:1398] T 28463d45bfde44429d56179c1eec56b0 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:54:03.600513 22177 raft_consensus.cc:1050] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3: attempting to promote NON_VOTER 2f306549407b4bb281ccf2994035e4f1 to VOTER
I20240502 01:54:03.601317 22177 consensus_queue.cc:237] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } }
I20240502 01:54:03.604321 17734 raft_consensus.cc:1261] T 377f8b554ad1448d981d2283b865f8fe P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:03.605090 20496 raft_consensus.cc:1261] T 377f8b554ad1448d981d2283b865f8fe P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:03.605394 17604 raft_consensus.cc:1261] T 377f8b554ad1448d981d2283b865f8fe P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:03.606563 22527 consensus_queue.cc:1035] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:03.607012 22527 consensus_queue.cc:1035] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:03.607430 22527 consensus_queue.cc:1035] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:03.611260 22568 raft_consensus.cc:1050] T b29218a879e74895bfe00021ad358aa4 P dfa1152987644bd781b46c7e42a7a685: attempting to promote NON_VOTER 2f306549407b4bb281ccf2994035e4f1 to VOTER
I20240502 01:54:03.612111 22568 consensus_queue.cc:237] T b29218a879e74895bfe00021ad358aa4 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } }
I20240502 01:54:03.625540 20494 raft_consensus.cc:1261] T b29218a879e74895bfe00021ad358aa4 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:03.626262 22816 consensus_queue.cc:1035] T b29218a879e74895bfe00021ad358aa4 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:03.634235 23119 tablet_bootstrap.cc:492] T b671a9ceacea4a819f750fd37130be0f P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:54:03.634517 22177 raft_consensus.cc:2932] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:03.636374 17604 raft_consensus.cc:1261] T b29218a879e74895bfe00021ad358aa4 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:03.636862 22816 consensus_queue.cc:1035] T b29218a879e74895bfe00021ad358aa4 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:03.638461 17473 raft_consensus.cc:1261] T b29218a879e74895bfe00021ad358aa4 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:03.639052 22977 consensus_queue.cc:1035] T b29218a879e74895bfe00021ad358aa4 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:03.639335 17605 raft_consensus.cc:2932] T 377f8b554ad1448d981d2283b865f8fe P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:03.645853 17334 catalog_manager.cc:5484] T 377f8b554ad1448d981d2283b865f8fe P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: config changed from index 4 to 5, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:03.646574 23119 ts_tablet_manager.cc:1367] T b671a9ceacea4a819f750fd37130be0f P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.097s user 0.014s sys 0.021s
I20240502 01:54:03.647490 23119 raft_consensus.cc:348] T b671a9ceacea4a819f750fd37130be0f P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:03.647953 23119 raft_consensus.cc:374] T b671a9ceacea4a819f750fd37130be0f P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:03.648083 23119 raft_consensus.cc:729] T b671a9ceacea4a819f750fd37130be0f P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:54:03.648456 23119 consensus_queue.cc:260] T b671a9ceacea4a819f750fd37130be0f P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:03.649320 23119 ts_tablet_manager.cc:1398] T b671a9ceacea4a819f750fd37130be0f P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:54:03.648854 20494 raft_consensus.cc:2932] T 377f8b554ad1448d981d2283b865f8fe P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:03.660887 22568 raft_consensus.cc:2932] T b29218a879e74895bfe00021ad358aa4 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:03.663594 17864 consensus_queue.cc:237] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } }
I20240502 01:54:03.664675 17734 raft_consensus.cc:2932] T 377f8b554ad1448d981d2283b865f8fe P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:03.665498 17604 raft_consensus.cc:1261] T 377f8b554ad1448d981d2283b865f8fe P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:03.665853 20494 raft_consensus.cc:1261] T 377f8b554ad1448d981d2283b865f8fe P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:03.665942 22527 consensus_queue.cc:1035] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:03.666386 22527 consensus_queue.cc:1035] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:03.669297 22876 raft_consensus.cc:482] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:03.669548 22876 raft_consensus.cc:504] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:03.669536 22527 raft_consensus.cc:2932] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:03.672355 17864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28463d45bfde44429d56179c1eec56b0" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:54:03.672529 17864 raft_consensus.cc:2454] T 28463d45bfde44429d56179c1eec56b0 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 0.
I20240502 01:54:03.672931 20629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28463d45bfde44429d56179c1eec56b0" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091" is_pre_election: true
I20240502 01:54:03.672982 20303 leader_election.cc:302] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:54:03.673188 20629 raft_consensus.cc:2454] T 28463d45bfde44429d56179c1eec56b0 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 0.
I20240502 01:54:03.673355 22863 raft_consensus.cc:2781] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:03.673470 22863 raft_consensus.cc:482] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:03.673553 22863 raft_consensus.cc:3037] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:03.674216 22876 leader_election.cc:288] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:54:03.674245 17605 raft_consensus.cc:2932] T 377f8b554ad1448d981d2283b865f8fe P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:03.674353 20496 raft_consensus.cc:2932] T b29218a879e74895bfe00021ad358aa4 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:03.675123 20494 raft_consensus.cc:2932] T 377f8b554ad1448d981d2283b865f8fe P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:03.676230 22863 raft_consensus.cc:504] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:03.676820 22863 leader_election.cc:288] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Requested vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:54:03.677243 17864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28463d45bfde44429d56179c1eec56b0" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:54:03.677414 17864 raft_consensus.cc:3037] T 28463d45bfde44429d56179c1eec56b0 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:03.677623 20629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28463d45bfde44429d56179c1eec56b0" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091"
I20240502 01:54:03.677803 20629 raft_consensus.cc:3037] T 28463d45bfde44429d56179c1eec56b0 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:03.680182 17864 raft_consensus.cc:2454] T 28463d45bfde44429d56179c1eec56b0 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 1.
I20240502 01:54:03.680665 20629 raft_consensus.cc:2454] T 28463d45bfde44429d56179c1eec56b0 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 1.
I20240502 01:54:03.681038 20302 leader_election.cc:302] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:54:03.681365 22863 raft_consensus.cc:2781] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:03.681502 22863 raft_consensus.cc:686] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Becoming Leader. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Running, Role: LEADER
I20240502 01:54:03.681785 22863 consensus_queue.cc:237] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:03.681813 17606 raft_consensus.cc:2932] T b29218a879e74895bfe00021ad358aa4 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:03.683883 17333 catalog_manager.cc:5484] T b29218a879e74895bfe00021ad358aa4 P dfa1152987644bd781b46c7e42a7a685 reported cstate change: config changed from index 4 to 5, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:03.685190 22991 raft_consensus.cc:482] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:03.685336 22991 raft_consensus.cc:504] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:03.686208 22991 leader_election.cc:288] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:54:03.688575 17473 raft_consensus.cc:2932] T b29218a879e74895bfe00021ad358aa4 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:03.689682 20366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfd7b1d1002c4b57a14d43ed9c6d9f50" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:54:03.689889 20366 raft_consensus.cc:2454] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 0.
I20240502 01:54:03.691139 17327 catalog_manager.cc:5484] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70). New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:03.691845 20564 leader_election.cc:302] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:54:03.696338 17734 consensus_queue.cc:237] T b29218a879e74895bfe00021ad358aa4 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } }
I20240502 01:54:03.696703 17330 catalog_manager.cc:5484] T 377f8b554ad1448d981d2283b865f8fe P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: config changed from index 5 to 6, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:03.697376 22940 raft_consensus.cc:2781] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:03.697468 22940 raft_consensus.cc:482] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:03.697566 22940 raft_consensus.cc:3037] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:03.699306 17322 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 377f8b554ad1448d981d2283b865f8fe with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:03.699961 22940 raft_consensus.cc:504] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:03.700599 22940 leader_election.cc:288] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 election: Requested vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:54:03.701889 17866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfd7b1d1002c4b57a14d43ed9c6d9f50" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:54:03.702069 17866 raft_consensus.cc:3037] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:03.702368 17864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfd7b1d1002c4b57a14d43ed9c6d9f50" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:54:03.704599 17866 raft_consensus.cc:2454] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 1.
I20240502 01:54:03.705246 20565 leader_election.cc:302] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb31bceba76d466db24b9341c04900c3, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:54:03.705752 17333 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:54:03.707903 17716 tablet_service.cc:1514] Processing DeleteTablet for tablet 377f8b554ad1448d981d2283b865f8fe with delete_type TABLET_DATA_TOMBSTONED (TS dfa1152987644bd781b46c7e42a7a685 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52198
I20240502 01:54:03.709177 20366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfd7b1d1002c4b57a14d43ed9c6d9f50" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
I20240502 01:54:03.709414 20366 raft_consensus.cc:3037] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:03.711516 20494 raft_consensus.cc:1261] T b29218a879e74895bfe00021ad358aa4 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:03.711673 20366 raft_consensus.cc:2454] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 1.
I20240502 01:54:03.712054 22991 raft_consensus.cc:2781] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:03.712198 22991 raft_consensus.cc:686] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Becoming Leader. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Running, Role: LEADER
I20240502 01:54:03.712546 22991 consensus_queue.cc:237] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:03.715797 22816 consensus_queue.cc:1035] T b29218a879e74895bfe00021ad358aa4 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:03.716401 22940 raft_consensus.cc:482] T b671a9ceacea4a819f750fd37130be0f P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:03.716570 22940 raft_consensus.cc:504] T b671a9ceacea4a819f750fd37130be0f P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:03.717474 17605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b671a9ceacea4a819f750fd37130be0f" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:54:03.717722 17605 raft_consensus.cc:2454] T b671a9ceacea4a819f750fd37130be0f P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 0.
I20240502 01:54:03.718154 20562 leader_election.cc:302] T b671a9ceacea4a819f750fd37130be0f P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:54:03.718590 20496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b671a9ceacea4a819f750fd37130be0f" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
I20240502 01:54:03.718834 20496 raft_consensus.cc:2454] T b671a9ceacea4a819f750fd37130be0f P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 0.
I20240502 01:54:03.719671 22977 raft_consensus.cc:2932] T b29218a879e74895bfe00021ad358aa4 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:03.721164 22940 leader_election.cc:288] T b671a9ceacea4a819f750fd37130be0f P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:54:03.721343 22940 raft_consensus.cc:2781] T b671a9ceacea4a819f750fd37130be0f P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:03.721413 22940 raft_consensus.cc:482] T b671a9ceacea4a819f750fd37130be0f P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:03.721530 22940 raft_consensus.cc:3037] T b671a9ceacea4a819f750fd37130be0f P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:03.723575 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b29218a879e74895bfe00021ad358aa4 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:03.724038 20496 raft_consensus.cc:2932] T b29218a879e74895bfe00021ad358aa4 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:03.724261 22940 raft_consensus.cc:504] T b671a9ceacea4a819f750fd37130be0f P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:03.724843 22940 leader_election.cc:288] T b671a9ceacea4a819f750fd37130be0f P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 election: Requested vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:54:03.725519 17473 raft_consensus.cc:1261] T b29218a879e74895bfe00021ad358aa4 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:03.732601 22977 consensus_queue.cc:1035] T b29218a879e74895bfe00021ad358aa4 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:03.735939 23003 tablet.cc:1722] T 23d171efbc844165944135fc1be2a099 P dfa1152987644bd781b46c7e42a7a685: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:03.738135 17333 catalog_manager.cc:5484] T b29218a879e74895bfe00021ad358aa4 P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: config changed from index 5 to 6, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:03.738476 17473 raft_consensus.cc:2932] T b29218a879e74895bfe00021ad358aa4 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:03.740960 23142 tablet.cc:1722] T 23d171efbc844165944135fc1be2a099 P 6a36c952c51542c0bf7ea3ec2857a9f9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:03.745630 20496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b671a9ceacea4a819f750fd37130be0f" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
I20240502 01:54:03.745968 20496 raft_consensus.cc:3037] T b671a9ceacea4a819f750fd37130be0f P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:03.749970 17866 raft_consensus.cc:1261] T aebde87a7f114537abd9c1d8b7e97004 P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:54:03.750569 22863 consensus_queue.cc:1035] T aebde87a7f114537abd9c1d8b7e97004 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:03.754374 20496 raft_consensus.cc:2454] T b671a9ceacea4a819f750fd37130be0f P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 1.
I20240502 01:54:03.754858 20565 leader_election.cc:302] T b671a9ceacea4a819f750fd37130be0f P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:54:03.755173 22940 raft_consensus.cc:2781] T b671a9ceacea4a819f750fd37130be0f P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:03.755537 22940 raft_consensus.cc:686] T b671a9ceacea4a819f750fd37130be0f P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Becoming Leader. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Running, Role: LEADER
I20240502 01:54:03.756043 17330 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table_new_name [id=56fc003514884047a0449e009700503d] alter complete (version 3)
I20240502 01:54:03.757331 22940 consensus_queue.cc:237] T b671a9ceacea4a819f750fd37130be0f P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:03.759218 17330 catalog_manager.cc:5484] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: term changed from 0 to 1, leader changed from <none> to f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72). New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:03.759851 17587 tablet_service.cc:1514] Processing DeleteTablet for tablet b29218a879e74895bfe00021ad358aa4 with delete_type TABLET_DATA_TOMBSTONED (TS 3fceccefba274b1dbbd8216af8b0bc29 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55232
I20240502 01:54:03.761312 23184 tablet_replica.cc:330] T b29218a879e74895bfe00021ad358aa4 P 3fceccefba274b1dbbd8216af8b0bc29: stopping tablet replica
I20240502 01:54:03.761559 23184 raft_consensus.cc:2229] T b29218a879e74895bfe00021ad358aa4 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:03.762006 23184 raft_consensus.cc:2258] T b29218a879e74895bfe00021ad358aa4 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:03.764942 23184 ts_tablet_manager.cc:1875] T b29218a879e74895bfe00021ad358aa4 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:03.774577 23184 ts_tablet_manager.cc:1888] T b29218a879e74895bfe00021ad358aa4 P 3fceccefba274b1dbbd8216af8b0bc29: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:03.774785 23184 log.cc:1196] T b29218a879e74895bfe00021ad358aa4 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/wals/b29218a879e74895bfe00021ad358aa4
I20240502 01:54:03.776049 17319 catalog_manager.cc:4830] TS 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): tablet b29218a879e74895bfe00021ad358aa4 (table rebalance_test_table_11 [id=5e2315325f6c4e7f8834471d07555cf5]) successfully deleted
W20240502 01:54:03.777551 20562 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.42.66:38531, user_credentials={real_user=slave}} blocked reactor thread for 59401.1us
I20240502 01:54:03.777786 23177 tablet_replica.cc:330] T 377f8b554ad1448d981d2283b865f8fe P dfa1152987644bd781b46c7e42a7a685: stopping tablet replica
I20240502 01:54:03.778003 23177 raft_consensus.cc:2229] T 377f8b554ad1448d981d2283b865f8fe P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:03.778244 23177 raft_consensus.cc:2258] T 377f8b554ad1448d981d2283b865f8fe P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:03.778151 17605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b671a9ceacea4a819f750fd37130be0f" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:54:03.778345 17605 raft_consensus.cc:3037] T b671a9ceacea4a819f750fd37130be0f P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:03.780800 23177 ts_tablet_manager.cc:1875] T 377f8b554ad1448d981d2283b865f8fe P dfa1152987644bd781b46c7e42a7a685: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:03.785722 17605 raft_consensus.cc:2454] T b671a9ceacea4a819f750fd37130be0f P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 1.
I20240502 01:54:03.786231 17333 catalog_manager.cc:5484] T b671a9ceacea4a819f750fd37130be0f P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: term changed from 0 to 1, leader changed from <none> to f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72). New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:03.794080 23177 ts_tablet_manager.cc:1888] T 377f8b554ad1448d981d2283b865f8fe P dfa1152987644bd781b46c7e42a7a685: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:03.794409 23177 log.cc:1196] T 377f8b554ad1448d981d2283b865f8fe P dfa1152987644bd781b46c7e42a7a685: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/wals/377f8b554ad1448d981d2283b865f8fe
I20240502 01:54:03.794708 23143 tablet.cc:1722] T 23d171efbc844165944135fc1be2a099 P f19cc6cb1e76460aa6711e49a5775091: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:03.795212 17321 catalog_manager.cc:4830] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): tablet 377f8b554ad1448d981d2283b865f8fe (table rebalance_test_table_5 [id=1567095c8de34d6a98c56d3bb926b668]) successfully deleted
I20240502 01:54:03.796514 23162 raft_consensus.cc:1050] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3: attempting to promote NON_VOTER fdc2ac372492454c85387589a91f63af to VOTER
I20240502 01:54:03.797329 23162 consensus_queue.cc:237] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } }
I20240502 01:54:03.808120 22568 raft_consensus.cc:1050] T 4099f4b2ae164020839faab53957750f P dfa1152987644bd781b46c7e42a7a685: attempting to promote NON_VOTER f19cc6cb1e76460aa6711e49a5775091 to VOTER
I20240502 01:54:03.808964 22568 consensus_queue.cc:237] T 4099f4b2ae164020839faab53957750f P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } }
I20240502 01:54:03.812888 20233 raft_consensus.cc:1261] T ed1522ec150a46f28fd553f9b563651f P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:03.814131 22527 consensus_queue.cc:1035] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:03.821501 20629 raft_consensus.cc:1261] T 4099f4b2ae164020839faab53957750f P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:03.821548 17605 raft_consensus.cc:1261] T 4099f4b2ae164020839faab53957750f P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:03.822057 22816 consensus_queue.cc:1035] T 4099f4b2ae164020839faab53957750f P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:03.823982 17734 raft_consensus.cc:1261] T ed1522ec150a46f28fd553f9b563651f P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:03.824077 22816 consensus_queue.cc:1035] T 4099f4b2ae164020839faab53957750f P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:03.824352 17473 raft_consensus.cc:1261] T ed1522ec150a46f28fd553f9b563651f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:03.826031 22527 consensus_queue.cc:1035] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:03.826509 22527 consensus_queue.cc:1035] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:03.831197 17474 raft_consensus.cc:1261] T 4099f4b2ae164020839faab53957750f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:03.831786 22816 consensus_queue.cc:1035] T 4099f4b2ae164020839faab53957750f P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:03.836181 23182 raft_consensus.cc:2932] T 4099f4b2ae164020839faab53957750f P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:03.840426 17604 raft_consensus.cc:2932] T 4099f4b2ae164020839faab53957750f P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:03.846633 20496 raft_consensus.cc:1261] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:54:03.847488 22863 consensus_queue.cc:1035] T aebde87a7f114537abd9c1d8b7e97004 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:03.848462 22177 raft_consensus.cc:2932] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:03.850240 17333 catalog_manager.cc:5484] T 4099f4b2ae164020839faab53957750f P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: config changed from index 4 to 5, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:03.852721 20629 raft_consensus.cc:2932] T 4099f4b2ae164020839faab53957750f P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:03.855437 17330 catalog_manager.cc:5484] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 reported cstate change: config changed from index 4 to 5, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:03.861428 17330 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:03.861640 17330 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:03.864859 20233 raft_consensus.cc:2932] T ed1522ec150a46f28fd553f9b563651f P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:03.866343 17330 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet 23d171efbc844165944135fc1be2a099
I20240502 01:54:03.867893 17473 raft_consensus.cc:2932] T ed1522ec150a46f28fd553f9b563651f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:03.868826 20347 tablet_service.cc:1514] Processing DeleteTablet for tablet 23d171efbc844165944135fc1be2a099 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:03 UTC) from {username='slave'} at 127.0.0.1:40196
I20240502 01:54:03.869391 20608 tablet_service.cc:1514] Processing DeleteTablet for tablet 23d171efbc844165944135fc1be2a099 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:03 UTC) from {username='slave'} at 127.0.0.1:33698
I20240502 01:54:03.869540 23199 tablet_replica.cc:330] T 23d171efbc844165944135fc1be2a099 P 6a36c952c51542c0bf7ea3ec2857a9f9: stopping tablet replica
I20240502 01:54:03.869745 23199 raft_consensus.cc:2229] T 23d171efbc844165944135fc1be2a099 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:03.870103 23199 raft_consensus.cc:2258] T 23d171efbc844165944135fc1be2a099 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:03.871277 23199 ts_tablet_manager.cc:1875] T 23d171efbc844165944135fc1be2a099 P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:03.871523 17866 consensus_queue.cc:237] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } }
I20240502 01:54:03.880285 23199 ts_tablet_manager.cc:1888] T 23d171efbc844165944135fc1be2a099 P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:03.880753 23199 log.cc:1196] T 23d171efbc844165944135fc1be2a099 P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal/wals/23d171efbc844165944135fc1be2a099
I20240502 01:54:03.881289 23199 ts_tablet_manager.cc:1909] T 23d171efbc844165944135fc1be2a099 P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting consensus metadata
I20240502 01:54:03.882310 17321 catalog_manager.cc:4830] TS 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): tablet 23d171efbc844165944135fc1be2a099 (table rebalancer_dynamic_table_new_name [id=56fc003514884047a0449e009700503d]) successfully deleted
I20240502 01:54:03.885115 20231 raft_consensus.cc:1261] T ed1522ec150a46f28fd553f9b563651f P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:03.887203 17475 raft_consensus.cc:1261] T ed1522ec150a46f28fd553f9b563651f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:03.888026 22177 consensus_queue.cc:1035] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:03.888638 22177 consensus_queue.cc:1035] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:03.890480 17737 raft_consensus.cc:2932] T ed1522ec150a46f28fd553f9b563651f P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:03.891484 22527 raft_consensus.cc:2932] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:03.890511 17716 tablet_service.cc:1514] Processing DeleteTablet for tablet 23d171efbc844165944135fc1be2a099 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:03 UTC) from {username='slave'} at 127.0.0.1:52198
I20240502 01:54:03.892113 17475 raft_consensus.cc:2932] T ed1522ec150a46f28fd553f9b563651f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:03.894227 23177 tablet_replica.cc:330] T 23d171efbc844165944135fc1be2a099 P dfa1152987644bd781b46c7e42a7a685: stopping tablet replica
I20240502 01:54:03.894443 23177 raft_consensus.cc:2229] T 23d171efbc844165944135fc1be2a099 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:03.894733 23177 raft_consensus.cc:2258] T 23d171efbc844165944135fc1be2a099 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:03.897567 17322 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ed1522ec150a46f28fd553f9b563651f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:03.899673 17333 catalog_manager.cc:5484] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 reported cstate change: config changed from index 5 to 6, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:03.901612 17734 raft_consensus.cc:1261] T aebde87a7f114537abd9c1d8b7e97004 P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:54:03.904531 17716 tablet_service.cc:1514] Processing DeleteTablet for tablet ed1522ec150a46f28fd553f9b563651f with delete_type TABLET_DATA_TOMBSTONED (TS dfa1152987644bd781b46c7e42a7a685 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52198
I20240502 01:54:03.905735 23200 tablet_replica.cc:330] T 23d171efbc844165944135fc1be2a099 P f19cc6cb1e76460aa6711e49a5775091: stopping tablet replica
I20240502 01:54:03.906020 22863 consensus_queue.cc:1035] T aebde87a7f114537abd9c1d8b7e97004 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:03.908149 20233 raft_consensus.cc:2932] T ed1522ec150a46f28fd553f9b563651f P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:03.910702 23177 ts_tablet_manager.cc:1875] T 23d171efbc844165944135fc1be2a099 P dfa1152987644bd781b46c7e42a7a685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:03.920890 23177 ts_tablet_manager.cc:1888] T 23d171efbc844165944135fc1be2a099 P dfa1152987644bd781b46c7e42a7a685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:03.921710 23200 raft_consensus.cc:2229] T 23d171efbc844165944135fc1be2a099 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:03.922077 23200 raft_consensus.cc:2258] T 23d171efbc844165944135fc1be2a099 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:03.922267 23177 log.cc:1196] T 23d171efbc844165944135fc1be2a099 P dfa1152987644bd781b46c7e42a7a685: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/wals/23d171efbc844165944135fc1be2a099
I20240502 01:54:03.922925 23177 ts_tablet_manager.cc:1909] T 23d171efbc844165944135fc1be2a099 P dfa1152987644bd781b46c7e42a7a685: Deleting consensus metadata
I20240502 01:54:03.923831 23200 ts_tablet_manager.cc:1875] T 23d171efbc844165944135fc1be2a099 P f19cc6cb1e76460aa6711e49a5775091: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:03.931836 23177 tablet_replica.cc:330] T ed1522ec150a46f28fd553f9b563651f P dfa1152987644bd781b46c7e42a7a685: stopping tablet replica
I20240502 01:54:03.932013 17321 catalog_manager.cc:4830] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): tablet 23d171efbc844165944135fc1be2a099 (table rebalancer_dynamic_table_new_name [id=56fc003514884047a0449e009700503d]) successfully deleted
I20240502 01:54:03.932078 23177 raft_consensus.cc:2229] T ed1522ec150a46f28fd553f9b563651f P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:03.932451 23177 raft_consensus.cc:2258] T ed1522ec150a46f28fd553f9b563651f P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:03.933432 23177 ts_tablet_manager.cc:1875] T ed1522ec150a46f28fd553f9b563651f P dfa1152987644bd781b46c7e42a7a685: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:03.939759 17473 raft_consensus.cc:2932] T 4099f4b2ae164020839faab53957750f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:03.944846 17737 consensus_queue.cc:237] T 4099f4b2ae164020839faab53957750f P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } }
I20240502 01:54:03.946852 20629 raft_consensus.cc:1261] T 4099f4b2ae164020839faab53957750f P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:03.947656 23182 consensus_queue.cc:1035] T 4099f4b2ae164020839faab53957750f P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:03.949988 23197 raft_consensus.cc:2932] T 4099f4b2ae164020839faab53957750f P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:03.950778 20629 raft_consensus.cc:2932] T 4099f4b2ae164020839faab53957750f P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:03.952837 17474 raft_consensus.cc:1261] T 4099f4b2ae164020839faab53957750f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:03.953363 22568 consensus_queue.cc:1035] T 4099f4b2ae164020839faab53957750f P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:03.954799 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4099f4b2ae164020839faab53957750f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:03.958657 17330 catalog_manager.cc:5484] T 4099f4b2ae164020839faab53957750f P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: config changed from index 5 to 6, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:03.962253 17473 raft_consensus.cc:2932] T 4099f4b2ae164020839faab53957750f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:03.963276 17587 tablet_service.cc:1514] Processing DeleteTablet for tablet 4099f4b2ae164020839faab53957750f with delete_type TABLET_DATA_TOMBSTONED (TS 3fceccefba274b1dbbd8216af8b0bc29 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55232
I20240502 01:54:03.963624 23184 tablet_replica.cc:330] T 4099f4b2ae164020839faab53957750f P 3fceccefba274b1dbbd8216af8b0bc29: stopping tablet replica
I20240502 01:54:03.963819 23184 raft_consensus.cc:2229] T 4099f4b2ae164020839faab53957750f P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:03.964066 23184 raft_consensus.cc:2258] T 4099f4b2ae164020839faab53957750f P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:03.965126 23184 ts_tablet_manager.cc:1875] T 4099f4b2ae164020839faab53957750f P 3fceccefba274b1dbbd8216af8b0bc29: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:03.967924 23177 ts_tablet_manager.cc:1888] T ed1522ec150a46f28fd553f9b563651f P dfa1152987644bd781b46c7e42a7a685: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:03.968034 23177 log.cc:1196] T ed1522ec150a46f28fd553f9b563651f P dfa1152987644bd781b46c7e42a7a685: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/wals/ed1522ec150a46f28fd553f9b563651f
I20240502 01:54:03.968700 17321 catalog_manager.cc:4830] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): tablet ed1522ec150a46f28fd553f9b563651f (table rebalance_test_table_11 [id=5e2315325f6c4e7f8834471d07555cf5]) successfully deleted
I20240502 01:54:03.963076 23200 ts_tablet_manager.cc:1888] T 23d171efbc844165944135fc1be2a099 P f19cc6cb1e76460aa6711e49a5775091: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:03.971010 23200 log.cc:1196] T 23d171efbc844165944135fc1be2a099 P f19cc6cb1e76460aa6711e49a5775091: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal/wals/23d171efbc844165944135fc1be2a099
I20240502 01:54:03.971557 23200 ts_tablet_manager.cc:1909] T 23d171efbc844165944135fc1be2a099 P f19cc6cb1e76460aa6711e49a5775091: Deleting consensus metadata
I20240502 01:54:03.972719 17321 catalog_manager.cc:4830] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): tablet 23d171efbc844165944135fc1be2a099 (table rebalancer_dynamic_table_new_name [id=56fc003514884047a0449e009700503d]) successfully deleted
I20240502 01:54:03.973099 17327 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:54:03.975821 23184 ts_tablet_manager.cc:1888] T 4099f4b2ae164020839faab53957750f P 3fceccefba274b1dbbd8216af8b0bc29: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:03.975931 23184 log.cc:1196] T 4099f4b2ae164020839faab53957750f P 3fceccefba274b1dbbd8216af8b0bc29: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/wals/4099f4b2ae164020839faab53957750f
I20240502 01:54:03.976622 17319 catalog_manager.cc:4830] TS 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): tablet 4099f4b2ae164020839faab53957750f (table rebalance_test_table_19 [id=b69eb497d10340fea8df98ad03a1e465]) successfully deleted
I20240502 01:54:03.984143 17716 tablet_service.cc:1467] Processing CreateTablet for tablet 3d51061767744aecbe46917043e05a7c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=537bb6fadbef4eae9de6f5cffebb0358]), partition=
I20240502 01:54:03.984503 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d51061767744aecbe46917043e05a7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:03.985051 17587 tablet_service.cc:1467] Processing CreateTablet for tablet 3d51061767744aecbe46917043e05a7c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=537bb6fadbef4eae9de6f5cffebb0358]), partition=
I20240502 01:54:03.985328 17587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d51061767744aecbe46917043e05a7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:03.988348 17456 tablet_service.cc:1467] Processing CreateTablet for tablet 3d51061767744aecbe46917043e05a7c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=537bb6fadbef4eae9de6f5cffebb0358]), partition=
I20240502 01:54:03.988691 17456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d51061767744aecbe46917043e05a7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:03.991359 23150 tablet_bootstrap.cc:492] T 3d51061767744aecbe46917043e05a7c P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:54:03.992578 23150 tablet_bootstrap.cc:654] T 3d51061767744aecbe46917043e05a7c P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:03.994040 23210 tablet_bootstrap.cc:492] T 3d51061767744aecbe46917043e05a7c P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:54:03.995396 23210 tablet_bootstrap.cc:654] T 3d51061767744aecbe46917043e05a7c P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:04.005506 23150 tablet_bootstrap.cc:492] T 3d51061767744aecbe46917043e05a7c P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:54:04.005694 23150 ts_tablet_manager.cc:1367] T 3d51061767744aecbe46917043e05a7c P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.009s
I20240502 01:54:04.006451 23150 raft_consensus.cc:348] T 3d51061767744aecbe46917043e05a7c P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:04.006675 23150 raft_consensus.cc:374] T 3d51061767744aecbe46917043e05a7c P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:04.006750 23150 raft_consensus.cc:729] T 3d51061767744aecbe46917043e05a7c P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:54:04.006996 23150 consensus_queue.cc:260] T 3d51061767744aecbe46917043e05a7c P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:04.007529 23150 ts_tablet_manager.cc:1398] T 3d51061767744aecbe46917043e05a7c P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:54:04.018544 23213 tablet_bootstrap.cc:492] T 3d51061767744aecbe46917043e05a7c P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:54:04.021276 23213 tablet_bootstrap.cc:654] T 3d51061767744aecbe46917043e05a7c P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:04.023455 23210 tablet_bootstrap.cc:492] T 3d51061767744aecbe46917043e05a7c P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:54:04.023671 23210 ts_tablet_manager.cc:1367] T 3d51061767744aecbe46917043e05a7c P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.020s
I20240502 01:54:04.024436 23210 raft_consensus.cc:348] T 3d51061767744aecbe46917043e05a7c P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:04.024832 23210 raft_consensus.cc:374] T 3d51061767744aecbe46917043e05a7c P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:04.024935 23210 raft_consensus.cc:729] T 3d51061767744aecbe46917043e05a7c P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:54:04.025245 23210 consensus_queue.cc:260] T 3d51061767744aecbe46917043e05a7c P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:04.025954 23210 ts_tablet_manager.cc:1398] T 3d51061767744aecbe46917043e05a7c P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:54:04.027436 23213 tablet_bootstrap.cc:492] T 3d51061767744aecbe46917043e05a7c P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:54:04.027590 23213 ts_tablet_manager.cc:1367] T 3d51061767744aecbe46917043e05a7c P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240502 01:54:04.028369 23213 raft_consensus.cc:348] T 3d51061767744aecbe46917043e05a7c P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:04.028586 23213 raft_consensus.cc:374] T 3d51061767744aecbe46917043e05a7c P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:04.029754 23213 raft_consensus.cc:729] T 3d51061767744aecbe46917043e05a7c P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:54:04.030006 23213 consensus_queue.cc:260] T 3d51061767744aecbe46917043e05a7c P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:04.030568 23213 ts_tablet_manager.cc:1398] T 3d51061767744aecbe46917043e05a7c P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:04.119145 22863 consensus_queue.cc:1035] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:04.126443 22876 consensus_queue.cc:1035] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:04.139403 23052 raft_consensus.cc:482] T 3d51061767744aecbe46917043e05a7c P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:04.139535 23052 raft_consensus.cc:504] T 3d51061767744aecbe46917043e05a7c P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:04.140092 23052 leader_election.cc:288] T 3d51061767744aecbe46917043e05a7c P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:54:04.140534 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d51061767744aecbe46917043e05a7c" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:54:04.140630 17474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d51061767744aecbe46917043e05a7c" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:54:04.140774 17737 raft_consensus.cc:2454] T 3d51061767744aecbe46917043e05a7c P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:54:04.140859 17474 raft_consensus.cc:2454] T 3d51061767744aecbe46917043e05a7c P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:54:04.141214 17542 leader_election.cc:302] T 3d51061767744aecbe46917043e05a7c P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:54:04.141458 23052 raft_consensus.cc:2781] T 3d51061767744aecbe46917043e05a7c P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:04.141528 23052 raft_consensus.cc:482] T 3d51061767744aecbe46917043e05a7c P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:04.141589 23052 raft_consensus.cc:3037] T 3d51061767744aecbe46917043e05a7c P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:04.143273 23052 raft_consensus.cc:504] T 3d51061767744aecbe46917043e05a7c P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:04.143694 23052 leader_election.cc:288] T 3d51061767744aecbe46917043e05a7c P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Requested vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:54:04.144145 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d51061767744aecbe46917043e05a7c" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:54:04.144181 17474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d51061767744aecbe46917043e05a7c" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:54:04.144320 17737 raft_consensus.cc:3037] T 3d51061767744aecbe46917043e05a7c P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:04.144378 17474 raft_consensus.cc:3037] T 3d51061767744aecbe46917043e05a7c P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:04.146771 17474 raft_consensus.cc:2454] T 3d51061767744aecbe46917043e05a7c P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:54:04.146924 17737 raft_consensus.cc:2454] T 3d51061767744aecbe46917043e05a7c P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:54:04.148521 17542 leader_election.cc:302] T 3d51061767744aecbe46917043e05a7c P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:54:04.149047 23220 raft_consensus.cc:2781] T 3d51061767744aecbe46917043e05a7c P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:04.149156 23220 raft_consensus.cc:686] T 3d51061767744aecbe46917043e05a7c P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Becoming Leader. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Running, Role: LEADER
I20240502 01:54:04.149413 23220 consensus_queue.cc:237] T 3d51061767744aecbe46917043e05a7c P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:04.151293 23180 consensus_queue.cc:1035] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:04.161958 17327 catalog_manager.cc:5484] T 3d51061767744aecbe46917043e05a7c P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66). New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:04.166914 22991 consensus_queue.cc:1035] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:04.177362 23180 consensus_queue.cc:1035] T b671a9ceacea4a819f750fd37130be0f P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:04.185420 22940 consensus_queue.cc:1035] T b671a9ceacea4a819f750fd37130be0f P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:04.286286 17327 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:54:04.293471 17737 raft_consensus.cc:1261] T 3d51061767744aecbe46917043e05a7c P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:04.293706 17474 raft_consensus.cc:1261] T 3d51061767744aecbe46917043e05a7c P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:04.294184 23052 consensus_queue.cc:1035] T 3d51061767744aecbe46917043e05a7c P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:04.294660 23052 consensus_queue.cc:1035] T 3d51061767744aecbe46917043e05a7c P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:04.299906 17327 catalog_manager.cc:5576] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115) does not have the latest schema for tablet 3d51061767744aecbe46917043e05a7c (table rebalancer_dynamic_table [id=537bb6fadbef4eae9de6f5cffebb0358]). Expected version 1 got 0
I20240502 01:54:04.301914 17327 catalog_manager.cc:5576] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321) does not have the latest schema for tablet 3d51061767744aecbe46917043e05a7c (table rebalancer_dynamic_table [id=537bb6fadbef4eae9de6f5cffebb0358]). Expected version 1 got 0
I20240502 01:54:04.309777 23258 tablet.cc:1722] T 3d51061767744aecbe46917043e05a7c P 3fceccefba274b1dbbd8216af8b0bc29: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:04.310447 23258 tablet.cc:1620] T 3d51061767744aecbe46917043e05a7c P 3fceccefba274b1dbbd8216af8b0bc29: MemRowSet was empty: no flush needed.
I20240502 01:54:04.311129 23259 tablet.cc:1722] T 3d51061767744aecbe46917043e05a7c P 4da690c0fa13407a85c0689ce8de9ade: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:04.311749 23259 tablet.cc:1620] T 3d51061767744aecbe46917043e05a7c P 4da690c0fa13407a85c0689ce8de9ade: MemRowSet was empty: no flush needed.
I20240502 01:54:04.312371 17327 catalog_manager.cc:5576] TS 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531) does not have the latest schema for tablet 3d51061767744aecbe46917043e05a7c (table rebalancer_dynamic_table [id=537bb6fadbef4eae9de6f5cffebb0358]). Expected version 1 got 0
I20240502 01:54:04.314594 17330 catalog_manager.cc:5576] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321) does not have the latest schema for tablet 3d51061767744aecbe46917043e05a7c (table rebalancer_dynamic_table [id=537bb6fadbef4eae9de6f5cffebb0358]). Expected version 1 got 0
I20240502 01:54:04.318593 23260 tablet.cc:1722] T 3d51061767744aecbe46917043e05a7c P dfa1152987644bd781b46c7e42a7a685: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:04.319049 23260 tablet.cc:1620] T 3d51061767744aecbe46917043e05a7c P dfa1152987644bd781b46c7e42a7a685: MemRowSet was empty: no flush needed.
I20240502 01:54:04.319332 17330 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_extra_table_32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:04.321728 17333 catalog_manager.cc:5576] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115) does not have the latest schema for tablet 3d51061767744aecbe46917043e05a7c (table rebalancer_dynamic_table [id=537bb6fadbef4eae9de6f5cffebb0358]). Expected version 1 got 0
I20240502 01:54:04.324648 17334 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=537bb6fadbef4eae9de6f5cffebb0358] alter complete (version 1)
I20240502 01:54:04.330773 20212 tablet_service.cc:1467] Processing CreateTablet for tablet da06571e020948059fb06f65f26c831d (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6cf4b74847bf491c93c43661f9c6d7a5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:04.331030 17587 tablet_service.cc:1467] Processing CreateTablet for tablet bffb870715a243d992107583d7c17f1b (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6cf4b74847bf491c93c43661f9c6d7a5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:04.331207 20212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da06571e020948059fb06f65f26c831d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:04.331386 17587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bffb870715a243d992107583d7c17f1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:04.331689 17716 tablet_service.cc:1467] Processing CreateTablet for tablet da06571e020948059fb06f65f26c831d (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6cf4b74847bf491c93c43661f9c6d7a5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:04.332073 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da06571e020948059fb06f65f26c831d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:04.333425 20211 tablet_service.cc:1467] Processing CreateTablet for tablet bffb870715a243d992107583d7c17f1b (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6cf4b74847bf491c93c43661f9c6d7a5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:04.333801 20211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bffb870715a243d992107583d7c17f1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:04.337702 20608 tablet_service.cc:1467] Processing CreateTablet for tablet da06571e020948059fb06f65f26c831d (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6cf4b74847bf491c93c43661f9c6d7a5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:04.337850 20609 tablet_service.cc:1467] Processing CreateTablet for tablet bffb870715a243d992107583d7c17f1b (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6cf4b74847bf491c93c43661f9c6d7a5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:04.338114 20608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da06571e020948059fb06f65f26c831d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:04.338819 20609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bffb870715a243d992107583d7c17f1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:04.344480 23210 tablet_bootstrap.cc:492] T da06571e020948059fb06f65f26c831d P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:54:04.345785 23210 tablet_bootstrap.cc:654] T da06571e020948059fb06f65f26c831d P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:04.348655 17585 tablet_service.cc:1467] Processing CreateTablet for tablet 074a08b2b64640eb9784b6a23921ce8d (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6cf4b74847bf491c93c43661f9c6d7a5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:04.349069 17585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 074a08b2b64640eb9784b6a23921ce8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:04.355463 20347 tablet_service.cc:1467] Processing CreateTablet for tablet 074a08b2b64640eb9784b6a23921ce8d (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6cf4b74847bf491c93c43661f9c6d7a5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:04.355669 20610 tablet_service.cc:1467] Processing CreateTablet for tablet 074a08b2b64640eb9784b6a23921ce8d (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6cf4b74847bf491c93c43661f9c6d7a5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:04.355880 20347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 074a08b2b64640eb9784b6a23921ce8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:04.356076 20610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 074a08b2b64640eb9784b6a23921ce8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:04.370461 23150 tablet_bootstrap.cc:492] T bffb870715a243d992107583d7c17f1b P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:54:04.371945 23150 tablet_bootstrap.cc:654] T bffb870715a243d992107583d7c17f1b P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:04.372151 23210 tablet_bootstrap.cc:492] T da06571e020948059fb06f65f26c831d P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:54:04.372277 23210 ts_tablet_manager.cc:1367] T da06571e020948059fb06f65f26c831d P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.000s
I20240502 01:54:04.372905 23210 raft_consensus.cc:348] T da06571e020948059fb06f65f26c831d P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:04.373147 23210 raft_consensus.cc:374] T da06571e020948059fb06f65f26c831d P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:04.373224 23210 raft_consensus.cc:729] T da06571e020948059fb06f65f26c831d P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:54:04.373452 23210 consensus_queue.cc:260] T da06571e020948059fb06f65f26c831d P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:04.373809 23150 tablet_bootstrap.cc:492] T bffb870715a243d992107583d7c17f1b P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:54:04.373930 23150 ts_tablet_manager.cc:1367] T bffb870715a243d992107583d7c17f1b P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 01:54:04.373975 23261 tablet_bootstrap.cc:492] T da06571e020948059fb06f65f26c831d P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:54:04.374538 23150 raft_consensus.cc:348] T bffb870715a243d992107583d7c17f1b P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:04.374717 23150 raft_consensus.cc:374] T bffb870715a243d992107583d7c17f1b P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:04.374778 23150 raft_consensus.cc:729] T bffb870715a243d992107583d7c17f1b P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:54:04.374969 23150 consensus_queue.cc:260] T bffb870715a243d992107583d7c17f1b P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:04.375468 23150 ts_tablet_manager.cc:1398] T bffb870715a243d992107583d7c17f1b P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:54:04.376259 23261 tablet_bootstrap.cc:654] T da06571e020948059fb06f65f26c831d P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:04.377306 23150 tablet_bootstrap.cc:492] T 074a08b2b64640eb9784b6a23921ce8d P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:54:04.377568 23210 ts_tablet_manager.cc:1398] T da06571e020948059fb06f65f26c831d P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:54:04.378808 23150 tablet_bootstrap.cc:654] T 074a08b2b64640eb9784b6a23921ce8d P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:04.381703 23265 tablet_bootstrap.cc:492] T bffb870715a243d992107583d7c17f1b P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:54:04.383232 23052 raft_consensus.cc:482] T bffb870715a243d992107583d7c17f1b P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:04.383358 23052 raft_consensus.cc:504] T bffb870715a243d992107583d7c17f1b P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:04.384091 23052 leader_election.cc:288] T bffb870715a243d992107583d7c17f1b P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:54:04.384635 20629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bffb870715a243d992107583d7c17f1b" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091" is_pre_election: true
W20240502 01:54:04.385424 17542 leader_election.cc:341] T bffb870715a243d992107583d7c17f1b P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:04.386274 20233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bffb870715a243d992107583d7c17f1b" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:54:04.388833 23265 tablet_bootstrap.cc:654] T bffb870715a243d992107583d7c17f1b P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:04.389815 23266 tablet_bootstrap.cc:492] T 074a08b2b64640eb9784b6a23921ce8d P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
W20240502 01:54:04.389892 17543 leader_election.cc:341] T bffb870715a243d992107583d7c17f1b P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:04.390018 17543 leader_election.cc:302] T bffb870715a243d992107583d7c17f1b P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29; no voters: f19cc6cb1e76460aa6711e49a5775091, fdc2ac372492454c85387589a91f63af
I20240502 01:54:04.390712 23052 raft_consensus.cc:2726] T bffb870715a243d992107583d7c17f1b P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:54:04.391335 23265 tablet_bootstrap.cc:492] T bffb870715a243d992107583d7c17f1b P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:54:04.391585 23265 ts_tablet_manager.cc:1367] T bffb870715a243d992107583d7c17f1b P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.002s
I20240502 01:54:04.392341 23266 tablet_bootstrap.cc:654] T 074a08b2b64640eb9784b6a23921ce8d P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:04.393258 23150 tablet_bootstrap.cc:492] T 074a08b2b64640eb9784b6a23921ce8d P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:54:04.393429 23150 ts_tablet_manager.cc:1367] T 074a08b2b64640eb9784b6a23921ce8d P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.005s
I20240502 01:54:04.393445 23265 raft_consensus.cc:348] T bffb870715a243d992107583d7c17f1b P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:04.393720 23265 raft_consensus.cc:374] T bffb870715a243d992107583d7c17f1b P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:04.393873 23265 raft_consensus.cc:729] T bffb870715a243d992107583d7c17f1b P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:54:04.393895 23261 tablet_bootstrap.cc:492] T da06571e020948059fb06f65f26c831d P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:54:04.394030 23261 ts_tablet_manager.cc:1367] T da06571e020948059fb06f65f26c831d P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.009s
I20240502 01:54:04.394167 23265 consensus_queue.cc:260] T bffb870715a243d992107583d7c17f1b P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:04.394148 23150 raft_consensus.cc:348] T 074a08b2b64640eb9784b6a23921ce8d P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:04.394356 23150 raft_consensus.cc:374] T 074a08b2b64640eb9784b6a23921ce8d P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:04.394423 23150 raft_consensus.cc:729] T 074a08b2b64640eb9784b6a23921ce8d P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:54:04.394625 23150 consensus_queue.cc:260] T 074a08b2b64640eb9784b6a23921ce8d P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:04.394825 23261 raft_consensus.cc:348] T da06571e020948059fb06f65f26c831d P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:04.395022 23261 raft_consensus.cc:374] T da06571e020948059fb06f65f26c831d P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:04.395107 23261 raft_consensus.cc:729] T da06571e020948059fb06f65f26c831d P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:54:04.395332 23261 consensus_queue.cc:260] T da06571e020948059fb06f65f26c831d P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:04.396082 23150 ts_tablet_manager.cc:1398] T 074a08b2b64640eb9784b6a23921ce8d P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:54:04.396957 23261 ts_tablet_manager.cc:1398] T da06571e020948059fb06f65f26c831d P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:54:04.397202 23261 tablet_bootstrap.cc:492] T bffb870715a243d992107583d7c17f1b P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:54:04.398005 23266 tablet_bootstrap.cc:492] T 074a08b2b64640eb9784b6a23921ce8d P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:54:04.398128 23266 ts_tablet_manager.cc:1367] T 074a08b2b64640eb9784b6a23921ce8d P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240502 01:54:04.398639 23261 tablet_bootstrap.cc:654] T bffb870715a243d992107583d7c17f1b P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:04.398990 23266 raft_consensus.cc:348] T 074a08b2b64640eb9784b6a23921ce8d P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:04.399183 23266 raft_consensus.cc:374] T 074a08b2b64640eb9784b6a23921ce8d P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:04.399327 23266 raft_consensus.cc:729] T 074a08b2b64640eb9784b6a23921ce8d P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:54:04.399595 23266 consensus_queue.cc:260] T 074a08b2b64640eb9784b6a23921ce8d P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:04.400159 23266 ts_tablet_manager.cc:1398] T 074a08b2b64640eb9784b6a23921ce8d P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:54:04.403661 23265 ts_tablet_manager.cc:1398] T bffb870715a243d992107583d7c17f1b P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20240502 01:54:04.403985 23265 tablet_bootstrap.cc:492] T da06571e020948059fb06f65f26c831d P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:54:04.408828 23265 tablet_bootstrap.cc:654] T da06571e020948059fb06f65f26c831d P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:04.408852 23261 tablet_bootstrap.cc:492] T bffb870715a243d992107583d7c17f1b P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:54:04.408988 23261 ts_tablet_manager.cc:1367] T bffb870715a243d992107583d7c17f1b P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.002s
I20240502 01:54:04.409808 23261 raft_consensus.cc:348] T bffb870715a243d992107583d7c17f1b P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:04.410006 23261 raft_consensus.cc:374] T bffb870715a243d992107583d7c17f1b P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:04.410379 23261 raft_consensus.cc:729] T bffb870715a243d992107583d7c17f1b P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:54:04.410673 23261 consensus_queue.cc:260] T bffb870715a243d992107583d7c17f1b P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:04.411187 23261 ts_tablet_manager.cc:1398] T bffb870715a243d992107583d7c17f1b P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:54:04.411298 23265 tablet_bootstrap.cc:492] T da06571e020948059fb06f65f26c831d P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:54:04.411530 23265 ts_tablet_manager.cc:1367] T da06571e020948059fb06f65f26c831d P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240502 01:54:04.412222 23265 raft_consensus.cc:348] T da06571e020948059fb06f65f26c831d P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:04.412454 23265 raft_consensus.cc:374] T da06571e020948059fb06f65f26c831d P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:04.412541 23265 raft_consensus.cc:729] T da06571e020948059fb06f65f26c831d P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:54:04.412830 23265 consensus_queue.cc:260] T da06571e020948059fb06f65f26c831d P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:04.414155 23265 ts_tablet_manager.cc:1398] T da06571e020948059fb06f65f26c831d P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 01:54:04.414465 23265 tablet_bootstrap.cc:492] T 074a08b2b64640eb9784b6a23921ce8d P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:54:04.417248 23265 tablet_bootstrap.cc:654] T 074a08b2b64640eb9784b6a23921ce8d P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:04.419327 23265 tablet_bootstrap.cc:492] T 074a08b2b64640eb9784b6a23921ce8d P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:54:04.419507 23265 ts_tablet_manager.cc:1367] T 074a08b2b64640eb9784b6a23921ce8d P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20240502 01:54:04.420423 23265 raft_consensus.cc:348] T 074a08b2b64640eb9784b6a23921ce8d P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:04.420629 23265 raft_consensus.cc:374] T 074a08b2b64640eb9784b6a23921ce8d P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:04.420778 23265 raft_consensus.cc:729] T 074a08b2b64640eb9784b6a23921ce8d P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:54:04.421077 23265 consensus_queue.cc:260] T 074a08b2b64640eb9784b6a23921ce8d P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:04.421782 23265 ts_tablet_manager.cc:1398] T 074a08b2b64640eb9784b6a23921ce8d P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:54:04.431491 22991 raft_consensus.cc:482] T da06571e020948059fb06f65f26c831d P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:04.431689 22991 raft_consensus.cc:504] T da06571e020948059fb06f65f26c831d P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:04.432461 22991 leader_election.cc:288] T da06571e020948059fb06f65f26c831d P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:54:04.432770 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da06571e020948059fb06f65f26c831d" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:54:04.432991 17737 raft_consensus.cc:2454] T da06571e020948059fb06f65f26c831d P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 0.
I20240502 01:54:04.433161 20233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da06571e020948059fb06f65f26c831d" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:54:04.433486 20233 raft_consensus.cc:2454] T da06571e020948059fb06f65f26c831d P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 0.
I20240502 01:54:04.434454 20564 leader_election.cc:302] T da06571e020948059fb06f65f26c831d P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dfa1152987644bd781b46c7e42a7a685, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:54:04.435076 22991 raft_consensus.cc:2781] T da06571e020948059fb06f65f26c831d P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:04.435217 22991 raft_consensus.cc:482] T da06571e020948059fb06f65f26c831d P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:04.435408 22991 raft_consensus.cc:3037] T da06571e020948059fb06f65f26c831d P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:04.437886 22991 raft_consensus.cc:504] T da06571e020948059fb06f65f26c831d P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:04.438546 22991 leader_election.cc:288] T da06571e020948059fb06f65f26c831d P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 election: Requested vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:54:04.438772 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da06571e020948059fb06f65f26c831d" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:54:04.438863 20233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da06571e020948059fb06f65f26c831d" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:54:04.438923 17737 raft_consensus.cc:3037] T da06571e020948059fb06f65f26c831d P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:04.439044 20233 raft_consensus.cc:3037] T da06571e020948059fb06f65f26c831d P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:04.440802 20233 raft_consensus.cc:2454] T da06571e020948059fb06f65f26c831d P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 1.
I20240502 01:54:04.441174 20565 leader_election.cc:302] T da06571e020948059fb06f65f26c831d P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f19cc6cb1e76460aa6711e49a5775091, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:54:04.441395 22991 raft_consensus.cc:2781] T da06571e020948059fb06f65f26c831d P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:04.441660 22991 raft_consensus.cc:686] T da06571e020948059fb06f65f26c831d P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Becoming Leader. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Running, Role: LEADER
I20240502 01:54:04.441650 17737 raft_consensus.cc:2454] T da06571e020948059fb06f65f26c831d P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 1.
I20240502 01:54:04.442113 22991 consensus_queue.cc:237] T da06571e020948059fb06f65f26c831d P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:04.442250 17330 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:54:04.445596 17327 catalog_manager.cc:5484] T da06571e020948059fb06f65f26c831d P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: term changed from 0 to 1, leader changed from <none> to f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72). New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:04.447453 23139 raft_consensus.cc:482] T bffb870715a243d992107583d7c17f1b P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:04.447564 23139 raft_consensus.cc:504] T bffb870715a243d992107583d7c17f1b P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:04.448230 23139 leader_election.cc:288] T bffb870715a243d992107583d7c17f1b P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:54:04.448591 17605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bffb870715a243d992107583d7c17f1b" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:54:04.448779 17605 raft_consensus.cc:2454] T bffb870715a243d992107583d7c17f1b P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 0.
I20240502 01:54:04.448792 20629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bffb870715a243d992107583d7c17f1b" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091" is_pre_election: true
I20240502 01:54:04.448992 20629 raft_consensus.cc:2454] T bffb870715a243d992107583d7c17f1b P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 0.
I20240502 01:54:04.449218 20165 leader_election.cc:302] T bffb870715a243d992107583d7c17f1b P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:54:04.449505 23139 raft_consensus.cc:2781] T bffb870715a243d992107583d7c17f1b P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:04.449666 23139 raft_consensus.cc:482] T bffb870715a243d992107583d7c17f1b P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:04.449771 23139 raft_consensus.cc:3037] T bffb870715a243d992107583d7c17f1b P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:04.452198 23139 raft_consensus.cc:504] T bffb870715a243d992107583d7c17f1b P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:04.452193 23258 tablet.cc:1722] T 3d51061767744aecbe46917043e05a7c P 3fceccefba274b1dbbd8216af8b0bc29: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:04.452553 23258 tablet.cc:1620] T 3d51061767744aecbe46917043e05a7c P 3fceccefba274b1dbbd8216af8b0bc29: MemRowSet was empty: no flush needed.
I20240502 01:54:04.452649 23139 leader_election.cc:288] T bffb870715a243d992107583d7c17f1b P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 election: Requested vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:54:04.453002 17605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bffb870715a243d992107583d7c17f1b" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:54:04.453095 20629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bffb870715a243d992107583d7c17f1b" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091"
I20240502 01:54:04.453195 17605 raft_consensus.cc:3037] T bffb870715a243d992107583d7c17f1b P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:04.453258 20629 raft_consensus.cc:3037] T bffb870715a243d992107583d7c17f1b P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:04.455251 20629 raft_consensus.cc:2454] T bffb870715a243d992107583d7c17f1b P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 1.
I20240502 01:54:04.455626 20167 leader_election.cc:302] T bffb870715a243d992107583d7c17f1b P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f19cc6cb1e76460aa6711e49a5775091, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:54:04.455639 17605 raft_consensus.cc:2454] T bffb870715a243d992107583d7c17f1b P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 1.
I20240502 01:54:04.456036 23139 raft_consensus.cc:2781] T bffb870715a243d992107583d7c17f1b P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:04.456228 23139 raft_consensus.cc:686] T bffb870715a243d992107583d7c17f1b P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Becoming Leader. State: Replica: fdc2ac372492454c85387589a91f63af, State: Running, Role: LEADER
I20240502 01:54:04.456471 22940 raft_consensus.cc:482] T 074a08b2b64640eb9784b6a23921ce8d P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:04.456542 23139 consensus_queue.cc:237] T bffb870715a243d992107583d7c17f1b P fdc2ac372492454c85387589a91f63af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:04.456611 22940 raft_consensus.cc:504] T 074a08b2b64640eb9784b6a23921ce8d P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:04.457223 22940 leader_election.cc:288] T 074a08b2b64640eb9784b6a23921ce8d P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:54:04.458782 20365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "074a08b2b64640eb9784b6a23921ce8d" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:54:04.458806 23259 tablet.cc:1722] T 3d51061767744aecbe46917043e05a7c P 4da690c0fa13407a85c0689ce8de9ade: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:04.459021 20365 raft_consensus.cc:2454] T 074a08b2b64640eb9784b6a23921ce8d P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 0.
I20240502 01:54:04.459271 23259 tablet.cc:1620] T 3d51061767744aecbe46917043e05a7c P 4da690c0fa13407a85c0689ce8de9ade: MemRowSet was empty: no flush needed.
I20240502 01:54:04.459564 20564 leader_election.cc:302] T 074a08b2b64640eb9784b6a23921ce8d P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:54:04.459894 22940 raft_consensus.cc:2781] T 074a08b2b64640eb9784b6a23921ce8d P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:04.459991 22940 raft_consensus.cc:482] T 074a08b2b64640eb9784b6a23921ce8d P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:04.460080 22940 raft_consensus.cc:3037] T 074a08b2b64640eb9784b6a23921ce8d P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:04.460908 23260 tablet.cc:1722] T 3d51061767744aecbe46917043e05a7c P dfa1152987644bd781b46c7e42a7a685: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:04.461369 23260 tablet.cc:1620] T 3d51061767744aecbe46917043e05a7c P dfa1152987644bd781b46c7e42a7a685: MemRowSet was empty: no flush needed.
I20240502 01:54:04.462411 22940 raft_consensus.cc:504] T 074a08b2b64640eb9784b6a23921ce8d P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:04.463346 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "074a08b2b64640eb9784b6a23921ce8d" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:54:04.463526 17604 raft_consensus.cc:3037] T 074a08b2b64640eb9784b6a23921ce8d P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:04.466050 17604 raft_consensus.cc:2454] T 074a08b2b64640eb9784b6a23921ce8d P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 1.
I20240502 01:54:04.468628 17326 catalog_manager.cc:5484] T bffb870715a243d992107583d7c17f1b P fdc2ac372492454c85387589a91f63af reported cstate change: term changed from 0 to 1, leader changed from <none> to fdc2ac372492454c85387589a91f63af (127.11.42.69). New cstate: current_term: 1 leader_uuid: "fdc2ac372492454c85387589a91f63af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:04.468852 17327 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=537bb6fadbef4eae9de6f5cffebb0358] alter complete (version 2)
I20240502 01:54:04.470753 20562 leader_election.cc:302] T 074a08b2b64640eb9784b6a23921ce8d P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:54:04.472460 22991 raft_consensus.cc:2781] T 074a08b2b64640eb9784b6a23921ce8d P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:04.473198 22991 raft_consensus.cc:686] T 074a08b2b64640eb9784b6a23921ce8d P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Becoming Leader. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Running, Role: LEADER
I20240502 01:54:04.473477 22991 consensus_queue.cc:237] T 074a08b2b64640eb9784b6a23921ce8d P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:04.476476 17605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "074a08b2b64640eb9784b6a23921ce8d" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:54:04.476683 17605 raft_consensus.cc:2362] T 074a08b2b64640eb9784b6a23921ce8d P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 1. Re-sending same reply.
I20240502 01:54:04.477245 22940 leader_election.cc:288] T 074a08b2b64640eb9784b6a23921ce8d P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 election: Requested vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:54:04.477427 17330 catalog_manager.cc:5484] T 074a08b2b64640eb9784b6a23921ce8d P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: term changed from 0 to 1, leader changed from <none> to f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72). New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:04.477887 20365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "074a08b2b64640eb9784b6a23921ce8d" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
I20240502 01:54:04.478081 20365 raft_consensus.cc:3037] T 074a08b2b64640eb9784b6a23921ce8d P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:04.480580 20365 raft_consensus.cc:2454] T 074a08b2b64640eb9784b6a23921ce8d P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 1.
I20240502 01:54:04.531993 20365 consensus_queue.cc:237] T aebde87a7f114537abd9c1d8b7e97004 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:04.533654 17866 raft_consensus.cc:1261] T aebde87a7f114537abd9c1d8b7e97004 P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:54:04.534088 17737 raft_consensus.cc:1261] T aebde87a7f114537abd9c1d8b7e97004 P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:54:04.534642 22876 consensus_queue.cc:1035] T aebde87a7f114537abd9c1d8b7e97004 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:04.534937 22863 consensus_queue.cc:1035] T aebde87a7f114537abd9c1d8b7e97004 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:04.537091 23283 raft_consensus.cc:2932] T aebde87a7f114537abd9c1d8b7e97004 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:04.537648 17866 raft_consensus.cc:2932] T aebde87a7f114537abd9c1d8b7e97004 P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:04.537751 17737 raft_consensus.cc:2932] T aebde87a7f114537abd9c1d8b7e97004 P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:04.541648 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet aebde87a7f114537abd9c1d8b7e97004 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:04.542572 17330 catalog_manager.cc:5484] T aebde87a7f114537abd9c1d8b7e97004 P dfa1152987644bd781b46c7e42a7a685 reported cstate change: config changed from index 3 to 5, VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) evicted. New cstate: current_term: 2 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:04.546018 20476 tablet_service.cc:1514] Processing DeleteTablet for tablet aebde87a7f114537abd9c1d8b7e97004 with delete_type TABLET_DATA_TOMBSTONED (TS 2f306549407b4bb281ccf2994035e4f1 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35018
I20240502 01:54:04.548352 23284 tablet_replica.cc:330] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1: stopping tablet replica
I20240502 01:54:04.548836 23284 raft_consensus.cc:2229] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:04.549069 23284 raft_consensus.cc:2258] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:04.549882 23284 ts_tablet_manager.cc:1875] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:04.554327 23284 ts_tablet_manager.cc:1888] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 01:54:04.554486 23284 log.cc:1196] T aebde87a7f114537abd9c1d8b7e97004 P 2f306549407b4bb281ccf2994035e4f1: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal/wals/aebde87a7f114537abd9c1d8b7e97004
I20240502 01:54:04.555156 17322 catalog_manager.cc:4830] TS 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): tablet aebde87a7f114537abd9c1d8b7e97004 (table rebalancer_extra_table_8 [id=c5f998b2cdaf41f6a49ea6ae65a975b8]) successfully deleted
I20240502 01:54:04.572706 17327 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:54:04.580090 23258 tablet.cc:1722] T 3d51061767744aecbe46917043e05a7c P 3fceccefba274b1dbbd8216af8b0bc29: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:04.580829 23259 tablet.cc:1722] T 3d51061767744aecbe46917043e05a7c P 4da690c0fa13407a85c0689ce8de9ade: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:04.585855 23260 tablet.cc:1722] T 3d51061767744aecbe46917043e05a7c P dfa1152987644bd781b46c7e42a7a685: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:04.590370 17330 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table_new_name [id=537bb6fadbef4eae9de6f5cffebb0358] alter complete (version 3)
I20240502 01:54:04.694552 17326 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:04.694743 17326 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:04.697942 17326 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet 3d51061767744aecbe46917043e05a7c
I20240502 01:54:04.698814 17716 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d51061767744aecbe46917043e05a7c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:04 UTC) from {username='slave'} at 127.0.0.1:52198
I20240502 01:54:04.699004 17456 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d51061767744aecbe46917043e05a7c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:04 UTC) from {username='slave'} at 127.0.0.1:54520
I20240502 01:54:04.699334 23285 tablet_replica.cc:330] T 3d51061767744aecbe46917043e05a7c P dfa1152987644bd781b46c7e42a7a685: stopping tablet replica
I20240502 01:54:04.699323 17587 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d51061767744aecbe46917043e05a7c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:04 UTC) from {username='slave'} at 127.0.0.1:55232
I20240502 01:54:04.699577 23285 raft_consensus.cc:2229] T 3d51061767744aecbe46917043e05a7c P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:04.699931 23285 raft_consensus.cc:2258] T 3d51061767744aecbe46917043e05a7c P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:04.700944 23285 ts_tablet_manager.cc:1875] T 3d51061767744aecbe46917043e05a7c P dfa1152987644bd781b46c7e42a7a685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:04.709828 23285 ts_tablet_manager.cc:1888] T 3d51061767744aecbe46917043e05a7c P dfa1152987644bd781b46c7e42a7a685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:04.709971 23285 log.cc:1196] T 3d51061767744aecbe46917043e05a7c P dfa1152987644bd781b46c7e42a7a685: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/wals/3d51061767744aecbe46917043e05a7c
I20240502 01:54:04.710395 23285 ts_tablet_manager.cc:1909] T 3d51061767744aecbe46917043e05a7c P dfa1152987644bd781b46c7e42a7a685: Deleting consensus metadata
I20240502 01:54:04.710903 23287 tablet_replica.cc:330] T 3d51061767744aecbe46917043e05a7c P 3fceccefba274b1dbbd8216af8b0bc29: stopping tablet replica
I20240502 01:54:04.711449 23287 raft_consensus.cc:2229] T 3d51061767744aecbe46917043e05a7c P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:04.712087 23287 raft_consensus.cc:2258] T 3d51061767744aecbe46917043e05a7c P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:04.712147 17321 catalog_manager.cc:4830] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): tablet 3d51061767744aecbe46917043e05a7c (table rebalancer_dynamic_table_new_name [id=537bb6fadbef4eae9de6f5cffebb0358]) successfully deleted
I20240502 01:54:04.713255 23287 ts_tablet_manager.cc:1875] T 3d51061767744aecbe46917043e05a7c P 3fceccefba274b1dbbd8216af8b0bc29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:04.719025 23287 ts_tablet_manager.cc:1888] T 3d51061767744aecbe46917043e05a7c P 3fceccefba274b1dbbd8216af8b0bc29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:04.719127 23287 log.cc:1196] T 3d51061767744aecbe46917043e05a7c P 3fceccefba274b1dbbd8216af8b0bc29: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/wals/3d51061767744aecbe46917043e05a7c
I20240502 01:54:04.719465 23287 ts_tablet_manager.cc:1909] T 3d51061767744aecbe46917043e05a7c P 3fceccefba274b1dbbd8216af8b0bc29: Deleting consensus metadata
I20240502 01:54:04.720340 17319 catalog_manager.cc:4830] TS 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): tablet 3d51061767744aecbe46917043e05a7c (table rebalancer_dynamic_table_new_name [id=537bb6fadbef4eae9de6f5cffebb0358]) successfully deleted
I20240502 01:54:04.720889 23286 tablet_replica.cc:330] T 3d51061767744aecbe46917043e05a7c P 4da690c0fa13407a85c0689ce8de9ade: stopping tablet replica
I20240502 01:54:04.721092 23286 raft_consensus.cc:2229] T 3d51061767744aecbe46917043e05a7c P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:04.721275 23286 raft_consensus.cc:2258] T 3d51061767744aecbe46917043e05a7c P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:04.722026 23286 ts_tablet_manager.cc:1875] T 3d51061767744aecbe46917043e05a7c P 4da690c0fa13407a85c0689ce8de9ade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:04.728468 23286 ts_tablet_manager.cc:1888] T 3d51061767744aecbe46917043e05a7c P 4da690c0fa13407a85c0689ce8de9ade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:04.728585 23286 log.cc:1196] T 3d51061767744aecbe46917043e05a7c P 4da690c0fa13407a85c0689ce8de9ade: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/3d51061767744aecbe46917043e05a7c
I20240502 01:54:04.729063 23286 ts_tablet_manager.cc:1909] T 3d51061767744aecbe46917043e05a7c P 4da690c0fa13407a85c0689ce8de9ade: Deleting consensus metadata
I20240502 01:54:04.729983 17321 catalog_manager.cc:4830] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): tablet 3d51061767744aecbe46917043e05a7c (table rebalancer_dynamic_table_new_name [id=537bb6fadbef4eae9de6f5cffebb0358]) successfully deleted
I20240502 01:54:04.800936 17326 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:54:04.808509 17716 tablet_service.cc:1467] Processing CreateTablet for tablet 4c22bfa55cbf41628539aaf8a47c6c1c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6f9eef247379455187a9871e7d04643f]), partition=
I20240502 01:54:04.808598 17845 tablet_service.cc:1467] Processing CreateTablet for tablet 4c22bfa55cbf41628539aaf8a47c6c1c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6f9eef247379455187a9871e7d04643f]), partition=
I20240502 01:54:04.808926 17845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c22bfa55cbf41628539aaf8a47c6c1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:04.808938 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c22bfa55cbf41628539aaf8a47c6c1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:04.813735 20609 tablet_service.cc:1467] Processing CreateTablet for tablet 4c22bfa55cbf41628539aaf8a47c6c1c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6f9eef247379455187a9871e7d04643f]), partition=
I20240502 01:54:04.814155 20609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c22bfa55cbf41628539aaf8a47c6c1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:04.816820 23300 tablet_bootstrap.cc:492] T 4c22bfa55cbf41628539aaf8a47c6c1c P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:54:04.818127 23300 tablet_bootstrap.cc:654] T 4c22bfa55cbf41628539aaf8a47c6c1c P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:04.819761 23265 tablet_bootstrap.cc:492] T 4c22bfa55cbf41628539aaf8a47c6c1c P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:54:04.821344 23265 tablet_bootstrap.cc:654] T 4c22bfa55cbf41628539aaf8a47c6c1c P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:04.822388 23210 tablet_bootstrap.cc:492] T 4c22bfa55cbf41628539aaf8a47c6c1c P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:54:04.823657 23210 tablet_bootstrap.cc:654] T 4c22bfa55cbf41628539aaf8a47c6c1c P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:04.825918 23265 tablet_bootstrap.cc:492] T 4c22bfa55cbf41628539aaf8a47c6c1c P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:54:04.826184 23265 ts_tablet_manager.cc:1367] T 4c22bfa55cbf41628539aaf8a47c6c1c P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:54:04.827353 23265 raft_consensus.cc:348] T 4c22bfa55cbf41628539aaf8a47c6c1c P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:04.827589 23265 raft_consensus.cc:374] T 4c22bfa55cbf41628539aaf8a47c6c1c P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:04.827704 23265 raft_consensus.cc:729] T 4c22bfa55cbf41628539aaf8a47c6c1c P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:54:04.827996 23265 consensus_queue.cc:260] T 4c22bfa55cbf41628539aaf8a47c6c1c P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:04.828805 23265 ts_tablet_manager.cc:1398] T 4c22bfa55cbf41628539aaf8a47c6c1c P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240502 01:54:04.830698 23300 tablet_bootstrap.cc:492] T 4c22bfa55cbf41628539aaf8a47c6c1c P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:54:04.830875 23300 ts_tablet_manager.cc:1367] T 4c22bfa55cbf41628539aaf8a47c6c1c P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20240502 01:54:04.831739 23300 raft_consensus.cc:348] T 4c22bfa55cbf41628539aaf8a47c6c1c P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:04.831979 23300 raft_consensus.cc:374] T 4c22bfa55cbf41628539aaf8a47c6c1c P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:04.832088 23300 raft_consensus.cc:729] T 4c22bfa55cbf41628539aaf8a47c6c1c P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:54:04.832340 23300 consensus_queue.cc:260] T 4c22bfa55cbf41628539aaf8a47c6c1c P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:04.834198 23300 ts_tablet_manager.cc:1398] T 4c22bfa55cbf41628539aaf8a47c6c1c P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:54:04.838559 23210 tablet_bootstrap.cc:492] T 4c22bfa55cbf41628539aaf8a47c6c1c P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:54:04.838706 23210 ts_tablet_manager.cc:1367] T 4c22bfa55cbf41628539aaf8a47c6c1c P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.004s
I20240502 01:54:04.839344 23210 raft_consensus.cc:348] T 4c22bfa55cbf41628539aaf8a47c6c1c P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:04.839541 23210 raft_consensus.cc:374] T 4c22bfa55cbf41628539aaf8a47c6c1c P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:04.839596 23210 raft_consensus.cc:729] T 4c22bfa55cbf41628539aaf8a47c6c1c P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:54:04.839792 23210 consensus_queue.cc:260] T 4c22bfa55cbf41628539aaf8a47c6c1c P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:04.840423 23210 ts_tablet_manager.cc:1398] T 4c22bfa55cbf41628539aaf8a47c6c1c P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:54:04.927737 22991 consensus_queue.cc:1035] T da06571e020948059fb06f65f26c831d P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:04.935365 22940 consensus_queue.cc:1035] T da06571e020948059fb06f65f26c831d P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:04.976637 23139 consensus_queue.cc:1035] T bffb870715a243d992107583d7c17f1b P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:04.986806 23139 consensus_queue.cc:1035] T bffb870715a243d992107583d7c17f1b P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:05.007201 17327 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_extra_table_34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:05.019722 17587 tablet_service.cc:1467] Processing CreateTablet for tablet 57ab576948e346cbbe6715bf53f7c665 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=b04d89d1ce13442db7041b4081ac54f4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:05.020169 17587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57ab576948e346cbbe6715bf53f7c665. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:05.025374 17716 tablet_service.cc:1467] Processing CreateTablet for tablet 57ab576948e346cbbe6715bf53f7c665 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=b04d89d1ce13442db7041b4081ac54f4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:05.025784 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57ab576948e346cbbe6715bf53f7c665. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:05.027325 17845 tablet_service.cc:1467] Processing CreateTablet for tablet 57ab576948e346cbbe6715bf53f7c665 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=b04d89d1ce13442db7041b4081ac54f4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:05.027696 17845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57ab576948e346cbbe6715bf53f7c665. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:05.030689 17456 tablet_service.cc:1467] Processing CreateTablet for tablet a390a222562f425eabb11d7f52cbe9a6 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=b04d89d1ce13442db7041b4081ac54f4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:05.031064 17456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a390a222562f425eabb11d7f52cbe9a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:05.031033 17715 tablet_service.cc:1467] Processing CreateTablet for tablet a390a222562f425eabb11d7f52cbe9a6 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=b04d89d1ce13442db7041b4081ac54f4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:05.031388 17715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a390a222562f425eabb11d7f52cbe9a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:05.032681 20347 tablet_service.cc:1467] Processing CreateTablet for tablet a390a222562f425eabb11d7f52cbe9a6 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=b04d89d1ce13442db7041b4081ac54f4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:05.033088 20347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a390a222562f425eabb11d7f52cbe9a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:05.034636 23210 tablet_bootstrap.cc:492] T 57ab576948e346cbbe6715bf53f7c665 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:54:05.035964 23210 tablet_bootstrap.cc:654] T 57ab576948e346cbbe6715bf53f7c665 P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:05.038018 17454 tablet_service.cc:1467] Processing CreateTablet for tablet 7b2eae408b9843acaa61e698a3e44cff (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=b04d89d1ce13442db7041b4081ac54f4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:05.038195 17585 tablet_service.cc:1467] Processing CreateTablet for tablet 7b2eae408b9843acaa61e698a3e44cff (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=b04d89d1ce13442db7041b4081ac54f4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:05.038369 17454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b2eae408b9843acaa61e698a3e44cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:05.038556 17585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b2eae408b9843acaa61e698a3e44cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:05.041994 23300 tablet_bootstrap.cc:492] T 57ab576948e346cbbe6715bf53f7c665 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:54:05.043833 23300 tablet_bootstrap.cc:654] T 57ab576948e346cbbe6715bf53f7c665 P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:05.046954 23300 tablet_bootstrap.cc:492] T 57ab576948e346cbbe6715bf53f7c665 P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:54:05.047080 23300 ts_tablet_manager.cc:1367] T 57ab576948e346cbbe6715bf53f7c665 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240502 01:54:05.047848 23300 raft_consensus.cc:348] T 57ab576948e346cbbe6715bf53f7c665 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:05.048135 23300 raft_consensus.cc:374] T 57ab576948e346cbbe6715bf53f7c665 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:05.048251 23300 raft_consensus.cc:729] T 57ab576948e346cbbe6715bf53f7c665 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:54:05.048532 23300 consensus_queue.cc:260] T 57ab576948e346cbbe6715bf53f7c665 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:05.051719 23300 ts_tablet_manager.cc:1398] T 57ab576948e346cbbe6715bf53f7c665 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 01:54:05.052860 22940 consensus_queue.cc:1035] T 074a08b2b64640eb9784b6a23921ce8d P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:05.052963 23348 tablet_bootstrap.cc:492] T a390a222562f425eabb11d7f52cbe9a6 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:54:05.055147 23210 tablet_bootstrap.cc:492] T 57ab576948e346cbbe6715bf53f7c665 P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:54:05.055444 23210 ts_tablet_manager.cc:1367] T 57ab576948e346cbbe6715bf53f7c665 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20240502 01:54:05.056141 23348 tablet_bootstrap.cc:654] T a390a222562f425eabb11d7f52cbe9a6 P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:05.056175 23210 raft_consensus.cc:348] T 57ab576948e346cbbe6715bf53f7c665 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:05.056429 23210 raft_consensus.cc:374] T 57ab576948e346cbbe6715bf53f7c665 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:05.056515 23210 raft_consensus.cc:729] T 57ab576948e346cbbe6715bf53f7c665 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:54:05.058353 23210 consensus_queue.cc:260] T 57ab576948e346cbbe6715bf53f7c665 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:05.063017 23210 ts_tablet_manager.cc:1398] T 57ab576948e346cbbe6715bf53f7c665 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240502 01:54:05.068751 23210 tablet_bootstrap.cc:492] T a390a222562f425eabb11d7f52cbe9a6 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:54:05.071909 23210 tablet_bootstrap.cc:654] T a390a222562f425eabb11d7f52cbe9a6 P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:05.091010 22940 consensus_queue.cc:1035] T 074a08b2b64640eb9784b6a23921ce8d P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:05.098249 23352 tablet_bootstrap.cc:492] T a390a222562f425eabb11d7f52cbe9a6 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:54:05.104697 23352 tablet_bootstrap.cc:654] T a390a222562f425eabb11d7f52cbe9a6 P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:05.116096 23352 tablet_bootstrap.cc:492] T a390a222562f425eabb11d7f52cbe9a6 P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:54:05.119189 23352 ts_tablet_manager.cc:1367] T a390a222562f425eabb11d7f52cbe9a6 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.006s
I20240502 01:54:05.122027 23352 raft_consensus.cc:348] T a390a222562f425eabb11d7f52cbe9a6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:05.122699 23352 raft_consensus.cc:374] T a390a222562f425eabb11d7f52cbe9a6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:05.122934 23352 raft_consensus.cc:729] T a390a222562f425eabb11d7f52cbe9a6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:54:05.123435 23352 consensus_queue.cc:260] T a390a222562f425eabb11d7f52cbe9a6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:05.129442 23352 ts_tablet_manager.cc:1398] T a390a222562f425eabb11d7f52cbe9a6 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240502 01:54:05.130936 22863 raft_consensus.cc:482] T a390a222562f425eabb11d7f52cbe9a6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:05.131579 22863 raft_consensus.cc:504] T a390a222562f425eabb11d7f52cbe9a6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:05.132295 22863 leader_election.cc:288] T a390a222562f425eabb11d7f52cbe9a6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:54:05.133699 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a390a222562f425eabb11d7f52cbe9a6" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
W20240502 01:54:05.134598 20302 leader_election.cc:341] T a390a222562f425eabb11d7f52cbe9a6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:05.135061 17474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a390a222562f425eabb11d7f52cbe9a6" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
W20240502 01:54:05.135776 20302 leader_election.cc:341] T a390a222562f425eabb11d7f52cbe9a6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:05.135900 20302 leader_election.cc:302] T a390a222562f425eabb11d7f52cbe9a6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9; no voters: 4da690c0fa13407a85c0689ce8de9ade, dfa1152987644bd781b46c7e42a7a685
I20240502 01:54:05.136258 23210 tablet_bootstrap.cc:492] T a390a222562f425eabb11d7f52cbe9a6 P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:54:05.136293 22863 raft_consensus.cc:2726] T a390a222562f425eabb11d7f52cbe9a6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:54:05.136401 23210 ts_tablet_manager.cc:1367] T a390a222562f425eabb11d7f52cbe9a6 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.068s user 0.001s sys 0.005s
I20240502 01:54:05.137092 23210 raft_consensus.cc:348] T a390a222562f425eabb11d7f52cbe9a6 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:05.137318 23210 raft_consensus.cc:374] T a390a222562f425eabb11d7f52cbe9a6 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:05.137373 23210 raft_consensus.cc:729] T a390a222562f425eabb11d7f52cbe9a6 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
W20240502 01:54:05.138731 20433 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.42.72:42411, user_credentials={real_user=slave}} blocked reactor thread for 97786.9us
I20240502 01:54:05.140719 20476 tablet_service.cc:1467] Processing CreateTablet for tablet 7b2eae408b9843acaa61e698a3e44cff (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=b04d89d1ce13442db7041b4081ac54f4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:05.141191 20476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b2eae408b9843acaa61e698a3e44cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:05.151830 23210 consensus_queue.cc:260] T a390a222562f425eabb11d7f52cbe9a6 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:05.152429 23344 tablet_bootstrap.cc:492] T 57ab576948e346cbbe6715bf53f7c665 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:54:05.154026 23210 ts_tablet_manager.cc:1398] T a390a222562f425eabb11d7f52cbe9a6 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20240502 01:54:05.165416 23357 tablet_bootstrap.cc:492] T 7b2eae408b9843acaa61e698a3e44cff P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:54:05.167009 23357 tablet_bootstrap.cc:654] T 7b2eae408b9843acaa61e698a3e44cff P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:05.167457 23344 tablet_bootstrap.cc:654] T 57ab576948e346cbbe6715bf53f7c665 P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:05.169083 23348 tablet_bootstrap.cc:492] T a390a222562f425eabb11d7f52cbe9a6 P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:54:05.169134 23357 tablet_bootstrap.cc:492] T 7b2eae408b9843acaa61e698a3e44cff P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:54:05.169234 23348 ts_tablet_manager.cc:1367] T a390a222562f425eabb11d7f52cbe9a6 P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.116s user 0.019s sys 0.064s
I20240502 01:54:05.169262 23357 ts_tablet_manager.cc:1367] T 7b2eae408b9843acaa61e698a3e44cff P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 01:54:05.169941 23348 raft_consensus.cc:348] T a390a222562f425eabb11d7f52cbe9a6 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:05.170135 23348 raft_consensus.cc:374] T a390a222562f425eabb11d7f52cbe9a6 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:05.170035 23357 raft_consensus.cc:348] T 7b2eae408b9843acaa61e698a3e44cff P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:05.170186 23348 raft_consensus.cc:729] T a390a222562f425eabb11d7f52cbe9a6 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:54:05.170212 23357 raft_consensus.cc:374] T 7b2eae408b9843acaa61e698a3e44cff P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:05.170281 23357 raft_consensus.cc:729] T 7b2eae408b9843acaa61e698a3e44cff P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:54:05.170396 23348 consensus_queue.cc:260] T a390a222562f425eabb11d7f52cbe9a6 P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:05.170918 23348 ts_tablet_manager.cc:1398] T a390a222562f425eabb11d7f52cbe9a6 P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:54:05.171162 23348 tablet_bootstrap.cc:492] T 7b2eae408b9843acaa61e698a3e44cff P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:54:05.170487 23357 consensus_queue.cc:260] T 7b2eae408b9843acaa61e698a3e44cff P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:05.172452 23348 tablet_bootstrap.cc:654] T 7b2eae408b9843acaa61e698a3e44cff P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:05.172474 23357 ts_tablet_manager.cc:1398] T 7b2eae408b9843acaa61e698a3e44cff P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:54:05.175599 23348 tablet_bootstrap.cc:492] T 7b2eae408b9843acaa61e698a3e44cff P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:54:05.175943 23348 ts_tablet_manager.cc:1367] T 7b2eae408b9843acaa61e698a3e44cff P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240502 01:54:05.176455 23348 raft_consensus.cc:348] T 7b2eae408b9843acaa61e698a3e44cff P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:05.176614 23348 raft_consensus.cc:374] T 7b2eae408b9843acaa61e698a3e44cff P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:05.176667 23348 raft_consensus.cc:729] T 7b2eae408b9843acaa61e698a3e44cff P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:54:05.176848 23348 consensus_queue.cc:260] T 7b2eae408b9843acaa61e698a3e44cff P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:05.177347 23348 ts_tablet_manager.cc:1398] T 7b2eae408b9843acaa61e698a3e44cff P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:54:05.179834 22991 raft_consensus.cc:482] T 4c22bfa55cbf41628539aaf8a47c6c1c P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:05.180011 22991 raft_consensus.cc:504] T 4c22bfa55cbf41628539aaf8a47c6c1c P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:05.180706 22991 leader_election.cc:288] T 4c22bfa55cbf41628539aaf8a47c6c1c P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:54:05.181799 17866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c22bfa55cbf41628539aaf8a47c6c1c" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:54:05.182011 17866 raft_consensus.cc:2454] T 4c22bfa55cbf41628539aaf8a47c6c1c P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 0.
I20240502 01:54:05.182109 23344 tablet_bootstrap.cc:492] T 57ab576948e346cbbe6715bf53f7c665 P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:54:05.182288 23344 ts_tablet_manager.cc:1367] T 57ab576948e346cbbe6715bf53f7c665 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.007s
I20240502 01:54:05.182293 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c22bfa55cbf41628539aaf8a47c6c1c" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:54:05.182500 17737 raft_consensus.cc:2454] T 4c22bfa55cbf41628539aaf8a47c6c1c P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 0.
I20240502 01:54:05.182896 20565 leader_election.cc:302] T 4c22bfa55cbf41628539aaf8a47c6c1c P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb31bceba76d466db24b9341c04900c3, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:54:05.183001 23344 raft_consensus.cc:348] T 57ab576948e346cbbe6715bf53f7c665 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:05.183185 23344 raft_consensus.cc:374] T 57ab576948e346cbbe6715bf53f7c665 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:05.183501 23344 raft_consensus.cc:729] T 57ab576948e346cbbe6715bf53f7c665 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:54:05.183858 23344 consensus_queue.cc:260] T 57ab576948e346cbbe6715bf53f7c665 P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:05.184510 23344 ts_tablet_manager.cc:1398] T 57ab576948e346cbbe6715bf53f7c665 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:54:05.184895 23344 tablet_bootstrap.cc:492] T 7b2eae408b9843acaa61e698a3e44cff P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:54:05.186559 23344 tablet_bootstrap.cc:654] T 7b2eae408b9843acaa61e698a3e44cff P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:05.188469 22991 raft_consensus.cc:2781] T 4c22bfa55cbf41628539aaf8a47c6c1c P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:05.188594 22991 raft_consensus.cc:482] T 4c22bfa55cbf41628539aaf8a47c6c1c P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:05.188683 22991 raft_consensus.cc:3037] T 4c22bfa55cbf41628539aaf8a47c6c1c P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:05.190681 22991 raft_consensus.cc:504] T 4c22bfa55cbf41628539aaf8a47c6c1c P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:05.191694 17866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c22bfa55cbf41628539aaf8a47c6c1c" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:54:05.191874 17866 raft_consensus.cc:3037] T 4c22bfa55cbf41628539aaf8a47c6c1c P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:05.194348 17866 raft_consensus.cc:2454] T 4c22bfa55cbf41628539aaf8a47c6c1c P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 1.
I20240502 01:54:05.194780 20565 leader_election.cc:302] T 4c22bfa55cbf41628539aaf8a47c6c1c P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb31bceba76d466db24b9341c04900c3, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:54:05.195413 22940 raft_consensus.cc:2781] T 4c22bfa55cbf41628539aaf8a47c6c1c P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:05.195555 22940 raft_consensus.cc:686] T 4c22bfa55cbf41628539aaf8a47c6c1c P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Becoming Leader. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Running, Role: LEADER
I20240502 01:54:05.195848 22940 consensus_queue.cc:237] T 4c22bfa55cbf41628539aaf8a47c6c1c P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:05.201296 22568 raft_consensus.cc:482] T a390a222562f425eabb11d7f52cbe9a6 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:05.201427 22568 raft_consensus.cc:504] T a390a222562f425eabb11d7f52cbe9a6 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:05.201994 22568 leader_election.cc:288] T a390a222562f425eabb11d7f52cbe9a6 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:54:05.202569 17474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a390a222562f425eabb11d7f52cbe9a6" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:54:05.202731 22568 raft_consensus.cc:482] T 57ab576948e346cbbe6715bf53f7c665 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:05.202776 17474 raft_consensus.cc:2454] T a390a222562f425eabb11d7f52cbe9a6 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:54:05.202849 20365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a390a222562f425eabb11d7f52cbe9a6" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:54:05.202859 22568 raft_consensus.cc:504] T 57ab576948e346cbbe6715bf53f7c665 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:05.203051 20365 raft_consensus.cc:2454] T a390a222562f425eabb11d7f52cbe9a6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:54:05.203230 17672 leader_election.cc:302] T a390a222562f425eabb11d7f52cbe9a6 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:54:05.203445 22568 leader_election.cc:288] T 57ab576948e346cbbe6715bf53f7c665 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:54:05.203581 22568 raft_consensus.cc:2781] T a390a222562f425eabb11d7f52cbe9a6 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:05.203639 22568 raft_consensus.cc:482] T a390a222562f425eabb11d7f52cbe9a6 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:05.203706 22568 raft_consensus.cc:3037] T a390a222562f425eabb11d7f52cbe9a6 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:05.205519 17866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ab576948e346cbbe6715bf53f7c665" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:54:05.205580 17605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ab576948e346cbbe6715bf53f7c665" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:54:05.205785 17605 raft_consensus.cc:2454] T 57ab576948e346cbbe6715bf53f7c665 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:54:05.205946 17866 raft_consensus.cc:2454] T 57ab576948e346cbbe6715bf53f7c665 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:54:05.206228 17670 leader_election.cc:302] T 57ab576948e346cbbe6715bf53f7c665 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:54:05.206593 22991 leader_election.cc:288] T 4c22bfa55cbf41628539aaf8a47c6c1c P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 election: Requested vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:54:05.207849 22568 raft_consensus.cc:504] T a390a222562f425eabb11d7f52cbe9a6 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:05.208410 22568 leader_election.cc:288] T a390a222562f425eabb11d7f52cbe9a6 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Requested vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:54:05.208547 22568 raft_consensus.cc:2781] T 57ab576948e346cbbe6715bf53f7c665 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:05.208609 22568 raft_consensus.cc:482] T 57ab576948e346cbbe6715bf53f7c665 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:05.208690 22568 raft_consensus.cc:3037] T 57ab576948e346cbbe6715bf53f7c665 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:05.211687 22568 raft_consensus.cc:504] T 57ab576948e346cbbe6715bf53f7c665 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:05.213496 23344 tablet_bootstrap.cc:492] T 7b2eae408b9843acaa61e698a3e44cff P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:54:05.213649 23344 ts_tablet_manager.cc:1367] T 7b2eae408b9843acaa61e698a3e44cff P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.004s
I20240502 01:54:05.214329 17328 catalog_manager.cc:5484] T 4c22bfa55cbf41628539aaf8a47c6c1c P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: term changed from 0 to 1, leader changed from <none> to f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72). New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:05.215191 17605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ab576948e346cbbe6715bf53f7c665" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:54:05.215382 17605 raft_consensus.cc:3037] T 57ab576948e346cbbe6715bf53f7c665 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:05.214360 23344 raft_consensus.cc:348] T 7b2eae408b9843acaa61e698a3e44cff P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:05.217036 23344 raft_consensus.cc:374] T 7b2eae408b9843acaa61e698a3e44cff P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:05.217121 23344 raft_consensus.cc:729] T 7b2eae408b9843acaa61e698a3e44cff P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:54:05.217362 23344 consensus_queue.cc:260] T 7b2eae408b9843acaa61e698a3e44cff P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:05.217825 17605 raft_consensus.cc:2454] T 57ab576948e346cbbe6715bf53f7c665 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:54:05.217940 23344 ts_tablet_manager.cc:1398] T 7b2eae408b9843acaa61e698a3e44cff P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:54:05.218132 17864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ab576948e346cbbe6715bf53f7c665" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:54:05.218297 17864 raft_consensus.cc:3037] T 57ab576948e346cbbe6715bf53f7c665 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:05.219925 17864 raft_consensus.cc:2454] T 57ab576948e346cbbe6715bf53f7c665 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:54:05.220240 17673 leader_election.cc:302] T 57ab576948e346cbbe6715bf53f7c665 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb31bceba76d466db24b9341c04900c3, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:54:05.223374 22568 leader_election.cc:288] T 57ab576948e346cbbe6715bf53f7c665 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Requested vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:54:05.223853 22568 raft_consensus.cc:2781] T 57ab576948e346cbbe6715bf53f7c665 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:05.224020 22568 raft_consensus.cc:686] T 57ab576948e346cbbe6715bf53f7c665 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Becoming Leader. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Running, Role: LEADER
I20240502 01:54:05.224354 22568 consensus_queue.cc:237] T 57ab576948e346cbbe6715bf53f7c665 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:05.226014 17474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a390a222562f425eabb11d7f52cbe9a6" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:54:05.226222 17474 raft_consensus.cc:3037] T a390a222562f425eabb11d7f52cbe9a6 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:05.228920 17474 raft_consensus.cc:2454] T a390a222562f425eabb11d7f52cbe9a6 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:54:05.229750 20365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a390a222562f425eabb11d7f52cbe9a6" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
I20240502 01:54:05.229941 20365 raft_consensus.cc:3037] T a390a222562f425eabb11d7f52cbe9a6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:05.232729 20365 raft_consensus.cc:2454] T a390a222562f425eabb11d7f52cbe9a6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:54:05.233812 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c22bfa55cbf41628539aaf8a47c6c1c" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:54:05.233968 17737 raft_consensus.cc:3037] T 4c22bfa55cbf41628539aaf8a47c6c1c P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:05.237906 17737 raft_consensus.cc:2454] T 4c22bfa55cbf41628539aaf8a47c6c1c P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 1.
I20240502 01:54:05.239555 17672 leader_election.cc:302] T a390a222562f425eabb11d7f52cbe9a6 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:54:05.239845 23363 raft_consensus.cc:2781] T a390a222562f425eabb11d7f52cbe9a6 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:05.239990 23363 raft_consensus.cc:686] T a390a222562f425eabb11d7f52cbe9a6 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Becoming Leader. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Running, Role: LEADER
I20240502 01:54:05.240406 23363 consensus_queue.cc:237] T a390a222562f425eabb11d7f52cbe9a6 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:05.261574 20476 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:05.274097 17330 catalog_manager.cc:5484] T 57ab576948e346cbbe6715bf53f7c665 P dfa1152987644bd781b46c7e42a7a685 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfa1152987644bd781b46c7e42a7a685 (127.11.42.67). New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:05.288928 17330 catalog_manager.cc:5484] T a390a222562f425eabb11d7f52cbe9a6 P dfa1152987644bd781b46c7e42a7a685 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfa1152987644bd781b46c7e42a7a685 (127.11.42.67). New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:05.349445 17587 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:05.392758 17456 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:54:05.395530 17330 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:54:05.422470 17737 raft_consensus.cc:1261] T 4c22bfa55cbf41628539aaf8a47c6c1c P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:05.423553 22991 consensus_queue.cc:1035] T 4c22bfa55cbf41628539aaf8a47c6c1c P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:05.432853 17328 catalog_manager.cc:5576] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115) does not have the latest schema for tablet 4c22bfa55cbf41628539aaf8a47c6c1c (table rebalancer_dynamic_table [id=6f9eef247379455187a9871e7d04643f]). Expected version 1 got 0
I20240502 01:54:05.435986 17845 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:05.436523 20347 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240502 01:54:05.438138 17543 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.42.71:38623, user_credentials={real_user=slave}} blocked reactor thread for 70391.3us
I20240502 01:54:05.444432 17864 raft_consensus.cc:1261] T 4c22bfa55cbf41628539aaf8a47c6c1c P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:05.449666 20212 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:54:05.450811 23389 tablet.cc:1722] T 4c22bfa55cbf41628539aaf8a47c6c1c P f19cc6cb1e76460aa6711e49a5775091: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:05.460836 23389 tablet.cc:1620] T 4c22bfa55cbf41628539aaf8a47c6c1c P f19cc6cb1e76460aa6711e49a5775091: MemRowSet was empty: no flush needed.
I20240502 01:54:05.466287 23390 tablet.cc:1722] T 4c22bfa55cbf41628539aaf8a47c6c1c P dfa1152987644bd781b46c7e42a7a685: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:05.468291 17328 catalog_manager.cc:5576] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411) does not have the latest schema for tablet 4c22bfa55cbf41628539aaf8a47c6c1c (table rebalancer_dynamic_table [id=6f9eef247379455187a9871e7d04643f]). Expected version 1 got 0
I20240502 01:54:05.468945 17330 catalog_manager.cc:5576] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115) does not have the latest schema for tablet 4c22bfa55cbf41628539aaf8a47c6c1c (table rebalancer_dynamic_table [id=6f9eef247379455187a9871e7d04643f]). Expected version 1 got 0
I20240502 01:54:05.470599 23390 tablet.cc:1620] T 4c22bfa55cbf41628539aaf8a47c6c1c P dfa1152987644bd781b46c7e42a7a685: MemRowSet was empty: no flush needed.
I20240502 01:54:05.472162 22940 consensus_queue.cc:1035] T 4c22bfa55cbf41628539aaf8a47c6c1c P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:05.474221 17715 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:05.480819 17328 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=6f9eef247379455187a9871e7d04643f] alter complete (version 1)
I20240502 01:54:05.491267 20609 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:54:05.549062 23393 tablet.cc:1722] T 4c22bfa55cbf41628539aaf8a47c6c1c P bb31bceba76d466db24b9341c04900c3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:05.551492 17328 catalog_manager.cc:5576] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083) does not have the latest schema for tablet 4c22bfa55cbf41628539aaf8a47c6c1c (table rebalancer_dynamic_table [id=6f9eef247379455187a9871e7d04643f]). Expected version 1 got 0
I20240502 01:54:05.562052 23393 tablet.cc:1620] T 4c22bfa55cbf41628539aaf8a47c6c1c P bb31bceba76d466db24b9341c04900c3: MemRowSet was empty: no flush needed.
I20240502 01:54:05.573688 23347 raft_consensus.cc:482] T 7b2eae408b9843acaa61e698a3e44cff P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:05.573853 23347 raft_consensus.cc:504] T 7b2eae408b9843acaa61e698a3e44cff P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:05.574631 23347 leader_election.cc:288] T 7b2eae408b9843acaa61e698a3e44cff P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:54:05.574946 17474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b2eae408b9843acaa61e698a3e44cff" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:54:05.575157 17474 raft_consensus.cc:2454] T 7b2eae408b9843acaa61e698a3e44cff P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 0.
I20240502 01:54:05.575225 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b2eae408b9843acaa61e698a3e44cff" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:54:05.575487 17604 raft_consensus.cc:2454] T 7b2eae408b9843acaa61e698a3e44cff P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 0.
I20240502 01:54:05.575713 20433 leader_election.cc:302] T 7b2eae408b9843acaa61e698a3e44cff P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 4da690c0fa13407a85c0689ce8de9ade; no voters:
I20240502 01:54:05.576187 23347 raft_consensus.cc:2781] T 7b2eae408b9843acaa61e698a3e44cff P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:05.576331 23347 raft_consensus.cc:482] T 7b2eae408b9843acaa61e698a3e44cff P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:05.576426 23347 raft_consensus.cc:3037] T 7b2eae408b9843acaa61e698a3e44cff P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:05.580114 23347 raft_consensus.cc:504] T 7b2eae408b9843acaa61e698a3e44cff P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:05.581032 17474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b2eae408b9843acaa61e698a3e44cff" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:54:05.581184 17474 raft_consensus.cc:3037] T 7b2eae408b9843acaa61e698a3e44cff P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:05.581156 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b2eae408b9843acaa61e698a3e44cff" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:54:05.581333 17604 raft_consensus.cc:3037] T 7b2eae408b9843acaa61e698a3e44cff P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:05.584539 17604 raft_consensus.cc:2454] T 7b2eae408b9843acaa61e698a3e44cff P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 1.
I20240502 01:54:05.585069 20431 leader_election.cc:302] T 7b2eae408b9843acaa61e698a3e44cff P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 3fceccefba274b1dbbd8216af8b0bc29; no voters:
I20240502 01:54:05.585628 17474 raft_consensus.cc:2454] T 7b2eae408b9843acaa61e698a3e44cff P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 1.
I20240502 01:54:05.585932 22626 raft_consensus.cc:2781] T 7b2eae408b9843acaa61e698a3e44cff P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:05.586097 22626 raft_consensus.cc:686] T 7b2eae408b9843acaa61e698a3e44cff P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Running, Role: LEADER
I20240502 01:54:05.586376 22626 consensus_queue.cc:237] T 7b2eae408b9843acaa61e698a3e44cff P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:05.588400 17328 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:54:05.588735 23347 leader_election.cc:288] T 7b2eae408b9843acaa61e698a3e44cff P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 election: Requested vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:54:05.596753 23389 tablet.cc:1722] T 4c22bfa55cbf41628539aaf8a47c6c1c P f19cc6cb1e76460aa6711e49a5775091: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:05.597368 23389 tablet.cc:1620] T 4c22bfa55cbf41628539aaf8a47c6c1c P f19cc6cb1e76460aa6711e49a5775091: MemRowSet was empty: no flush needed.
I20240502 01:54:05.599149 17330 catalog_manager.cc:5576] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411) does not have the latest schema for tablet 4c22bfa55cbf41628539aaf8a47c6c1c (table rebalancer_dynamic_table [id=6f9eef247379455187a9871e7d04643f]). Expected version 2 got 1
I20240502 01:54:05.602943 23390 tablet.cc:1722] T 4c22bfa55cbf41628539aaf8a47c6c1c P dfa1152987644bd781b46c7e42a7a685: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:05.603387 23390 tablet.cc:1620] T 4c22bfa55cbf41628539aaf8a47c6c1c P dfa1152987644bd781b46c7e42a7a685: MemRowSet was empty: no flush needed.
I20240502 01:54:05.603684 23393 tablet.cc:1722] T 4c22bfa55cbf41628539aaf8a47c6c1c P bb31bceba76d466db24b9341c04900c3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:05.604276 23393 tablet.cc:1620] T 4c22bfa55cbf41628539aaf8a47c6c1c P bb31bceba76d466db24b9341c04900c3: MemRowSet was empty: no flush needed.
I20240502 01:54:05.619040 17327 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=6f9eef247379455187a9871e7d04643f] alter complete (version 2)
I20240502 01:54:05.624429 17328 catalog_manager.cc:5484] T 7b2eae408b9843acaa61e698a3e44cff P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71). New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:05.629377 23363 consensus_queue.cc:1035] T 57ab576948e346cbbe6715bf53f7c665 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:05.635638 23363 consensus_queue.cc:1035] T 57ab576948e346cbbe6715bf53f7c665 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:05.692142 20366 consensus_queue.cc:237] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } }
I20240502 01:54:05.694658 20630 raft_consensus.cc:1261] T 28463d45bfde44429d56179c1eec56b0 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:54:05.694885 20302 consensus_peers.cc:485] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Couldn't send request to peer 4da690c0fa13407a85c0689ce8de9ade. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 28463d45bfde44429d56179c1eec56b0. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:05.695173 17864 raft_consensus.cc:1261] T 28463d45bfde44429d56179c1eec56b0 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:05.695175 22863 consensus_queue.cc:1035] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:05.695719 22863 consensus_queue.cc:1035] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:05.698617 22876 raft_consensus.cc:2932] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } } }
I20240502 01:54:05.699960 17864 raft_consensus.cc:2932] T 28463d45bfde44429d56179c1eec56b0 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } } }
I20240502 01:54:05.702670 20630 raft_consensus.cc:2932] T 28463d45bfde44429d56179c1eec56b0 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } } }
I20240502 01:54:05.708175 17330 catalog_manager.cc:5484] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: config changed from index -1 to 2, NON_VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) added. New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:05.722506 17330 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:54:05.732966 23389 tablet.cc:1722] T 4c22bfa55cbf41628539aaf8a47c6c1c P f19cc6cb1e76460aa6711e49a5775091: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:05.733817 23393 tablet.cc:1722] T 4c22bfa55cbf41628539aaf8a47c6c1c P bb31bceba76d466db24b9341c04900c3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:05.735003 23390 tablet.cc:1722] T 4c22bfa55cbf41628539aaf8a47c6c1c P dfa1152987644bd781b46c7e42a7a685: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:05.746002 17330 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table_new_name [id=6f9eef247379455187a9871e7d04643f] alter complete (version 3)
I20240502 01:54:05.779989 20494 consensus_queue.cc:237] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:54:05.783979 20366 raft_consensus.cc:1261] T d10d76ac7b094fe9b230b1afcbed9ca6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:54:05.784015 20433 consensus_peers.cc:485] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d10d76ac7b094fe9b230b1afcbed9ca6. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:05.784478 20230 raft_consensus.cc:1261] T d10d76ac7b094fe9b230b1afcbed9ca6 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:05.784504 22626 consensus_queue.cc:1035] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:05.786784 22626 consensus_queue.cc:1035] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:05.787261 23343 raft_consensus.cc:2932] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:05.791371 20366 raft_consensus.cc:2932] T d10d76ac7b094fe9b230b1afcbed9ca6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:05.798461 17330 catalog_manager.cc:5484] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:05.812175 20230 raft_consensus.cc:2932] T d10d76ac7b094fe9b230b1afcbed9ca6 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:05.822989 17734 consensus_queue.cc:237] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } }
I20240502 01:54:05.825692 20630 raft_consensus.cc:1261] T 0d9f168de0d1410da2518abf936035ff P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:05.825757 17864 raft_consensus.cc:1261] T 0d9f168de0d1410da2518abf936035ff P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 01:54:05.826045 17673 consensus_peers.cc:485] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0d9f168de0d1410da2518abf936035ff. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:05.826228 23396 consensus_queue.cc:1035] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:05.826562 23364 consensus_queue.cc:1035] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:05.829272 23396 raft_consensus.cc:2932] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:54:05.829885 17864 raft_consensus.cc:2932] T 0d9f168de0d1410da2518abf936035ff P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:54:05.830498 20630 raft_consensus.cc:2932] T 0d9f168de0d1410da2518abf936035ff P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:54:05.836069 17327 catalog_manager.cc:5484] T 0d9f168de0d1410da2518abf936035ff P bb31bceba76d466db24b9341c04900c3 reported cstate change: config changed from index 6 to 7, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:54:05.838829 23363 consensus_queue.cc:1035] T a390a222562f425eabb11d7f52cbe9a6 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:05.844452 23364 consensus_queue.cc:1035] T a390a222562f425eabb11d7f52cbe9a6 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:05.853559 17330 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:05.853761 17330 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:05.858090 17330 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet 4c22bfa55cbf41628539aaf8a47c6c1c
I20240502 01:54:05.856447 20630 consensus_queue.cc:237] T 81f335c023444796ab933e19d809cc03 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } }
I20240502 01:54:05.859192 17845 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c22bfa55cbf41628539aaf8a47c6c1c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:05 UTC) from {username='slave'} at 127.0.0.1:47226
I20240502 01:54:05.859745 17715 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c22bfa55cbf41628539aaf8a47c6c1c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:05 UTC) from {username='slave'} at 127.0.0.1:52198
I20240502 01:54:05.862260 23436 tablet_replica.cc:330] T 4c22bfa55cbf41628539aaf8a47c6c1c P bb31bceba76d466db24b9341c04900c3: stopping tablet replica
I20240502 01:54:05.863502 23436 raft_consensus.cc:2229] T 4c22bfa55cbf41628539aaf8a47c6c1c P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:05.863976 23436 raft_consensus.cc:2258] T 4c22bfa55cbf41628539aaf8a47c6c1c P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:05.865255 23436 ts_tablet_manager.cc:1875] T 4c22bfa55cbf41628539aaf8a47c6c1c P bb31bceba76d466db24b9341c04900c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:05.866955 20609 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c22bfa55cbf41628539aaf8a47c6c1c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:05 UTC) from {username='slave'} at 127.0.0.1:33698
I20240502 01:54:05.872607 20494 raft_consensus.cc:1261] T 81f335c023444796ab933e19d809cc03 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:05.874004 20230 raft_consensus.cc:1261] T 81f335c023444796ab933e19d809cc03 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:54:05.874066 20562 consensus_peers.cc:485] T 81f335c023444796ab933e19d809cc03 P f19cc6cb1e76460aa6711e49a5775091 -> Peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): Couldn't send request to peer 3fceccefba274b1dbbd8216af8b0bc29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 81f335c023444796ab933e19d809cc03. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:05.874241 22991 consensus_queue.cc:1035] T 81f335c023444796ab933e19d809cc03 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:05.874698 23180 consensus_queue.cc:1035] T 81f335c023444796ab933e19d809cc03 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:05.876066 23439 tablet_replica.cc:330] T 4c22bfa55cbf41628539aaf8a47c6c1c P f19cc6cb1e76460aa6711e49a5775091: stopping tablet replica
I20240502 01:54:05.886260 23436 ts_tablet_manager.cc:1888] T 4c22bfa55cbf41628539aaf8a47c6c1c P bb31bceba76d466db24b9341c04900c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:05.886561 23439 raft_consensus.cc:2229] T 4c22bfa55cbf41628539aaf8a47c6c1c P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:05.887037 23439 raft_consensus.cc:2258] T 4c22bfa55cbf41628539aaf8a47c6c1c P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:05.888370 23436 log.cc:1196] T 4c22bfa55cbf41628539aaf8a47c6c1c P bb31bceba76d466db24b9341c04900c3: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/wals/4c22bfa55cbf41628539aaf8a47c6c1c
I20240502 01:54:05.889092 23436 ts_tablet_manager.cc:1909] T 4c22bfa55cbf41628539aaf8a47c6c1c P bb31bceba76d466db24b9341c04900c3: Deleting consensus metadata
I20240502 01:54:05.889498 23180 raft_consensus.cc:2932] T 81f335c023444796ab933e19d809cc03 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } } }
I20240502 01:54:05.890309 17322 catalog_manager.cc:4830] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): tablet 4c22bfa55cbf41628539aaf8a47c6c1c (table rebalancer_dynamic_table_new_name [id=6f9eef247379455187a9871e7d04643f]) successfully deleted
I20240502 01:54:05.890551 20494 raft_consensus.cc:2932] T 81f335c023444796ab933e19d809cc03 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } } }
I20240502 01:54:05.891180 23439 ts_tablet_manager.cc:1875] T 4c22bfa55cbf41628539aaf8a47c6c1c P f19cc6cb1e76460aa6711e49a5775091: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:05.889376 23437 tablet_replica.cc:330] T 4c22bfa55cbf41628539aaf8a47c6c1c P dfa1152987644bd781b46c7e42a7a685: stopping tablet replica
I20240502 01:54:05.894796 23437 raft_consensus.cc:2229] T 4c22bfa55cbf41628539aaf8a47c6c1c P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:05.895849 23437 raft_consensus.cc:2258] T 4c22bfa55cbf41628539aaf8a47c6c1c P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:05.896500 17330 catalog_manager.cc:5484] T 81f335c023444796ab933e19d809cc03 P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: config changed from index -1 to 2, NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) added. New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } } }
I20240502 01:54:05.897392 23439 ts_tablet_manager.cc:1888] T 4c22bfa55cbf41628539aaf8a47c6c1c P f19cc6cb1e76460aa6711e49a5775091: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:05.897549 23439 log.cc:1196] T 4c22bfa55cbf41628539aaf8a47c6c1c P f19cc6cb1e76460aa6711e49a5775091: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal/wals/4c22bfa55cbf41628539aaf8a47c6c1c
I20240502 01:54:05.898545 23439 ts_tablet_manager.cc:1909] T 4c22bfa55cbf41628539aaf8a47c6c1c P f19cc6cb1e76460aa6711e49a5775091: Deleting consensus metadata
I20240502 01:54:05.900137 17321 catalog_manager.cc:4830] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): tablet 4c22bfa55cbf41628539aaf8a47c6c1c (table rebalancer_dynamic_table_new_name [id=6f9eef247379455187a9871e7d04643f]) successfully deleted
I20240502 01:54:05.901226 20230 raft_consensus.cc:2932] T 81f335c023444796ab933e19d809cc03 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } } }
I20240502 01:54:05.905558 23437 ts_tablet_manager.cc:1875] T 4c22bfa55cbf41628539aaf8a47c6c1c P dfa1152987644bd781b46c7e42a7a685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:05.911669 23437 ts_tablet_manager.cc:1888] T 4c22bfa55cbf41628539aaf8a47c6c1c P dfa1152987644bd781b46c7e42a7a685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:05.911760 23437 log.cc:1196] T 4c22bfa55cbf41628539aaf8a47c6c1c P dfa1152987644bd781b46c7e42a7a685: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/wals/4c22bfa55cbf41628539aaf8a47c6c1c
I20240502 01:54:05.912075 23437 ts_tablet_manager.cc:1909] T 4c22bfa55cbf41628539aaf8a47c6c1c P dfa1152987644bd781b46c7e42a7a685: Deleting consensus metadata
I20240502 01:54:05.913139 17321 catalog_manager.cc:4830] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): tablet 4c22bfa55cbf41628539aaf8a47c6c1c (table rebalancer_dynamic_table_new_name [id=6f9eef247379455187a9871e7d04643f]) successfully deleted
I20240502 01:54:05.922230 20494 consensus_queue.cc:237] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } }
I20240502 01:54:05.924731 20629 raft_consensus.cc:1261] T fac0ba26fabd4643bc38b7559faa471c P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:05.924719 20366 raft_consensus.cc:1261] T fac0ba26fabd4643bc38b7559faa471c P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:54:05.924918 20433 consensus_peers.cc:485] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1 -> Peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Couldn't send request to peer 4da690c0fa13407a85c0689ce8de9ade. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fac0ba26fabd4643bc38b7559faa471c. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:05.926143 22626 consensus_queue.cc:1035] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:05.926429 23343 consensus_queue.cc:1035] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:05.929543 23343 raft_consensus.cc:2932] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } } }
I20240502 01:54:05.930780 20366 raft_consensus.cc:2932] T fac0ba26fabd4643bc38b7559faa471c P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } } }
I20240502 01:54:05.935839 17328 catalog_manager.cc:5484] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: config changed from index -1 to 2, NON_VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) added. New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:05.947016 20629 raft_consensus.cc:2932] T fac0ba26fabd4643bc38b7559faa471c P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } } }
I20240502 01:54:05.955835 20366 consensus_queue.cc:237] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:54:05.958242 17474 raft_consensus.cc:1261] T a212c17ba64842c99b3d6a78a3b7cc8a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:05.958420 20629 raft_consensus.cc:1261] T a212c17ba64842c99b3d6a78a3b7cc8a P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:05.958838 23410 consensus_queue.cc:1035] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 01:54:05.959122 20302 consensus_peers.cc:485] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a212c17ba64842c99b3d6a78a3b7cc8a. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:05.959437 23406 consensus_queue.cc:1035] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:05.961673 23410 raft_consensus.cc:2932] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:05.962327 17474 raft_consensus.cc:2932] T a212c17ba64842c99b3d6a78a3b7cc8a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:05.962745 20629 raft_consensus.cc:2932] T a212c17ba64842c99b3d6a78a3b7cc8a P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:05.968729 17328 catalog_manager.cc:5484] T a212c17ba64842c99b3d6a78a3b7cc8a P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: config changed from index 4 to 5, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:05.968950 17334 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:54:05.984640 20494 consensus_queue.cc:237] T 139304da92ab4f48b8a08fa24ed60c95 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } }
I20240502 01:54:05.998772 20629 raft_consensus.cc:1261] T 139304da92ab4f48b8a08fa24ed60c95 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:05.999269 20230 raft_consensus.cc:1261] T 139304da92ab4f48b8a08fa24ed60c95 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:06.000391 23347 consensus_queue.cc:1035] T 139304da92ab4f48b8a08fa24ed60c95 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:06.000974 23347 consensus_queue.cc:1035] T 139304da92ab4f48b8a08fa24ed60c95 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:54:06.003787 20434 consensus_peers.cc:485] T 139304da92ab4f48b8a08fa24ed60c95 P 2f306549407b4bb281ccf2994035e4f1 -> Peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): Couldn't send request to peer bb31bceba76d466db24b9341c04900c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 139304da92ab4f48b8a08fa24ed60c95. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:06.004014 23343 raft_consensus.cc:2932] T 139304da92ab4f48b8a08fa24ed60c95 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } } }
I20240502 01:54:06.004518 20629 raft_consensus.cc:2932] T 139304da92ab4f48b8a08fa24ed60c95 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } } }
I20240502 01:54:06.006404 17715 tablet_service.cc:1467] Processing CreateTablet for tablet 44809ca119e145ddab852626a5af007f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a7102d81aee247d087c72c0dd2bb3c44]), partition=
I20240502 01:54:06.006738 17715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44809ca119e145ddab852626a5af007f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:06.006852 20476 tablet_service.cc:1467] Processing CreateTablet for tablet 44809ca119e145ddab852626a5af007f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a7102d81aee247d087c72c0dd2bb3c44]), partition=
I20240502 01:54:06.007174 20476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44809ca119e145ddab852626a5af007f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:06.007524 20212 tablet_service.cc:1467] Processing CreateTablet for tablet 44809ca119e145ddab852626a5af007f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a7102d81aee247d087c72c0dd2bb3c44]), partition=
I20240502 01:54:06.007829 20212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44809ca119e145ddab852626a5af007f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:06.012527 17328 catalog_manager.cc:5484] T 139304da92ab4f48b8a08fa24ed60c95 P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: config changed from index -1 to 2, NON_VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) added. New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } } }
I20240502 01:54:06.015440 23469 tablet_bootstrap.cc:492] T 44809ca119e145ddab852626a5af007f P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:54:06.017678 23469 tablet_bootstrap.cc:654] T 44809ca119e145ddab852626a5af007f P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:06.025192 23471 tablet_bootstrap.cc:492] T 44809ca119e145ddab852626a5af007f P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:54:06.027482 23471 tablet_bootstrap.cc:654] T 44809ca119e145ddab852626a5af007f P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:06.030416 23469 tablet_bootstrap.cc:492] T 44809ca119e145ddab852626a5af007f P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:54:06.030601 23469 ts_tablet_manager.cc:1367] T 44809ca119e145ddab852626a5af007f P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.013s
I20240502 01:54:06.032372 23469 raft_consensus.cc:348] T 44809ca119e145ddab852626a5af007f P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:06.032675 23469 raft_consensus.cc:374] T 44809ca119e145ddab852626a5af007f P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:06.034162 23469 raft_consensus.cc:729] T 44809ca119e145ddab852626a5af007f P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:54:06.034518 23469 consensus_queue.cc:260] T 44809ca119e145ddab852626a5af007f P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:06.035274 23469 ts_tablet_manager.cc:1398] T 44809ca119e145ddab852626a5af007f P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 01:54:06.035385 20233 raft_consensus.cc:2932] T 139304da92ab4f48b8a08fa24ed60c95 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } } }
I20240502 01:54:06.044992 23468 tablet_bootstrap.cc:492] T 44809ca119e145ddab852626a5af007f P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:54:06.054549 23468 tablet_bootstrap.cc:654] T 44809ca119e145ddab852626a5af007f P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:06.056886 23471 tablet_bootstrap.cc:492] T 44809ca119e145ddab852626a5af007f P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:54:06.057057 23471 ts_tablet_manager.cc:1367] T 44809ca119e145ddab852626a5af007f P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.032s user 0.007s sys 0.014s
I20240502 01:54:06.057848 23471 raft_consensus.cc:348] T 44809ca119e145ddab852626a5af007f P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:06.058108 23471 raft_consensus.cc:374] T 44809ca119e145ddab852626a5af007f P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:06.058219 23471 raft_consensus.cc:729] T 44809ca119e145ddab852626a5af007f P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:54:06.058518 23471 consensus_queue.cc:260] T 44809ca119e145ddab852626a5af007f P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:06.061194 23471 ts_tablet_manager.cc:1398] T 44809ca119e145ddab852626a5af007f P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:54:06.062312 23468 tablet_bootstrap.cc:492] T 44809ca119e145ddab852626a5af007f P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:54:06.062466 23468 ts_tablet_manager.cc:1367] T 44809ca119e145ddab852626a5af007f P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20240502 01:54:06.063143 23468 raft_consensus.cc:348] T 44809ca119e145ddab852626a5af007f P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:06.063364 23468 raft_consensus.cc:374] T 44809ca119e145ddab852626a5af007f P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:06.063457 23468 raft_consensus.cc:729] T 44809ca119e145ddab852626a5af007f P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:54:06.063709 23468 consensus_queue.cc:260] T 44809ca119e145ddab852626a5af007f P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:06.064244 23468 ts_tablet_manager.cc:1398] T 44809ca119e145ddab852626a5af007f P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:54:06.065449 20629 consensus_queue.cc:237] T 3ff84ce60ac1438d92b2c944bb27aabb P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } }
I20240502 01:54:06.074508 20233 raft_consensus.cc:1261] T 3ff84ce60ac1438d92b2c944bb27aabb P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:54:06.074514 20562 consensus_peers.cc:485] T 3ff84ce60ac1438d92b2c944bb27aabb P f19cc6cb1e76460aa6711e49a5775091 -> Peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): Couldn't send request to peer 3fceccefba274b1dbbd8216af8b0bc29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3ff84ce60ac1438d92b2c944bb27aabb. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:06.074604 20494 raft_consensus.cc:1261] T 3ff84ce60ac1438d92b2c944bb27aabb P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:06.075052 22991 consensus_queue.cc:1035] T 3ff84ce60ac1438d92b2c944bb27aabb P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:06.075372 22940 consensus_queue.cc:1035] T 3ff84ce60ac1438d92b2c944bb27aabb P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:06.080305 22940 raft_consensus.cc:2932] T 3ff84ce60ac1438d92b2c944bb27aabb P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } } }
I20240502 01:54:06.083138 20494 raft_consensus.cc:2932] T 3ff84ce60ac1438d92b2c944bb27aabb P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } } }
I20240502 01:54:06.088196 17327 catalog_manager.cc:5484] T 3ff84ce60ac1438d92b2c944bb27aabb P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: config changed from index -1 to 2, NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) added. New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:06.095033 20233 raft_consensus.cc:2932] T 3ff84ce60ac1438d92b2c944bb27aabb P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } } }
I20240502 01:54:06.113272 20233 consensus_queue.cc:237] T bffb870715a243d992107583d7c17f1b P fdc2ac372492454c85387589a91f63af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } }
I20240502 01:54:06.115407 17605 raft_consensus.cc:1261] T bffb870715a243d992107583d7c17f1b P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:54:06.115861 20167 consensus_peers.cc:485] T bffb870715a243d992107583d7c17f1b P fdc2ac372492454c85387589a91f63af -> Peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Couldn't send request to peer 4da690c0fa13407a85c0689ce8de9ade. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bffb870715a243d992107583d7c17f1b. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:06.115857 20629 raft_consensus.cc:1261] T bffb870715a243d992107583d7c17f1b P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:06.115994 23340 consensus_queue.cc:1035] T bffb870715a243d992107583d7c17f1b P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:06.116361 23339 consensus_queue.cc:1035] T bffb870715a243d992107583d7c17f1b P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:06.124199 23339 raft_consensus.cc:2932] T bffb870715a243d992107583d7c17f1b P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } } }
I20240502 01:54:06.127228 20629 raft_consensus.cc:2932] T bffb870715a243d992107583d7c17f1b P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } } }
I20240502 01:54:06.130448 17605 raft_consensus.cc:2932] T bffb870715a243d992107583d7c17f1b P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } } }
I20240502 01:54:06.131863 17327 catalog_manager.cc:5484] T bffb870715a243d992107583d7c17f1b P fdc2ac372492454c85387589a91f63af reported cstate change: config changed from index -1 to 2, NON_VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) added. New cstate: current_term: 1 leader_uuid: "fdc2ac372492454c85387589a91f63af" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:06.139824 22626 consensus_queue.cc:1035] T 7b2eae408b9843acaa61e698a3e44cff P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:06.146121 17327 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_extra_table_36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:06.163964 17456 tablet_service.cc:1467] Processing CreateTablet for tablet 29fb2b31cd9c4c8fb5923bb849a097d9 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=a95d02d60f11435a8aafb0c271c0aff0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:06.164466 17456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29fb2b31cd9c4c8fb5923bb849a097d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:06.168807 17716 tablet_service.cc:1467] Processing CreateTablet for tablet ad067712087e4f78885bd8e1cf102694 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=a95d02d60f11435a8aafb0c271c0aff0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:06.169229 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad067712087e4f78885bd8e1cf102694. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:06.172622 20347 tablet_service.cc:1467] Processing CreateTablet for tablet ad067712087e4f78885bd8e1cf102694 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=a95d02d60f11435a8aafb0c271c0aff0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:06.173086 20347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad067712087e4f78885bd8e1cf102694. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:06.168763 17715 tablet_service.cc:1467] Processing CreateTablet for tablet ee28f4aa31504dbfac977c4178aefdb1 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=a95d02d60f11435a8aafb0c271c0aff0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:06.175244 17715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee28f4aa31504dbfac977c4178aefdb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:06.182057 20476 tablet_service.cc:1467] Processing CreateTablet for tablet ee28f4aa31504dbfac977c4178aefdb1 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=a95d02d60f11435a8aafb0c271c0aff0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:06.182516 20476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee28f4aa31504dbfac977c4178aefdb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:06.182591 20478 tablet_service.cc:1467] Processing CreateTablet for tablet ad067712087e4f78885bd8e1cf102694 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=a95d02d60f11435a8aafb0c271c0aff0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:06.182778 20609 tablet_service.cc:1467] Processing CreateTablet for tablet ee28f4aa31504dbfac977c4178aefdb1 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=a95d02d60f11435a8aafb0c271c0aff0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:06.182917 20478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad067712087e4f78885bd8e1cf102694. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:06.183202 20609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee28f4aa31504dbfac977c4178aefdb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:06.185421 20610 tablet_service.cc:1467] Processing CreateTablet for tablet 29fb2b31cd9c4c8fb5923bb849a097d9 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=a95d02d60f11435a8aafb0c271c0aff0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:06.185796 20610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29fb2b31cd9c4c8fb5923bb849a097d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:06.182055 20477 tablet_service.cc:1467] Processing CreateTablet for tablet 29fb2b31cd9c4c8fb5923bb849a097d9 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=a95d02d60f11435a8aafb0c271c0aff0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:06.189509 20477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29fb2b31cd9c4c8fb5923bb849a097d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:06.200637 23469 tablet_bootstrap.cc:492] T ad067712087e4f78885bd8e1cf102694 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:54:06.201762 23343 consensus_queue.cc:1035] T 7b2eae408b9843acaa61e698a3e44cff P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:06.202229 23469 tablet_bootstrap.cc:654] T ad067712087e4f78885bd8e1cf102694 P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:06.212430 23500 tablet_bootstrap.cc:492] T ad067712087e4f78885bd8e1cf102694 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:54:06.217998 23503 tablet_bootstrap.cc:492] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:54:06.218282 23468 tablet_bootstrap.cc:492] T ad067712087e4f78885bd8e1cf102694 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:54:06.219849 23468 tablet_bootstrap.cc:654] T ad067712087e4f78885bd8e1cf102694 P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:06.222666 23503 tablet_bootstrap.cc:654] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:06.224013 23500 tablet_bootstrap.cc:654] T ad067712087e4f78885bd8e1cf102694 P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:06.253621 23503 tablet_bootstrap.cc:492] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:54:06.253834 23503 ts_tablet_manager.cc:1367] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.003s
I20240502 01:54:06.253996 23500 tablet_bootstrap.cc:492] T ad067712087e4f78885bd8e1cf102694 P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:54:06.254168 23500 ts_tablet_manager.cc:1367] T ad067712087e4f78885bd8e1cf102694 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.000s
I20240502 01:54:06.254756 23503 raft_consensus.cc:348] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:06.255029 23503 raft_consensus.cc:374] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:06.254984 23500 raft_consensus.cc:348] T ad067712087e4f78885bd8e1cf102694 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:06.255215 23500 raft_consensus.cc:374] T ad067712087e4f78885bd8e1cf102694 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:06.255141 23503 raft_consensus.cc:729] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:54:06.255414 23500 raft_consensus.cc:729] T ad067712087e4f78885bd8e1cf102694 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:54:06.255697 23500 consensus_queue.cc:260] T ad067712087e4f78885bd8e1cf102694 P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:06.256633 23503 consensus_queue.cc:260] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:06.261215 23500 ts_tablet_manager.cc:1398] T ad067712087e4f78885bd8e1cf102694 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240502 01:54:06.277655 23468 tablet_bootstrap.cc:492] T ad067712087e4f78885bd8e1cf102694 P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:54:06.278662 23468 ts_tablet_manager.cc:1367] T ad067712087e4f78885bd8e1cf102694 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.060s user 0.002s sys 0.010s
I20240502 01:54:06.282001 23469 tablet_bootstrap.cc:492] T ad067712087e4f78885bd8e1cf102694 P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:54:06.283762 23503 ts_tablet_manager.cc:1398] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.030s user 0.001s sys 0.001s
I20240502 01:54:06.283797 23469 ts_tablet_manager.cc:1367] T ad067712087e4f78885bd8e1cf102694 P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.083s user 0.003s sys 0.018s
I20240502 01:54:06.283169 23468 raft_consensus.cc:348] T ad067712087e4f78885bd8e1cf102694 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:06.284332 23468 raft_consensus.cc:374] T ad067712087e4f78885bd8e1cf102694 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:06.286046 23468 raft_consensus.cc:729] T ad067712087e4f78885bd8e1cf102694 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:54:06.286433 23468 consensus_queue.cc:260] T ad067712087e4f78885bd8e1cf102694 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:06.287789 23507 ts_tablet_manager.cc:902] T 28463d45bfde44429d56179c1eec56b0 P 4da690c0fa13407a85c0689ce8de9ade: Initiating tablet copy from peer 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:54:06.289076 23469 raft_consensus.cc:348] T ad067712087e4f78885bd8e1cf102694 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:06.290416 23469 raft_consensus.cc:374] T ad067712087e4f78885bd8e1cf102694 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:06.290931 23469 raft_consensus.cc:729] T ad067712087e4f78885bd8e1cf102694 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:54:06.291735 23469 consensus_queue.cc:260] T ad067712087e4f78885bd8e1cf102694 P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:06.293576 23468 ts_tablet_manager.cc:1398] T ad067712087e4f78885bd8e1cf102694 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240502 01:54:06.293582 23469 ts_tablet_manager.cc:1398] T ad067712087e4f78885bd8e1cf102694 P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240502 01:54:06.286593 23508 ts_tablet_manager.cc:902] T d10d76ac7b094fe9b230b1afcbed9ca6 P dfa1152987644bd781b46c7e42a7a685: Initiating tablet copy from peer 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:54:06.293874 23469 tablet_bootstrap.cc:492] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:54:06.295533 23469 tablet_bootstrap.cc:654] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:06.300383 23468 tablet_bootstrap.cc:492] T ee28f4aa31504dbfac977c4178aefdb1 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:54:06.303493 23468 tablet_bootstrap.cc:654] T ee28f4aa31504dbfac977c4178aefdb1 P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:06.300091 23347 raft_consensus.cc:482] T 44809ca119e145ddab852626a5af007f P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:06.306782 23347 raft_consensus.cc:504] T 44809ca119e145ddab852626a5af007f P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:06.307937 17734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44809ca119e145ddab852626a5af007f" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:54:06.308188 17734 raft_consensus.cc:2454] T 44809ca119e145ddab852626a5af007f P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 0.
I20240502 01:54:06.310922 20433 leader_election.cc:302] T 44809ca119e145ddab852626a5af007f P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:54:06.311406 23343 raft_consensus.cc:2781] T 44809ca119e145ddab852626a5af007f P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:06.311535 23343 raft_consensus.cc:482] T 44809ca119e145ddab852626a5af007f P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:06.311620 23343 raft_consensus.cc:3037] T 44809ca119e145ddab852626a5af007f P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:06.314208 23343 raft_consensus.cc:504] T 44809ca119e145ddab852626a5af007f P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:06.315088 20231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44809ca119e145ddab852626a5af007f" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:54:06.315320 20231 raft_consensus.cc:2454] T 44809ca119e145ddab852626a5af007f P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 0.
I20240502 01:54:06.316363 17734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44809ca119e145ddab852626a5af007f" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:54:06.316551 17734 raft_consensus.cc:3037] T 44809ca119e145ddab852626a5af007f P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:06.316759 23343 leader_election.cc:288] T 44809ca119e145ddab852626a5af007f P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 election: Requested vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:54:06.317265 20231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44809ca119e145ddab852626a5af007f" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:54:06.317488 20231 raft_consensus.cc:3037] T 44809ca119e145ddab852626a5af007f P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:06.319216 17734 raft_consensus.cc:2454] T 44809ca119e145ddab852626a5af007f P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 1.
I20240502 01:54:06.320070 20231 raft_consensus.cc:2454] T 44809ca119e145ddab852626a5af007f P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 1.
I20240502 01:54:06.320607 20434 leader_election.cc:302] T 44809ca119e145ddab852626a5af007f P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:54:06.322578 23343 raft_consensus.cc:2781] T 44809ca119e145ddab852626a5af007f P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:06.322839 23343 raft_consensus.cc:686] T 44809ca119e145ddab852626a5af007f P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Running, Role: LEADER
I20240502 01:54:06.323148 23343 consensus_queue.cc:237] T 44809ca119e145ddab852626a5af007f P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:06.311406 23347 leader_election.cc:288] T 44809ca119e145ddab852626a5af007f P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:54:06.337427 23499 tablet_bootstrap.cc:492] T ee28f4aa31504dbfac977c4178aefdb1 P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:54:06.339202 23499 tablet_bootstrap.cc:654] T ee28f4aa31504dbfac977c4178aefdb1 P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:06.345463 23512 ts_tablet_manager.cc:902] T 0d9f168de0d1410da2518abf936035ff P fdc2ac372492454c85387589a91f63af: Initiating tablet copy from peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:54:06.347821 23468 tablet_bootstrap.cc:492] T ee28f4aa31504dbfac977c4178aefdb1 P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:54:06.348069 23468 ts_tablet_manager.cc:1367] T ee28f4aa31504dbfac977c4178aefdb1 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.048s user 0.003s sys 0.000s
I20240502 01:54:06.348948 23468 raft_consensus.cc:348] T ee28f4aa31504dbfac977c4178aefdb1 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:06.349347 23468 raft_consensus.cc:374] T ee28f4aa31504dbfac977c4178aefdb1 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:06.349490 23468 raft_consensus.cc:729] T ee28f4aa31504dbfac977c4178aefdb1 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:54:06.349831 23468 consensus_queue.cc:260] T ee28f4aa31504dbfac977c4178aefdb1 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:06.350435 17328 catalog_manager.cc:5484] T 44809ca119e145ddab852626a5af007f P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71). New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:06.354207 22568 raft_consensus.cc:482] T ad067712087e4f78885bd8e1cf102694 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:06.354408 22568 raft_consensus.cc:504] T ad067712087e4f78885bd8e1cf102694 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:06.355058 22568 leader_election.cc:288] T ad067712087e4f78885bd8e1cf102694 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:54:06.355648 20366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad067712087e4f78885bd8e1cf102694" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:54:06.355875 20366 raft_consensus.cc:2454] T ad067712087e4f78885bd8e1cf102694 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:54:06.356278 17672 leader_election.cc:302] T ad067712087e4f78885bd8e1cf102694 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:54:06.356649 22568 raft_consensus.cc:2781] T ad067712087e4f78885bd8e1cf102694 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:06.356611 20496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad067712087e4f78885bd8e1cf102694" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
I20240502 01:54:06.356735 22568 raft_consensus.cc:482] T ad067712087e4f78885bd8e1cf102694 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:06.356835 22568 raft_consensus.cc:3037] T ad067712087e4f78885bd8e1cf102694 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:06.356822 20496 raft_consensus.cc:2454] T ad067712087e4f78885bd8e1cf102694 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 0.
I20240502 01:54:06.358093 23468 ts_tablet_manager.cc:1398] T ee28f4aa31504dbfac977c4178aefdb1 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240502 01:54:06.359611 22568 raft_consensus.cc:504] T ad067712087e4f78885bd8e1cf102694 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:06.361480 20366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad067712087e4f78885bd8e1cf102694" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
I20240502 01:54:06.361702 20366 raft_consensus.cc:3037] T ad067712087e4f78885bd8e1cf102694 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:06.362787 23499 tablet_bootstrap.cc:492] T ee28f4aa31504dbfac977c4178aefdb1 P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:54:06.363073 23469 tablet_bootstrap.cc:492] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:54:06.363255 23469 ts_tablet_manager.cc:1367] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.069s user 0.006s sys 0.009s
I20240502 01:54:06.364606 20366 raft_consensus.cc:2454] T ad067712087e4f78885bd8e1cf102694 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:54:06.363000 23499 ts_tablet_manager.cc:1367] T ee28f4aa31504dbfac977c4178aefdb1 P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.058s user 0.000s sys 0.003s
I20240502 01:54:06.364745 23469 raft_consensus.cc:348] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:06.364984 23469 raft_consensus.cc:374] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:06.365275 23469 raft_consensus.cc:729] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:54:06.365620 17672 leader_election.cc:302] T ad067712087e4f78885bd8e1cf102694 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:54:06.366075 23396 raft_consensus.cc:2781] T ad067712087e4f78885bd8e1cf102694 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:06.366204 23396 raft_consensus.cc:686] T ad067712087e4f78885bd8e1cf102694 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Becoming Leader. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Running, Role: LEADER
I20240502 01:54:06.366700 23469 consensus_queue.cc:260] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:06.371564 23396 consensus_queue.cc:237] T ad067712087e4f78885bd8e1cf102694 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:06.374074 22568 leader_election.cc:288] T ad067712087e4f78885bd8e1cf102694 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 election: Requested vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:54:06.374835 20494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad067712087e4f78885bd8e1cf102694" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
I20240502 01:54:06.375041 20494 raft_consensus.cc:3037] T ad067712087e4f78885bd8e1cf102694 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:06.376435 23469 ts_tablet_manager.cc:1398] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20240502 01:54:06.376960 20494 raft_consensus.cc:2454] T ad067712087e4f78885bd8e1cf102694 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 1.
I20240502 01:54:06.379091 23469 tablet_bootstrap.cc:492] T ee28f4aa31504dbfac977c4178aefdb1 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:54:06.381592 23469 tablet_bootstrap.cc:654] T ee28f4aa31504dbfac977c4178aefdb1 P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:06.388031 23499 raft_consensus.cc:348] T ee28f4aa31504dbfac977c4178aefdb1 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:06.388855 17328 catalog_manager.cc:5484] T ad067712087e4f78885bd8e1cf102694 P dfa1152987644bd781b46c7e42a7a685 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfa1152987644bd781b46c7e42a7a685 (127.11.42.67). New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:06.389478 23499 raft_consensus.cc:374] T ee28f4aa31504dbfac977c4178aefdb1 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:06.389601 23499 raft_consensus.cc:729] T ee28f4aa31504dbfac977c4178aefdb1 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:54:06.390760 23499 consensus_queue.cc:260] T ee28f4aa31504dbfac977c4178aefdb1 P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:06.394366 23499 ts_tablet_manager.cc:1398] T ee28f4aa31504dbfac977c4178aefdb1 P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.030s user 0.000s sys 0.002s
I20240502 01:54:06.395975 23499 tablet_bootstrap.cc:492] T 29fb2b31cd9c4c8fb5923bb849a097d9 P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:54:06.401086 23521 ts_tablet_manager.cc:902] T 81f335c023444796ab933e19d809cc03 P 3fceccefba274b1dbbd8216af8b0bc29: Initiating tablet copy from peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:54:06.401929 20629 consensus_queue.cc:237] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:54:06.404076 23469 tablet_bootstrap.cc:492] T ee28f4aa31504dbfac977c4178aefdb1 P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:54:06.404305 23469 ts_tablet_manager.cc:1367] T ee28f4aa31504dbfac977c4178aefdb1 P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.003s
I20240502 01:54:06.405315 23469 raft_consensus.cc:348] T ee28f4aa31504dbfac977c4178aefdb1 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:06.405622 23469 raft_consensus.cc:374] T ee28f4aa31504dbfac977c4178aefdb1 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:06.405750 23469 raft_consensus.cc:729] T ee28f4aa31504dbfac977c4178aefdb1 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:54:06.406143 23469 consensus_queue.cc:260] T ee28f4aa31504dbfac977c4178aefdb1 P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:06.408464 23469 ts_tablet_manager.cc:1398] T ee28f4aa31504dbfac977c4178aefdb1 P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:54:06.409085 23499 tablet_bootstrap.cc:654] T 29fb2b31cd9c4c8fb5923bb849a097d9 P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:06.418500 17866 raft_consensus.cc:1261] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:06.418555 20366 raft_consensus.cc:1261] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:54:06.419456 20564 consensus_peers.cc:485] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P f19cc6cb1e76460aa6711e49a5775091 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bfd7b1d1002c4b57a14d43ed9c6d9f50. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:06.420049 23180 consensus_queue.cc:1035] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:06.423000 22940 consensus_queue.cc:1035] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:06.425731 23507 tablet_copy_client.cc:308] T 28463d45bfde44429d56179c1eec56b0 P 4da690c0fa13407a85c0689ce8de9ade: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.70:39155
I20240502 01:54:06.426684 20387 tablet_copy_service.cc:140] P 6a36c952c51542c0bf7ea3ec2857a9f9: Received BeginTabletCopySession request for tablet 28463d45bfde44429d56179c1eec56b0 from peer 4da690c0fa13407a85c0689ce8de9ade ({username='slave'} at 127.11.42.65:57879)
I20240502 01:54:06.426862 20387 tablet_copy_service.cc:161] P 6a36c952c51542c0bf7ea3ec2857a9f9: Beginning new tablet copy session on tablet 28463d45bfde44429d56179c1eec56b0 from peer 4da690c0fa13407a85c0689ce8de9ade at {username='slave'} at 127.11.42.65:57879: session id = 4da690c0fa13407a85c0689ce8de9ade-28463d45bfde44429d56179c1eec56b0
I20240502 01:54:06.430341 20387 tablet_copy_source_session.cc:205] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:06.432237 23507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28463d45bfde44429d56179c1eec56b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:06.437393 23499 tablet_bootstrap.cc:492] T 29fb2b31cd9c4c8fb5923bb849a097d9 P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:54:06.437994 23499 ts_tablet_manager.cc:1367] T 29fb2b31cd9c4c8fb5923bb849a097d9 P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.000s
I20240502 01:54:06.438794 23531 ts_tablet_manager.cc:902] T 139304da92ab4f48b8a08fa24ed60c95 P bb31bceba76d466db24b9341c04900c3: Initiating tablet copy from peer 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:54:06.440701 23499 raft_consensus.cc:348] T 29fb2b31cd9c4c8fb5923bb849a097d9 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:06.441453 23508 tablet_copy_client.cc:308] T d10d76ac7b094fe9b230b1afcbed9ca6 P dfa1152987644bd781b46c7e42a7a685: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.71:38623
I20240502 01:54:06.442160 20518 tablet_copy_service.cc:140] P 2f306549407b4bb281ccf2994035e4f1: Received BeginTabletCopySession request for tablet d10d76ac7b094fe9b230b1afcbed9ca6 from peer dfa1152987644bd781b46c7e42a7a685 ({username='slave'} at 127.11.42.67:59949)
I20240502 01:54:06.442214 23499 raft_consensus.cc:374] T 29fb2b31cd9c4c8fb5923bb849a097d9 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:06.442329 23499 raft_consensus.cc:729] T 29fb2b31cd9c4c8fb5923bb849a097d9 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:54:06.442329 20518 tablet_copy_service.cc:161] P 2f306549407b4bb281ccf2994035e4f1: Beginning new tablet copy session on tablet d10d76ac7b094fe9b230b1afcbed9ca6 from peer dfa1152987644bd781b46c7e42a7a685 at {username='slave'} at 127.11.42.67:59949: session id = dfa1152987644bd781b46c7e42a7a685-d10d76ac7b094fe9b230b1afcbed9ca6
I20240502 01:54:06.442572 23499 consensus_queue.cc:260] T 29fb2b31cd9c4c8fb5923bb849a097d9 P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:06.444391 23499 ts_tablet_manager.cc:1398] T 29fb2b31cd9c4c8fb5923bb849a097d9 P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:54:06.445145 20518 tablet_copy_source_session.cc:205] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:06.446696 23508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d10d76ac7b094fe9b230b1afcbed9ca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:06.448508 23512 tablet_copy_client.cc:308] T 0d9f168de0d1410da2518abf936035ff P fdc2ac372492454c85387589a91f63af: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.67:36115
I20240502 01:54:06.449231 17757 tablet_copy_service.cc:140] P dfa1152987644bd781b46c7e42a7a685: Received BeginTabletCopySession request for tablet 0d9f168de0d1410da2518abf936035ff from peer fdc2ac372492454c85387589a91f63af ({username='slave'} at 127.11.42.69:44021)
I20240502 01:54:06.449393 17757 tablet_copy_service.cc:161] P dfa1152987644bd781b46c7e42a7a685: Beginning new tablet copy session on tablet 0d9f168de0d1410da2518abf936035ff from peer fdc2ac372492454c85387589a91f63af at {username='slave'} at 127.11.42.69:44021: session id = fdc2ac372492454c85387589a91f63af-0d9f168de0d1410da2518abf936035ff
I20240502 01:54:06.450621 23507 tablet_copy_client.cc:783] T 28463d45bfde44429d56179c1eec56b0 P 4da690c0fa13407a85c0689ce8de9ade: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:06.450821 23507 tablet_copy_client.cc:647] T 28463d45bfde44429d56179c1eec56b0 P 4da690c0fa13407a85c0689ce8de9ade: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:06.452894 17757 tablet_copy_source_session.cc:205] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:06.453143 23507 tablet_copy_client.cc:523] T 28463d45bfde44429d56179c1eec56b0 P 4da690c0fa13407a85c0689ce8de9ade: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:06.454319 23512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d9f168de0d1410da2518abf936035ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:06.457726 23507 tablet_bootstrap.cc:492] T 28463d45bfde44429d56179c1eec56b0 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:54:06.468892 23508 tablet_copy_client.cc:783] T d10d76ac7b094fe9b230b1afcbed9ca6 P dfa1152987644bd781b46c7e42a7a685: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:06.469338 23508 tablet_copy_client.cc:647] T d10d76ac7b094fe9b230b1afcbed9ca6 P dfa1152987644bd781b46c7e42a7a685: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:06.471715 23508 tablet_copy_client.cc:523] T d10d76ac7b094fe9b230b1afcbed9ca6 P dfa1152987644bd781b46c7e42a7a685: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:06.472201 23507 tablet_bootstrap.cc:492] T 28463d45bfde44429d56179c1eec56b0 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:06.474473 23507 tablet_bootstrap.cc:492] T 28463d45bfde44429d56179c1eec56b0 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap complete.
I20240502 01:54:06.474709 23507 ts_tablet_manager.cc:1367] T 28463d45bfde44429d56179c1eec56b0 P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.001s
I20240502 01:54:06.475492 23507 raft_consensus.cc:348] T 28463d45bfde44429d56179c1eec56b0 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } }
I20240502 01:54:06.475718 23507 raft_consensus.cc:729] T 28463d45bfde44429d56179c1eec56b0 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: LEARNER
I20240502 01:54:06.475896 23507 consensus_queue.cc:260] T 28463d45bfde44429d56179c1eec56b0 P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } }
I20240502 01:54:06.476871 23507 ts_tablet_manager.cc:1398] T 28463d45bfde44429d56179c1eec56b0 P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:06.477631 20387 tablet_copy_service.cc:342] P 6a36c952c51542c0bf7ea3ec2857a9f9: Request end of tablet copy session 4da690c0fa13407a85c0689ce8de9ade-28463d45bfde44429d56179c1eec56b0 received from {username='slave'} at 127.11.42.65:57879
I20240502 01:54:06.477757 20387 tablet_copy_service.cc:433] P 6a36c952c51542c0bf7ea3ec2857a9f9: ending tablet copy session 4da690c0fa13407a85c0689ce8de9ade-28463d45bfde44429d56179c1eec56b0 on tablet 28463d45bfde44429d56179c1eec56b0 with peer 4da690c0fa13407a85c0689ce8de9ade
I20240502 01:54:06.477922 23512 tablet_copy_client.cc:783] T 0d9f168de0d1410da2518abf936035ff P fdc2ac372492454c85387589a91f63af: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:06.478327 23512 tablet_copy_client.cc:647] T 0d9f168de0d1410da2518abf936035ff P fdc2ac372492454c85387589a91f63af: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:06.479189 23507 ts_tablet_manager.cc:902] T fac0ba26fabd4643bc38b7559faa471c P 4da690c0fa13407a85c0689ce8de9ade: Initiating tablet copy from peer 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:54:06.480749 23531 tablet_copy_client.cc:308] T 139304da92ab4f48b8a08fa24ed60c95 P bb31bceba76d466db24b9341c04900c3: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.71:38623
I20240502 01:54:06.481900 20518 tablet_copy_service.cc:140] P 2f306549407b4bb281ccf2994035e4f1: Received BeginTabletCopySession request for tablet 139304da92ab4f48b8a08fa24ed60c95 from peer bb31bceba76d466db24b9341c04900c3 ({username='slave'} at 127.11.42.68:49559)
I20240502 01:54:06.482056 20518 tablet_copy_service.cc:161] P 2f306549407b4bb281ccf2994035e4f1: Beginning new tablet copy session on tablet 139304da92ab4f48b8a08fa24ed60c95 from peer bb31bceba76d466db24b9341c04900c3 at {username='slave'} at 127.11.42.68:49559: session id = bb31bceba76d466db24b9341c04900c3-139304da92ab4f48b8a08fa24ed60c95
I20240502 01:54:06.484880 20518 tablet_copy_source_session.cc:205] T 139304da92ab4f48b8a08fa24ed60c95 P 2f306549407b4bb281ccf2994035e4f1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:06.486724 23531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 139304da92ab4f48b8a08fa24ed60c95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:06.490725 23512 tablet_copy_client.cc:523] T 0d9f168de0d1410da2518abf936035ff P fdc2ac372492454c85387589a91f63af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:06.495292 23508 tablet_bootstrap.cc:492] T d10d76ac7b094fe9b230b1afcbed9ca6 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:54:06.499467 23398 raft_consensus.cc:482] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:06.499611 23398 raft_consensus.cc:504] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:06.500455 23398 leader_election.cc:288] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:54:06.500820 23512 tablet_bootstrap.cc:492] T 0d9f168de0d1410da2518abf936035ff P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:54:06.500823 20629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29fb2b31cd9c4c8fb5923bb849a097d9" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091" is_pre_election: true
I20240502 01:54:06.501150 20629 raft_consensus.cc:2454] T 29fb2b31cd9c4c8fb5923bb849a097d9 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 0.
I20240502 01:54:06.501255 20496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29fb2b31cd9c4c8fb5923bb849a097d9" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
I20240502 01:54:06.501474 20496 raft_consensus.cc:2454] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 0.
I20240502 01:54:06.501605 17411 leader_election.cc:302] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:54:06.501881 23398 raft_consensus.cc:2781] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:06.501950 23398 raft_consensus.cc:482] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:06.502007 23398 raft_consensus.cc:3037] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:06.504184 23540 ts_tablet_manager.cc:902] T 3ff84ce60ac1438d92b2c944bb27aabb P 3fceccefba274b1dbbd8216af8b0bc29: Initiating tablet copy from peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:54:06.504849 23398 raft_consensus.cc:504] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:06.505684 22940 raft_consensus.cc:2932] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:06.506289 20630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29fb2b31cd9c4c8fb5923bb849a097d9" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091"
I20240502 01:54:06.506479 20630 raft_consensus.cc:3037] T 29fb2b31cd9c4c8fb5923bb849a097d9 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:06.509707 20630 raft_consensus.cc:2454] T 29fb2b31cd9c4c8fb5923bb849a097d9 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
I20240502 01:54:06.510032 17334 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:54:06.510138 17411 leader_election.cc:302] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:54:06.510159 17865 raft_consensus.cc:2932] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:06.510855 23551 raft_consensus.cc:2781] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:06.513229 23398 leader_election.cc:288] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Requested vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:54:06.516348 20494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29fb2b31cd9c4c8fb5923bb849a097d9" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
I20240502 01:54:06.516557 20494 raft_consensus.cc:3037] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:06.519456 20231 raft_consensus.cc:1261] T 44809ca119e145ddab852626a5af007f P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:06.519455 20494 raft_consensus.cc:2454] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
I20240502 01:54:06.519927 17734 raft_consensus.cc:1261] T 44809ca119e145ddab852626a5af007f P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:06.520932 23343 consensus_queue.cc:1035] T 44809ca119e145ddab852626a5af007f P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:06.521917 23343 consensus_queue.cc:1035] T 44809ca119e145ddab852626a5af007f P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:06.527839 23531 tablet_copy_client.cc:783] T 139304da92ab4f48b8a08fa24ed60c95 P bb31bceba76d466db24b9341c04900c3: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:06.528069 23531 tablet_copy_client.cc:647] T 139304da92ab4f48b8a08fa24ed60c95 P bb31bceba76d466db24b9341c04900c3: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:06.529718 23531 tablet_copy_client.cc:523] T 139304da92ab4f48b8a08fa24ed60c95 P bb31bceba76d466db24b9341c04900c3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:06.533120 23531 tablet_bootstrap.cc:492] T 139304da92ab4f48b8a08fa24ed60c95 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:54:06.532872 17334 catalog_manager.cc:5484] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P bb31bceba76d466db24b9341c04900c3 reported cstate change: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:06.538151 17328 catalog_manager.cc:5576] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115) does not have the latest schema for tablet 44809ca119e145ddab852626a5af007f (table rebalancer_dynamic_table [id=a7102d81aee247d087c72c0dd2bb3c44]). Expected version 1 got 0
I20240502 01:54:06.546185 23559 ts_tablet_manager.cc:902] T a212c17ba64842c99b3d6a78a3b7cc8a P dfa1152987644bd781b46c7e42a7a685: Initiating tablet copy from peer 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:54:06.564492 22991 raft_consensus.cc:482] T ee28f4aa31504dbfac977c4178aefdb1 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:06.564673 22991 raft_consensus.cc:504] T ee28f4aa31504dbfac977c4178aefdb1 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:06.565330 22991 leader_election.cc:288] T ee28f4aa31504dbfac977c4178aefdb1 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:54:06.565892 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee28f4aa31504dbfac977c4178aefdb1" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:54:06.566128 17735 raft_consensus.cc:2454] T ee28f4aa31504dbfac977c4178aefdb1 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 0.
I20240502 01:54:06.566599 20564 leader_election.cc:302] T ee28f4aa31504dbfac977c4178aefdb1 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dfa1152987644bd781b46c7e42a7a685, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:54:06.567023 23180 raft_consensus.cc:2781] T ee28f4aa31504dbfac977c4178aefdb1 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:06.567127 23180 raft_consensus.cc:482] T ee28f4aa31504dbfac977c4178aefdb1 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:06.567198 23180 raft_consensus.cc:3037] T ee28f4aa31504dbfac977c4178aefdb1 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:06.569751 23180 raft_consensus.cc:504] T ee28f4aa31504dbfac977c4178aefdb1 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:06.570408 23180 leader_election.cc:288] T ee28f4aa31504dbfac977c4178aefdb1 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 election: Requested vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:54:06.571926 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee28f4aa31504dbfac977c4178aefdb1" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:54:06.572142 17735 raft_consensus.cc:3037] T ee28f4aa31504dbfac977c4178aefdb1 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:06.574709 17735 raft_consensus.cc:2454] T ee28f4aa31504dbfac977c4178aefdb1 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 1.
I20240502 01:54:06.575299 20564 leader_election.cc:302] T ee28f4aa31504dbfac977c4178aefdb1 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dfa1152987644bd781b46c7e42a7a685, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:54:06.577772 20496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee28f4aa31504dbfac977c4178aefdb1" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
I20240502 01:54:06.577994 20496 raft_consensus.cc:3037] T ee28f4aa31504dbfac977c4178aefdb1 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:06.580513 20496 raft_consensus.cc:2454] T ee28f4aa31504dbfac977c4178aefdb1 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 1.
I20240502 01:54:06.581538 22991 raft_consensus.cc:2781] T ee28f4aa31504dbfac977c4178aefdb1 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:06.581670 22991 raft_consensus.cc:686] T ee28f4aa31504dbfac977c4178aefdb1 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Becoming Leader. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Running, Role: LEADER
I20240502 01:54:06.581936 22991 consensus_queue.cc:237] T ee28f4aa31504dbfac977c4178aefdb1 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:06.586010 20494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee28f4aa31504dbfac977c4178aefdb1" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
I20240502 01:54:06.586239 20494 raft_consensus.cc:2362] T ee28f4aa31504dbfac977c4178aefdb1 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 1. Re-sending same reply.
I20240502 01:54:06.610181 23561 tablet.cc:1722] T 44809ca119e145ddab852626a5af007f P 2f306549407b4bb281ccf2994035e4f1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:06.611155 23561 tablet.cc:1620] T 44809ca119e145ddab852626a5af007f P 2f306549407b4bb281ccf2994035e4f1: MemRowSet was empty: no flush needed.
I20240502 01:54:06.613886 23562 tablet.cc:1722] T 44809ca119e145ddab852626a5af007f P dfa1152987644bd781b46c7e42a7a685: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:06.614579 23562 tablet.cc:1620] T 44809ca119e145ddab852626a5af007f P dfa1152987644bd781b46c7e42a7a685: MemRowSet was empty: no flush needed.
I20240502 01:54:06.619690 17327 catalog_manager.cc:5576] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115) does not have the latest schema for tablet 44809ca119e145ddab852626a5af007f (table rebalancer_dynamic_table [id=a7102d81aee247d087c72c0dd2bb3c44]). Expected version 1 got 0
I20240502 01:54:06.628561 17334 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=a7102d81aee247d087c72c0dd2bb3c44] alter complete (version 1)
W20240502 01:54:06.635483 20564 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.42.70:39155, user_credentials={real_user=slave}} blocked reactor thread for 56893.9us
I20240502 01:54:06.640697 20366 raft_consensus.cc:2932] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
W20240502 01:54:06.643081 17411 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.42.72:42411, user_credentials={real_user=slave}} blocked reactor thread for 132975us
W20240502 01:54:06.651252 20565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.42.69:34339, user_credentials={real_user=slave}} blocked reactor thread for 72483.2us
I20240502 01:54:06.653286 17327 catalog_manager.cc:5576] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339) does not have the latest schema for tablet 44809ca119e145ddab852626a5af007f (table rebalancer_dynamic_table [id=a7102d81aee247d087c72c0dd2bb3c44]). Expected version 1 got 0
I20240502 01:54:06.660153 23559 tablet_copy_client.cc:308] T a212c17ba64842c99b3d6a78a3b7cc8a P dfa1152987644bd781b46c7e42a7a685: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.70:39155
I20240502 01:54:06.661468 20387 tablet_copy_service.cc:140] P 6a36c952c51542c0bf7ea3ec2857a9f9: Received BeginTabletCopySession request for tablet a212c17ba64842c99b3d6a78a3b7cc8a from peer dfa1152987644bd781b46c7e42a7a685 ({username='slave'} at 127.11.42.67:35525)
I20240502 01:54:06.661617 20387 tablet_copy_service.cc:161] P 6a36c952c51542c0bf7ea3ec2857a9f9: Beginning new tablet copy session on tablet a212c17ba64842c99b3d6a78a3b7cc8a from peer dfa1152987644bd781b46c7e42a7a685 at {username='slave'} at 127.11.42.67:35525: session id = dfa1152987644bd781b46c7e42a7a685-a212c17ba64842c99b3d6a78a3b7cc8a
I20240502 01:54:06.664546 20387 tablet_copy_source_session.cc:205] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:06.665903 23559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a212c17ba64842c99b3d6a78a3b7cc8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:06.667006 23531 tablet_bootstrap.cc:492] T 139304da92ab4f48b8a08fa24ed60c95 P bb31bceba76d466db24b9341c04900c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:06.667466 23531 tablet_bootstrap.cc:492] T 139304da92ab4f48b8a08fa24ed60c95 P bb31bceba76d466db24b9341c04900c3: Bootstrap complete.
I20240502 01:54:06.667726 23531 ts_tablet_manager.cc:1367] T 139304da92ab4f48b8a08fa24ed60c95 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.135s user 0.018s sys 0.014s
I20240502 01:54:06.668218 23577 tablet.cc:1722] T 44809ca119e145ddab852626a5af007f P fdc2ac372492454c85387589a91f63af: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:06.668429 23531 raft_consensus.cc:348] T 139304da92ab4f48b8a08fa24ed60c95 P bb31bceba76d466db24b9341c04900c3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } }
I20240502 01:54:06.668668 23531 raft_consensus.cc:729] T 139304da92ab4f48b8a08fa24ed60c95 P bb31bceba76d466db24b9341c04900c3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: LEARNER
I20240502 01:54:06.668856 23531 consensus_queue.cc:260] T 139304da92ab4f48b8a08fa24ed60c95 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } }
I20240502 01:54:06.669947 23577 tablet.cc:1620] T 44809ca119e145ddab852626a5af007f P fdc2ac372492454c85387589a91f63af: MemRowSet was empty: no flush needed.
I20240502 01:54:06.670159 17327 catalog_manager.cc:5576] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339) does not have the latest schema for tablet 44809ca119e145ddab852626a5af007f (table rebalancer_dynamic_table [id=a7102d81aee247d087c72c0dd2bb3c44]). Expected version 1 got 0
I20240502 01:54:06.673856 23551 raft_consensus.cc:686] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Becoming Leader. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Running, Role: LEADER
I20240502 01:54:06.674229 23551 consensus_queue.cc:237] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:06.677172 17334 catalog_manager.cc:5484] T ee28f4aa31504dbfac977c4178aefdb1 P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: term changed from 0 to 1, leader changed from <none> to f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72). New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:06.681077 23531 ts_tablet_manager.cc:1398] T 139304da92ab4f48b8a08fa24ed60c95 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.013s user 0.002s sys 0.003s
I20240502 01:54:06.682317 23507 tablet_copy_client.cc:308] T fac0ba26fabd4643bc38b7559faa471c P 4da690c0fa13407a85c0689ce8de9ade: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.71:38623
I20240502 01:54:06.682994 20518 tablet_copy_service.cc:140] P 2f306549407b4bb281ccf2994035e4f1: Received BeginTabletCopySession request for tablet fac0ba26fabd4643bc38b7559faa471c from peer 4da690c0fa13407a85c0689ce8de9ade ({username='slave'} at 127.11.42.65:53097)
I20240502 01:54:06.683193 20518 tablet_copy_service.cc:161] P 2f306549407b4bb281ccf2994035e4f1: Beginning new tablet copy session on tablet fac0ba26fabd4643bc38b7559faa471c from peer 4da690c0fa13407a85c0689ce8de9ade at {username='slave'} at 127.11.42.65:53097: session id = 4da690c0fa13407a85c0689ce8de9ade-fac0ba26fabd4643bc38b7559faa471c
I20240502 01:54:06.685969 20518 tablet_copy_source_session.cc:205] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:06.687144 23507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fac0ba26fabd4643bc38b7559faa471c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:06.689033 17327 catalog_manager.cc:5484] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: term changed from 0 to 1, leader changed from <none> to 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65). New cstate: current_term: 1 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:06.689548 23559 tablet_copy_client.cc:783] T a212c17ba64842c99b3d6a78a3b7cc8a P dfa1152987644bd781b46c7e42a7a685: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:06.689965 23559 tablet_copy_client.cc:647] T a212c17ba64842c99b3d6a78a3b7cc8a P dfa1152987644bd781b46c7e42a7a685: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:06.690382 20518 tablet_copy_service.cc:342] P 2f306549407b4bb281ccf2994035e4f1: Request end of tablet copy session bb31bceba76d466db24b9341c04900c3-139304da92ab4f48b8a08fa24ed60c95 received from {username='slave'} at 127.11.42.68:49559
I20240502 01:54:06.690483 20518 tablet_copy_service.cc:433] P 2f306549407b4bb281ccf2994035e4f1: ending tablet copy session bb31bceba76d466db24b9341c04900c3-139304da92ab4f48b8a08fa24ed60c95 on tablet 139304da92ab4f48b8a08fa24ed60c95 with peer bb31bceba76d466db24b9341c04900c3
I20240502 01:54:06.701215 23507 tablet_copy_client.cc:783] T fac0ba26fabd4643bc38b7559faa471c P 4da690c0fa13407a85c0689ce8de9ade: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:06.701423 23507 tablet_copy_client.cc:647] T fac0ba26fabd4643bc38b7559faa471c P 4da690c0fa13407a85c0689ce8de9ade: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:06.707564 23508 tablet_bootstrap.cc:492] T d10d76ac7b094fe9b230b1afcbed9ca6 P dfa1152987644bd781b46c7e42a7a685: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:06.707974 23508 tablet_bootstrap.cc:492] T d10d76ac7b094fe9b230b1afcbed9ca6 P dfa1152987644bd781b46c7e42a7a685: Bootstrap complete.
I20240502 01:54:06.708220 23508 ts_tablet_manager.cc:1367] T d10d76ac7b094fe9b230b1afcbed9ca6 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.213s user 0.009s sys 0.012s
I20240502 01:54:06.708917 23508 raft_consensus.cc:348] T d10d76ac7b094fe9b230b1afcbed9ca6 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:54:06.709254 23508 raft_consensus.cc:729] T d10d76ac7b094fe9b230b1afcbed9ca6 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: LEARNER
I20240502 01:54:06.709496 23508 consensus_queue.cc:260] T d10d76ac7b094fe9b230b1afcbed9ca6 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:54:06.714460 23508 ts_tablet_manager.cc:1398] T d10d76ac7b094fe9b230b1afcbed9ca6 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240502 01:54:06.717514 17737 raft_consensus.cc:1203] T d10d76ac7b094fe9b230b1afcbed9ca6 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:06.718283 20518 tablet_copy_service.cc:342] P 2f306549407b4bb281ccf2994035e4f1: Request end of tablet copy session dfa1152987644bd781b46c7e42a7a685-d10d76ac7b094fe9b230b1afcbed9ca6 received from {username='slave'} at 127.11.42.67:59949
I20240502 01:54:06.718400 20518 tablet_copy_service.cc:433] P 2f306549407b4bb281ccf2994035e4f1: ending tablet copy session dfa1152987644bd781b46c7e42a7a685-d10d76ac7b094fe9b230b1afcbed9ca6 on tablet d10d76ac7b094fe9b230b1afcbed9ca6 with peer dfa1152987644bd781b46c7e42a7a685
I20240502 01:54:06.724694 23559 tablet_copy_client.cc:523] T a212c17ba64842c99b3d6a78a3b7cc8a P dfa1152987644bd781b46c7e42a7a685: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:06.729218 23540 tablet_copy_client.cc:308] T 3ff84ce60ac1438d92b2c944bb27aabb P 3fceccefba274b1dbbd8216af8b0bc29: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.72:42411
I20240502 01:54:06.730005 20650 tablet_copy_service.cc:140] P f19cc6cb1e76460aa6711e49a5775091: Received BeginTabletCopySession request for tablet 3ff84ce60ac1438d92b2c944bb27aabb from peer 3fceccefba274b1dbbd8216af8b0bc29 ({username='slave'} at 127.11.42.66:54577)
I20240502 01:54:06.730162 20650 tablet_copy_service.cc:161] P f19cc6cb1e76460aa6711e49a5775091: Beginning new tablet copy session on tablet 3ff84ce60ac1438d92b2c944bb27aabb from peer 3fceccefba274b1dbbd8216af8b0bc29 at {username='slave'} at 127.11.42.66:54577: session id = 3fceccefba274b1dbbd8216af8b0bc29-3ff84ce60ac1438d92b2c944bb27aabb
I20240502 01:54:06.733839 23559 tablet_bootstrap.cc:492] T a212c17ba64842c99b3d6a78a3b7cc8a P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:54:06.732970 20650 tablet_copy_source_session.cc:205] T 3ff84ce60ac1438d92b2c944bb27aabb P f19cc6cb1e76460aa6711e49a5775091: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:06.734769 23507 tablet_copy_client.cc:523] T fac0ba26fabd4643bc38b7559faa471c P 4da690c0fa13407a85c0689ce8de9ade: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:06.736729 17327 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:54:06.738627 23507 tablet_bootstrap.cc:492] T fac0ba26fabd4643bc38b7559faa471c P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:54:06.749503 23532 ts_tablet_manager.cc:902] T bffb870715a243d992107583d7c17f1b P 4da690c0fa13407a85c0689ce8de9ade: Initiating tablet copy from peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:54:06.749580 23561 tablet.cc:1722] T 44809ca119e145ddab852626a5af007f P 2f306549407b4bb281ccf2994035e4f1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:06.749898 23540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ff84ce60ac1438d92b2c944bb27aabb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:06.750092 23561 tablet.cc:1620] T 44809ca119e145ddab852626a5af007f P 2f306549407b4bb281ccf2994035e4f1: MemRowSet was empty: no flush needed.
I20240502 01:54:06.750491 23521 tablet_copy_client.cc:308] T 81f335c023444796ab933e19d809cc03 P 3fceccefba274b1dbbd8216af8b0bc29: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.72:42411
I20240502 01:54:06.752099 23562 tablet.cc:1722] T 44809ca119e145ddab852626a5af007f P dfa1152987644bd781b46c7e42a7a685: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:06.752115 23577 tablet.cc:1722] T 44809ca119e145ddab852626a5af007f P fdc2ac372492454c85387589a91f63af: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:06.752570 23562 tablet.cc:1620] T 44809ca119e145ddab852626a5af007f P dfa1152987644bd781b46c7e42a7a685: MemRowSet was empty: no flush needed.
I20240502 01:54:06.752596 23577 tablet.cc:1620] T 44809ca119e145ddab852626a5af007f P fdc2ac372492454c85387589a91f63af: MemRowSet was empty: no flush needed.
I20240502 01:54:06.757767 20650 tablet_copy_service.cc:140] P f19cc6cb1e76460aa6711e49a5775091: Received BeginTabletCopySession request for tablet 81f335c023444796ab933e19d809cc03 from peer 3fceccefba274b1dbbd8216af8b0bc29 ({username='slave'} at 127.11.42.66:54577)
I20240502 01:54:06.757958 20650 tablet_copy_service.cc:161] P f19cc6cb1e76460aa6711e49a5775091: Beginning new tablet copy session on tablet 81f335c023444796ab933e19d809cc03 from peer 3fceccefba274b1dbbd8216af8b0bc29 at {username='slave'} at 127.11.42.66:54577: session id = 3fceccefba274b1dbbd8216af8b0bc29-81f335c023444796ab933e19d809cc03
I20240502 01:54:06.759673 23559 tablet_bootstrap.cc:492] T a212c17ba64842c99b3d6a78a3b7cc8a P dfa1152987644bd781b46c7e42a7a685: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:06.760059 23559 tablet_bootstrap.cc:492] T a212c17ba64842c99b3d6a78a3b7cc8a P dfa1152987644bd781b46c7e42a7a685: Bootstrap complete.
I20240502 01:54:06.760309 23559 ts_tablet_manager.cc:1367] T a212c17ba64842c99b3d6a78a3b7cc8a P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.009s
I20240502 01:54:06.760723 20650 tablet_copy_source_session.cc:205] T 81f335c023444796ab933e19d809cc03 P f19cc6cb1e76460aa6711e49a5775091: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:06.760978 23559 raft_consensus.cc:348] T a212c17ba64842c99b3d6a78a3b7cc8a P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:54:06.761248 23559 raft_consensus.cc:729] T a212c17ba64842c99b3d6a78a3b7cc8a P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: LEARNER
I20240502 01:54:06.761488 23559 consensus_queue.cc:260] T a212c17ba64842c99b3d6a78a3b7cc8a P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:54:06.762130 23559 ts_tablet_manager.cc:1398] T a212c17ba64842c99b3d6a78a3b7cc8a P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:06.762838 20387 tablet_copy_service.cc:342] P 6a36c952c51542c0bf7ea3ec2857a9f9: Request end of tablet copy session dfa1152987644bd781b46c7e42a7a685-a212c17ba64842c99b3d6a78a3b7cc8a received from {username='slave'} at 127.11.42.67:35525
I20240502 01:54:06.762979 20387 tablet_copy_service.cc:433] P 6a36c952c51542c0bf7ea3ec2857a9f9: ending tablet copy session dfa1152987644bd781b46c7e42a7a685-a212c17ba64842c99b3d6a78a3b7cc8a on tablet a212c17ba64842c99b3d6a78a3b7cc8a with peer dfa1152987644bd781b46c7e42a7a685
I20240502 01:54:06.763687 17328 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=a7102d81aee247d087c72c0dd2bb3c44] alter complete (version 2)
I20240502 01:54:06.764763 23521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81f335c023444796ab933e19d809cc03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:06.771201 23507 tablet_bootstrap.cc:492] T fac0ba26fabd4643bc38b7559faa471c P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:06.776636 23507 tablet_bootstrap.cc:492] T fac0ba26fabd4643bc38b7559faa471c P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap complete.
I20240502 01:54:06.776883 23507 ts_tablet_manager.cc:1367] T fac0ba26fabd4643bc38b7559faa471c P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.038s user 0.006s sys 0.000s
I20240502 01:54:06.777504 23512 tablet_bootstrap.cc:492] T 0d9f168de0d1410da2518abf936035ff P fdc2ac372492454c85387589a91f63af: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:06.777662 23507 raft_consensus.cc:348] T fac0ba26fabd4643bc38b7559faa471c P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } }
I20240502 01:54:06.777909 23512 tablet_bootstrap.cc:492] T 0d9f168de0d1410da2518abf936035ff P fdc2ac372492454c85387589a91f63af: Bootstrap complete.
I20240502 01:54:06.777899 23507 raft_consensus.cc:729] T fac0ba26fabd4643bc38b7559faa471c P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: LEARNER
I20240502 01:54:06.778158 23512 ts_tablet_manager.cc:1367] T 0d9f168de0d1410da2518abf936035ff P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.277s user 0.040s sys 0.009s
I20240502 01:54:06.778138 23507 consensus_queue.cc:260] T fac0ba26fabd4643bc38b7559faa471c P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } }
I20240502 01:54:06.778821 23507 ts_tablet_manager.cc:1398] T fac0ba26fabd4643bc38b7559faa471c P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:06.778864 23512 raft_consensus.cc:348] T 0d9f168de0d1410da2518abf936035ff P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } }
I20240502 01:54:06.778843 20496 consensus_queue.cc:237] T 246cec994d274786aad7672f2a30e5d8 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } }
I20240502 01:54:06.779135 23512 raft_consensus.cc:729] T 0d9f168de0d1410da2518abf936035ff P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: LEARNER
I20240502 01:54:06.779335 23512 consensus_queue.cc:260] T 0d9f168de0d1410da2518abf936035ff P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } }
I20240502 01:54:06.779958 23512 ts_tablet_manager.cc:1398] T 0d9f168de0d1410da2518abf936035ff P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240502 01:54:06.781039 17757 tablet_copy_service.cc:342] P dfa1152987644bd781b46c7e42a7a685: Request end of tablet copy session fdc2ac372492454c85387589a91f63af-0d9f168de0d1410da2518abf936035ff received from {username='slave'} at 127.11.42.69:44021
I20240502 01:54:06.781178 17757 tablet_copy_service.cc:433] P dfa1152987644bd781b46c7e42a7a685: ending tablet copy session fdc2ac372492454c85387589a91f63af-0d9f168de0d1410da2518abf936035ff on tablet 0d9f168de0d1410da2518abf936035ff with peer fdc2ac372492454c85387589a91f63af
I20240502 01:54:06.787017 23521 tablet_copy_client.cc:783] T 81f335c023444796ab933e19d809cc03 P 3fceccefba274b1dbbd8216af8b0bc29: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:06.787569 20518 tablet_copy_service.cc:342] P 2f306549407b4bb281ccf2994035e4f1: Request end of tablet copy session 4da690c0fa13407a85c0689ce8de9ade-fac0ba26fabd4643bc38b7559faa471c received from {username='slave'} at 127.11.42.65:53097
I20240502 01:54:06.787707 20518 tablet_copy_service.cc:433] P 2f306549407b4bb281ccf2994035e4f1: ending tablet copy session 4da690c0fa13407a85c0689ce8de9ade-fac0ba26fabd4643bc38b7559faa471c on tablet fac0ba26fabd4643bc38b7559faa471c with peer 4da690c0fa13407a85c0689ce8de9ade
W20240502 01:54:06.789410 20434 consensus_peers.cc:485] T 246cec994d274786aad7672f2a30e5d8 P 2f306549407b4bb281ccf2994035e4f1 -> Peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): Couldn't send request to peer bb31bceba76d466db24b9341c04900c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 246cec994d274786aad7672f2a30e5d8. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:06.790416 20230 raft_consensus.cc:1261] T 246cec994d274786aad7672f2a30e5d8 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:06.791054 23343 consensus_queue.cc:1035] T 246cec994d274786aad7672f2a30e5d8 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:06.793043 17866 raft_consensus.cc:1203] T 139304da92ab4f48b8a08fa24ed60c95 P bb31bceba76d466db24b9341c04900c3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:06.795965 23521 tablet_copy_client.cc:647] T 81f335c023444796ab933e19d809cc03 P 3fceccefba274b1dbbd8216af8b0bc29: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:06.796234 23396 consensus_queue.cc:1035] T ad067712087e4f78885bd8e1cf102694 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:06.799099 23521 tablet_copy_client.cc:523] T 81f335c023444796ab933e19d809cc03 P 3fceccefba274b1dbbd8216af8b0bc29: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:06.804687 23521 tablet_bootstrap.cc:492] T 81f335c023444796ab933e19d809cc03 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:54:06.813656 17737 raft_consensus.cc:1261] T 246cec994d274786aad7672f2a30e5d8 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:06.815158 23347 consensus_queue.cc:1035] T 246cec994d274786aad7672f2a30e5d8 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:06.818910 22568 consensus_queue.cc:1035] T ad067712087e4f78885bd8e1cf102694 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:06.820276 23347 raft_consensus.cc:2932] T 246cec994d274786aad7672f2a30e5d8 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } } }
I20240502 01:54:06.825742 20233 raft_consensus.cc:2932] T 246cec994d274786aad7672f2a30e5d8 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } } }
I20240502 01:54:06.833067 17734 raft_consensus.cc:2932] T 246cec994d274786aad7672f2a30e5d8 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } } }
I20240502 01:54:06.835453 17330 catalog_manager.cc:5484] T 246cec994d274786aad7672f2a30e5d8 P fdc2ac372492454c85387589a91f63af reported cstate change: config changed from index -1 to 2, NON_VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) added. New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } } }
I20240502 01:54:06.849610 23540 tablet_copy_client.cc:783] T 3ff84ce60ac1438d92b2c944bb27aabb P 3fceccefba274b1dbbd8216af8b0bc29: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:06.858283 23532 tablet_copy_client.cc:308] T bffb870715a243d992107583d7c17f1b P 4da690c0fa13407a85c0689ce8de9ade: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.69:34339
I20240502 01:54:06.859154 20253 tablet_copy_service.cc:140] P fdc2ac372492454c85387589a91f63af: Received BeginTabletCopySession request for tablet bffb870715a243d992107583d7c17f1b from peer 4da690c0fa13407a85c0689ce8de9ade ({username='slave'} at 127.11.42.65:54337)
I20240502 01:54:06.859330 20253 tablet_copy_service.cc:161] P fdc2ac372492454c85387589a91f63af: Beginning new tablet copy session on tablet bffb870715a243d992107583d7c17f1b from peer 4da690c0fa13407a85c0689ce8de9ade at {username='slave'} at 127.11.42.65:54337: session id = 4da690c0fa13407a85c0689ce8de9ade-bffb870715a243d992107583d7c17f1b
I20240502 01:54:06.859448 23540 tablet_copy_client.cc:647] T 3ff84ce60ac1438d92b2c944bb27aabb P 3fceccefba274b1dbbd8216af8b0bc29: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:06.859222 23521 tablet_bootstrap.cc:492] T 81f335c023444796ab933e19d809cc03 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:06.859908 23521 tablet_bootstrap.cc:492] T 81f335c023444796ab933e19d809cc03 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap complete.
I20240502 01:54:06.860159 23521 ts_tablet_manager.cc:1367] T 81f335c023444796ab933e19d809cc03 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.056s user 0.006s sys 0.001s
I20240502 01:54:06.860985 23540 tablet_copy_client.cc:523] T 3ff84ce60ac1438d92b2c944bb27aabb P 3fceccefba274b1dbbd8216af8b0bc29: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:06.860870 23521 raft_consensus.cc:348] T 81f335c023444796ab933e19d809cc03 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } }
I20240502 01:54:06.861152 23521 raft_consensus.cc:729] T 81f335c023444796ab933e19d809cc03 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: LEARNER
I20240502 01:54:06.861366 23521 consensus_queue.cc:260] T 81f335c023444796ab933e19d809cc03 P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } }
I20240502 01:54:06.862920 20253 tablet_copy_source_session.cc:205] T bffb870715a243d992107583d7c17f1b P fdc2ac372492454c85387589a91f63af: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:06.863344 23521 ts_tablet_manager.cc:1398] T 81f335c023444796ab933e19d809cc03 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:06.864140 20650 tablet_copy_service.cc:342] P f19cc6cb1e76460aa6711e49a5775091: Request end of tablet copy session 3fceccefba274b1dbbd8216af8b0bc29-81f335c023444796ab933e19d809cc03 received from {username='slave'} at 127.11.42.66:54577
I20240502 01:54:06.864250 20650 tablet_copy_service.cc:433] P f19cc6cb1e76460aa6711e49a5775091: ending tablet copy session 3fceccefba274b1dbbd8216af8b0bc29-81f335c023444796ab933e19d809cc03 on tablet 81f335c023444796ab933e19d809cc03 with peer 3fceccefba274b1dbbd8216af8b0bc29
I20240502 01:54:06.864945 23532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bffb870715a243d992107583d7c17f1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:06.871160 20233 raft_consensus.cc:1203] T 0d9f168de0d1410da2518abf936035ff P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 01:54:06.873584 17327 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:54:06.878477 17330 catalog_manager.cc:5576] TS 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623) does not have the latest schema for tablet 44809ca119e145ddab852626a5af007f (table rebalancer_dynamic_table_new_name [id=a7102d81aee247d087c72c0dd2bb3c44]). Expected version 3 got 2
I20240502 01:54:06.889329 17474 raft_consensus.cc:1203] T 28463d45bfde44429d56179c1eec56b0 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:06.891319 23561 tablet.cc:1722] T 44809ca119e145ddab852626a5af007f P 2f306549407b4bb281ccf2994035e4f1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:06.891588 23577 tablet.cc:1722] T 44809ca119e145ddab852626a5af007f P fdc2ac372492454c85387589a91f63af: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:06.902729 17327 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table_new_name [id=a7102d81aee247d087c72c0dd2bb3c44] alter complete (version 3)
I20240502 01:54:06.906633 23532 tablet_copy_client.cc:783] T bffb870715a243d992107583d7c17f1b P 4da690c0fa13407a85c0689ce8de9ade: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:06.907552 20366 consensus_queue.cc:237] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:54:06.907909 23562 tablet.cc:1722] T 44809ca119e145ddab852626a5af007f P dfa1152987644bd781b46c7e42a7a685: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:06.908619 23540 tablet_bootstrap.cc:492] T 3ff84ce60ac1438d92b2c944bb27aabb P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:54:06.906862 23532 tablet_copy_client.cc:647] T bffb870715a243d992107583d7c17f1b P 4da690c0fa13407a85c0689ce8de9ade: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:06.916839 23532 tablet_copy_client.cc:523] T bffb870715a243d992107583d7c17f1b P 4da690c0fa13407a85c0689ce8de9ade: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:06.923707 20496 raft_consensus.cc:1261] T 18549019e31b47d899d011cef0637221 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:06.924531 17605 raft_consensus.cc:1261] T 18549019e31b47d899d011cef0637221 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:06.924885 23410 consensus_queue.cc:1035] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:54:06.925266 20302 consensus_peers.cc:485] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 18549019e31b47d899d011cef0637221. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:06.926550 23410 consensus_queue.cc:1035] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:06.928742 23410 raft_consensus.cc:2932] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:54:06.929188 20496 raft_consensus.cc:2932] T 18549019e31b47d899d011cef0637221 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:54:06.937644 17327 catalog_manager.cc:5484] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: config changed from index -1 to 2, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:06.939761 23532 tablet_bootstrap.cc:492] T bffb870715a243d992107583d7c17f1b P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:54:06.960251 17605 raft_consensus.cc:2932] T 18549019e31b47d899d011cef0637221 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:54:06.962605 23540 tablet_bootstrap.cc:492] T 3ff84ce60ac1438d92b2c944bb27aabb P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:06.962991 23540 tablet_bootstrap.cc:492] T 3ff84ce60ac1438d92b2c944bb27aabb P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap complete.
I20240502 01:54:06.963227 23540 ts_tablet_manager.cc:1367] T 3ff84ce60ac1438d92b2c944bb27aabb P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.055s user 0.003s sys 0.002s
I20240502 01:54:06.963927 23540 raft_consensus.cc:348] T 3ff84ce60ac1438d92b2c944bb27aabb P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } }
I20240502 01:54:06.964152 23540 raft_consensus.cc:729] T 3ff84ce60ac1438d92b2c944bb27aabb P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: LEARNER
I20240502 01:54:06.964380 23540 consensus_queue.cc:260] T 3ff84ce60ac1438d92b2c944bb27aabb P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } }
I20240502 01:54:06.966652 23540 ts_tablet_manager.cc:1398] T 3ff84ce60ac1438d92b2c944bb27aabb P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20240502 01:54:06.967422 20650 tablet_copy_service.cc:342] P f19cc6cb1e76460aa6711e49a5775091: Request end of tablet copy session 3fceccefba274b1dbbd8216af8b0bc29-3ff84ce60ac1438d92b2c944bb27aabb received from {username='slave'} at 127.11.42.66:54577
I20240502 01:54:06.967533 20650 tablet_copy_service.cc:433] P f19cc6cb1e76460aa6711e49a5775091: ending tablet copy session 3fceccefba274b1dbbd8216af8b0bc29-3ff84ce60ac1438d92b2c944bb27aabb on tablet 3ff84ce60ac1438d92b2c944bb27aabb with peer 3fceccefba274b1dbbd8216af8b0bc29
I20240502 01:54:06.970176 23569 consensus_queue.cc:1035] T ee28f4aa31504dbfac977c4178aefdb1 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:06.973942 17474 raft_consensus.cc:1203] T fac0ba26fabd4643bc38b7559faa471c P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:06.974673 17865 consensus_queue.cc:237] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } }
I20240502 01:54:06.979110 22991 consensus_queue.cc:1035] T ee28f4aa31504dbfac977c4178aefdb1 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:06.983295 23532 tablet_bootstrap.cc:492] T bffb870715a243d992107583d7c17f1b P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:06.986786 23532 tablet_bootstrap.cc:492] T bffb870715a243d992107583d7c17f1b P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap complete.
I20240502 01:54:06.988358 23532 ts_tablet_manager.cc:1367] T bffb870715a243d992107583d7c17f1b P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.049s user 0.006s sys 0.001s
I20240502 01:54:06.989145 23532 raft_consensus.cc:348] T bffb870715a243d992107583d7c17f1b P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } }
I20240502 01:54:06.992357 23532 raft_consensus.cc:729] T bffb870715a243d992107583d7c17f1b P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: LEARNER
I20240502 01:54:06.995340 23532 consensus_queue.cc:260] T bffb870715a243d992107583d7c17f1b P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } }
I20240502 01:54:07.001408 23532 ts_tablet_manager.cc:1398] T bffb870715a243d992107583d7c17f1b P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240502 01:54:07.006436 20253 tablet_copy_service.cc:342] P fdc2ac372492454c85387589a91f63af: Request end of tablet copy session 4da690c0fa13407a85c0689ce8de9ade-bffb870715a243d992107583d7c17f1b received from {username='slave'} at 127.11.42.65:54337
I20240502 01:54:07.006561 20253 tablet_copy_service.cc:433] P fdc2ac372492454c85387589a91f63af: ending tablet copy session 4da690c0fa13407a85c0689ce8de9ade-bffb870715a243d992107583d7c17f1b on tablet bffb870715a243d992107583d7c17f1b with peer 4da690c0fa13407a85c0689ce8de9ade
W20240502 01:54:07.010635 17802 consensus_peers.cc:485] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 -> Peer 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): Couldn't send request to peer 6a36c952c51542c0bf7ea3ec2857a9f9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 377f8b554ad1448d981d2283b865f8fe. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:07.011606 20496 raft_consensus.cc:1261] T 377f8b554ad1448d981d2283b865f8fe P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:07.012187 21667 consensus_queue.cc:1035] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:07.014918 17605 raft_consensus.cc:1261] T 377f8b554ad1448d981d2283b865f8fe P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:07.015874 23586 consensus_queue.cc:1035] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:07.017001 23586 raft_consensus.cc:2932] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } } }
I20240502 01:54:07.017709 17328 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:07.017915 17328 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:07.019905 17604 raft_consensus.cc:2932] T 377f8b554ad1448d981d2283b865f8fe P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } } }
I20240502 01:54:07.021284 17328 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet 44809ca119e145ddab852626a5af007f
I20240502 01:54:07.022091 17716 tablet_service.cc:1514] Processing DeleteTablet for tablet 44809ca119e145ddab852626a5af007f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:07 UTC) from {username='slave'} at 127.0.0.1:52198
I20240502 01:54:07.024431 20476 tablet_service.cc:1514] Processing DeleteTablet for tablet 44809ca119e145ddab852626a5af007f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:07 UTC) from {username='slave'} at 127.0.0.1:35018
I20240502 01:54:07.025115 23627 tablet_replica.cc:330] T 44809ca119e145ddab852626a5af007f P 2f306549407b4bb281ccf2994035e4f1: stopping tablet replica
I20240502 01:54:07.025538 20212 tablet_service.cc:1514] Processing DeleteTablet for tablet 44809ca119e145ddab852626a5af007f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:07 UTC) from {username='slave'} at 127.0.0.1:44624
I20240502 01:54:07.025707 20496 raft_consensus.cc:2932] T 377f8b554ad1448d981d2283b865f8fe P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } } }
I20240502 01:54:07.033927 17327 catalog_manager.cc:5484] T 377f8b554ad1448d981d2283b865f8fe P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: config changed from index 6 to 7, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } } }
I20240502 01:54:07.036583 23626 tablet_replica.cc:330] T 44809ca119e145ddab852626a5af007f P dfa1152987644bd781b46c7e42a7a685: stopping tablet replica
I20240502 01:54:07.036780 23626 raft_consensus.cc:2229] T 44809ca119e145ddab852626a5af007f P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:07.036949 23626 raft_consensus.cc:2258] T 44809ca119e145ddab852626a5af007f P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:07.039709 23559 ts_tablet_manager.cc:902] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P dfa1152987644bd781b46c7e42a7a685: Initiating tablet copy from peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:54:07.040798 23629 tablet_replica.cc:330] T 44809ca119e145ddab852626a5af007f P fdc2ac372492454c85387589a91f63af: stopping tablet replica
I20240502 01:54:07.040999 23629 raft_consensus.cc:2229] T 44809ca119e145ddab852626a5af007f P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:07.041252 23629 raft_consensus.cc:2258] T 44809ca119e145ddab852626a5af007f P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:07.041443 23626 ts_tablet_manager.cc:1875] T 44809ca119e145ddab852626a5af007f P dfa1152987644bd781b46c7e42a7a685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:07.043148 23629 ts_tablet_manager.cc:1875] T 44809ca119e145ddab852626a5af007f P fdc2ac372492454c85387589a91f63af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:07.025319 23627 raft_consensus.cc:2229] T 44809ca119e145ddab852626a5af007f P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:07.049460 23627 raft_consensus.cc:2258] T 44809ca119e145ddab852626a5af007f P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:07.050634 23627 ts_tablet_manager.cc:1875] T 44809ca119e145ddab852626a5af007f P 2f306549407b4bb281ccf2994035e4f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:07.059579 17604 raft_consensus.cc:1203] T 3ff84ce60ac1438d92b2c944bb27aabb P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:07.064285 23629 ts_tablet_manager.cc:1888] T 44809ca119e145ddab852626a5af007f P fdc2ac372492454c85387589a91f63af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:07.065932 23629 log.cc:1196] T 44809ca119e145ddab852626a5af007f P fdc2ac372492454c85387589a91f63af: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal/wals/44809ca119e145ddab852626a5af007f
I20240502 01:54:07.066607 23629 ts_tablet_manager.cc:1909] T 44809ca119e145ddab852626a5af007f P fdc2ac372492454c85387589a91f63af: Deleting consensus metadata
I20240502 01:54:07.068709 17322 catalog_manager.cc:4830] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): tablet 44809ca119e145ddab852626a5af007f (table rebalancer_dynamic_table_new_name [id=a7102d81aee247d087c72c0dd2bb3c44]) successfully deleted
I20240502 01:54:07.072532 23627 ts_tablet_manager.cc:1888] T 44809ca119e145ddab852626a5af007f P 2f306549407b4bb281ccf2994035e4f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:07.072762 23627 log.cc:1196] T 44809ca119e145ddab852626a5af007f P 2f306549407b4bb281ccf2994035e4f1: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal/wals/44809ca119e145ddab852626a5af007f
I20240502 01:54:07.073441 23627 ts_tablet_manager.cc:1909] T 44809ca119e145ddab852626a5af007f P 2f306549407b4bb281ccf2994035e4f1: Deleting consensus metadata
I20240502 01:54:07.074221 23626 ts_tablet_manager.cc:1888] T 44809ca119e145ddab852626a5af007f P dfa1152987644bd781b46c7e42a7a685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:07.074400 23626 log.cc:1196] T 44809ca119e145ddab852626a5af007f P dfa1152987644bd781b46c7e42a7a685: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/wals/44809ca119e145ddab852626a5af007f
I20240502 01:54:07.075687 23626 ts_tablet_manager.cc:1909] T 44809ca119e145ddab852626a5af007f P dfa1152987644bd781b46c7e42a7a685: Deleting consensus metadata
I20240502 01:54:07.076232 17322 catalog_manager.cc:4830] TS 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): tablet 44809ca119e145ddab852626a5af007f (table rebalancer_dynamic_table_new_name [id=a7102d81aee247d087c72c0dd2bb3c44]) successfully deleted
I20240502 01:54:07.080879 17321 catalog_manager.cc:4830] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): tablet 44809ca119e145ddab852626a5af007f (table rebalancer_dynamic_table_new_name [id=a7102d81aee247d087c72c0dd2bb3c44]) successfully deleted
I20240502 01:54:07.091606 17474 raft_consensus.cc:1203] T bffb870715a243d992107583d7c17f1b P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:07.102533 17734 raft_consensus.cc:1203] T a212c17ba64842c99b3d6a78a3b7cc8a P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 01:54:07.113320 23559 tablet_copy_client.cc:308] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P dfa1152987644bd781b46c7e42a7a685: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.72:42411
I20240502 01:54:07.114066 20650 tablet_copy_service.cc:140] P f19cc6cb1e76460aa6711e49a5775091: Received BeginTabletCopySession request for tablet bfd7b1d1002c4b57a14d43ed9c6d9f50 from peer dfa1152987644bd781b46c7e42a7a685 ({username='slave'} at 127.11.42.67:36277)
I20240502 01:54:07.114224 20650 tablet_copy_service.cc:161] P f19cc6cb1e76460aa6711e49a5775091: Beginning new tablet copy session on tablet bfd7b1d1002c4b57a14d43ed9c6d9f50 from peer dfa1152987644bd781b46c7e42a7a685 at {username='slave'} at 127.11.42.67:36277: session id = dfa1152987644bd781b46c7e42a7a685-bfd7b1d1002c4b57a14d43ed9c6d9f50
I20240502 01:54:07.116844 20650 tablet_copy_source_session.cc:205] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P f19cc6cb1e76460aa6711e49a5775091: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:07.118065 23559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfd7b1d1002c4b57a14d43ed9c6d9f50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:07.123530 23559 tablet_copy_client.cc:783] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P dfa1152987644bd781b46c7e42a7a685: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:07.123767 23559 tablet_copy_client.cc:647] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P dfa1152987644bd781b46c7e42a7a685: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:07.125313 23559 tablet_copy_client.cc:523] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P dfa1152987644bd781b46c7e42a7a685: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:07.128098 23559 tablet_bootstrap.cc:492] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:54:07.128522 17327 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:54:07.138545 20347 tablet_service.cc:1467] Processing CreateTablet for tablet 0f953dbfbe0b4e93880c5665003395e0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5b37c5f6d0cc4a34b264799964d1ffe9]), partition=
I20240502 01:54:07.138962 20347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f953dbfbe0b4e93880c5665003395e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:07.139128 17845 tablet_service.cc:1467] Processing CreateTablet for tablet 0f953dbfbe0b4e93880c5665003395e0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5b37c5f6d0cc4a34b264799964d1ffe9]), partition=
I20240502 01:54:07.139415 17845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f953dbfbe0b4e93880c5665003395e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:07.142287 20212 tablet_service.cc:1467] Processing CreateTablet for tablet 0f953dbfbe0b4e93880c5665003395e0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5b37c5f6d0cc4a34b264799964d1ffe9]), partition=
I20240502 01:54:07.142645 20212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f953dbfbe0b4e93880c5665003395e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:07.158665 20233 consensus_queue.cc:237] T 062b9d90fb0444f4be37bf09770c9caf P fdc2ac372492454c85387589a91f63af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } }
I20240502 01:54:07.165197 23551 consensus_queue.cc:1035] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:07.168074 23531 ts_tablet_manager.cc:902] T 246cec994d274786aad7672f2a30e5d8 P bb31bceba76d466db24b9341c04900c3: Initiating tablet copy from peer 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
W20240502 01:54:07.174358 20167 consensus_peers.cc:485] T 062b9d90fb0444f4be37bf09770c9caf P fdc2ac372492454c85387589a91f63af -> Peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Couldn't send request to peer 4da690c0fa13407a85c0689ce8de9ade. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 062b9d90fb0444f4be37bf09770c9caf. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:07.176223 17865 raft_consensus.cc:1261] T 062b9d90fb0444f4be37bf09770c9caf P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:07.176260 20494 raft_consensus.cc:1261] T 062b9d90fb0444f4be37bf09770c9caf P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:07.177577 23638 consensus_queue.cc:1035] T 062b9d90fb0444f4be37bf09770c9caf P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:07.178192 23638 consensus_queue.cc:1035] T 062b9d90fb0444f4be37bf09770c9caf P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:07.207017 23638 raft_consensus.cc:2932] T 062b9d90fb0444f4be37bf09770c9caf P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } } }
I20240502 01:54:07.219060 17328 catalog_manager.cc:5484] T 062b9d90fb0444f4be37bf09770c9caf P fdc2ac372492454c85387589a91f63af reported cstate change: config changed from index -1 to 2, NON_VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) added. New cstate: current_term: 1 leader_uuid: "fdc2ac372492454c85387589a91f63af" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:07.225742 23644 tablet_bootstrap.cc:492] T 0f953dbfbe0b4e93880c5665003395e0 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:54:07.217624 20496 raft_consensus.cc:2932] T 062b9d90fb0444f4be37bf09770c9caf P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } } }
I20240502 01:54:07.227649 23644 tablet_bootstrap.cc:654] T 0f953dbfbe0b4e93880c5665003395e0 P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:07.231061 23559 tablet_bootstrap.cc:492] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P dfa1152987644bd781b46c7e42a7a685: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:07.232017 23559 tablet_bootstrap.cc:492] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P dfa1152987644bd781b46c7e42a7a685: Bootstrap complete.
I20240502 01:54:07.232270 23559 ts_tablet_manager.cc:1367] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.104s user 0.006s sys 0.003s
I20240502 01:54:07.234540 23643 tablet_bootstrap.cc:492] T 0f953dbfbe0b4e93880c5665003395e0 P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:54:07.246214 17865 raft_consensus.cc:2932] T 062b9d90fb0444f4be37bf09770c9caf P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } } }
I20240502 01:54:07.252223 23343 raft_consensus.cc:1050] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1: attempting to promote NON_VOTER dfa1152987644bd781b46c7e42a7a685 to VOTER
I20240502 01:54:07.253278 23343 consensus_queue.cc:237] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:07.258656 23531 tablet_copy_client.cc:308] T 246cec994d274786aad7672f2a30e5d8 P bb31bceba76d466db24b9341c04900c3: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.71:38623
I20240502 01:54:07.261682 20518 tablet_copy_service.cc:140] P 2f306549407b4bb281ccf2994035e4f1: Received BeginTabletCopySession request for tablet 246cec994d274786aad7672f2a30e5d8 from peer bb31bceba76d466db24b9341c04900c3 ({username='slave'} at 127.11.42.68:49559)
I20240502 01:54:07.261862 20518 tablet_copy_service.cc:161] P 2f306549407b4bb281ccf2994035e4f1: Beginning new tablet copy session on tablet 246cec994d274786aad7672f2a30e5d8 from peer bb31bceba76d466db24b9341c04900c3 at {username='slave'} at 127.11.42.68:49559: session id = bb31bceba76d466db24b9341c04900c3-246cec994d274786aad7672f2a30e5d8
I20240502 01:54:07.264744 20518 tablet_copy_source_session.cc:205] T 246cec994d274786aad7672f2a30e5d8 P 2f306549407b4bb281ccf2994035e4f1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:07.265884 23643 tablet_bootstrap.cc:654] T 0f953dbfbe0b4e93880c5665003395e0 P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:07.268458 23531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 246cec994d274786aad7672f2a30e5d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:07.232981 23559 raft_consensus.cc:348] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:54:07.276672 23559 raft_consensus.cc:729] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: LEARNER
I20240502 01:54:07.276890 23559 consensus_queue.cc:260] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:54:07.277426 23644 tablet_bootstrap.cc:492] T 0f953dbfbe0b4e93880c5665003395e0 P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:54:07.277586 23644 ts_tablet_manager.cc:1367] T 0f953dbfbe0b4e93880c5665003395e0 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.052s user 0.003s sys 0.000s
I20240502 01:54:07.278393 23644 raft_consensus.cc:348] T 0f953dbfbe0b4e93880c5665003395e0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:07.278645 23644 raft_consensus.cc:374] T 0f953dbfbe0b4e93880c5665003395e0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:07.278743 23644 raft_consensus.cc:729] T 0f953dbfbe0b4e93880c5665003395e0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:54:07.278951 23559 ts_tablet_manager.cc:1398] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.047s user 0.002s sys 0.000s
I20240502 01:54:07.278951 23644 consensus_queue.cc:260] T 0f953dbfbe0b4e93880c5665003395e0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:07.279583 23644 ts_tablet_manager.cc:1398] T 0f953dbfbe0b4e93880c5665003395e0 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:07.280880 22876 raft_consensus.cc:1050] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9: attempting to promote NON_VOTER 4da690c0fa13407a85c0689ce8de9ade to VOTER
I20240502 01:54:07.281246 23642 tablet_bootstrap.cc:492] T 0f953dbfbe0b4e93880c5665003395e0 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:54:07.282189 22876 consensus_queue.cc:237] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } }
I20240502 01:54:07.283525 23642 tablet_bootstrap.cc:654] T 0f953dbfbe0b4e93880c5665003395e0 P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:07.286171 20650 tablet_copy_service.cc:342] P f19cc6cb1e76460aa6711e49a5775091: Request end of tablet copy session dfa1152987644bd781b46c7e42a7a685-bfd7b1d1002c4b57a14d43ed9c6d9f50 received from {username='slave'} at 127.11.42.67:36277
I20240502 01:54:07.286345 20650 tablet_copy_service.cc:433] P f19cc6cb1e76460aa6711e49a5775091: ending tablet copy session dfa1152987644bd781b46c7e42a7a685-bfd7b1d1002c4b57a14d43ed9c6d9f50 on tablet bfd7b1d1002c4b57a14d43ed9c6d9f50 with peer dfa1152987644bd781b46c7e42a7a685
I20240502 01:54:07.286993 17327 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_extra_table_38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:07.307394 17846 tablet_service.cc:1467] Processing CreateTablet for tablet d25f2ab1a7544cec893dd02c7c89285c (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=a08d0126730a4231bfed6fa214450fc4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:07.307809 17846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d25f2ab1a7544cec893dd02c7c89285c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:07.308018 17456 tablet_service.cc:1467] Processing CreateTablet for tablet d25f2ab1a7544cec893dd02c7c89285c (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=a08d0126730a4231bfed6fa214450fc4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:07.308394 17456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d25f2ab1a7544cec893dd02c7c89285c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:07.309485 17585 tablet_service.cc:1467] Processing CreateTablet for tablet 0399c9d1d7b2408292dc78c185605ee8 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=a08d0126730a4231bfed6fa214450fc4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:07.310005 17585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0399c9d1d7b2408292dc78c185605ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:07.313383 17587 tablet_service.cc:1467] Processing CreateTablet for tablet bbf05e3b3ad44354be7600c3c07756fd (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=a08d0126730a4231bfed6fa214450fc4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:07.313756 17587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbf05e3b3ad44354be7600c3c07756fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:07.313916 20212 tablet_service.cc:1467] Processing CreateTablet for tablet bbf05e3b3ad44354be7600c3c07756fd (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=a08d0126730a4231bfed6fa214450fc4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:07.314328 20212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbf05e3b3ad44354be7600c3c07756fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:07.315536 20345 tablet_service.cc:1467] Processing CreateTablet for tablet 0399c9d1d7b2408292dc78c185605ee8 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=a08d0126730a4231bfed6fa214450fc4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:07.315943 20345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0399c9d1d7b2408292dc78c185605ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:07.317174 20347 tablet_service.cc:1467] Processing CreateTablet for tablet bbf05e3b3ad44354be7600c3c07756fd (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=a08d0126730a4231bfed6fa214450fc4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:07.317544 20347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbf05e3b3ad44354be7600c3c07756fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:07.317447 20609 tablet_service.cc:1467] Processing CreateTablet for tablet d25f2ab1a7544cec893dd02c7c89285c (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=a08d0126730a4231bfed6fa214450fc4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:07.317836 20609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d25f2ab1a7544cec893dd02c7c89285c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:07.324544 17847 tablet_service.cc:1467] Processing CreateTablet for tablet 0399c9d1d7b2408292dc78c185605ee8 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=a08d0126730a4231bfed6fa214450fc4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:07.324904 17847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0399c9d1d7b2408292dc78c185605ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:07.329209 23644 tablet_bootstrap.cc:492] T 0399c9d1d7b2408292dc78c185605ee8 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:54:07.330785 23644 tablet_bootstrap.cc:654] T 0399c9d1d7b2408292dc78c185605ee8 P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:07.335397 22568 raft_consensus.cc:1050] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685: attempting to promote NON_VOTER fdc2ac372492454c85387589a91f63af to VOTER
I20240502 01:54:07.336237 22568 consensus_queue.cc:237] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } }
I20240502 01:54:07.340680 20233 raft_consensus.cc:1261] T d10d76ac7b094fe9b230b1afcbed9ca6 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:07.341248 23591 consensus_queue.cc:1035] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:07.350592 23673 tablet_bootstrap.cc:492] T 0399c9d1d7b2408292dc78c185605ee8 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:54:07.352715 20629 raft_consensus.cc:1261] T 0d9f168de0d1410da2518abf936035ff P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:07.355748 22568 consensus_queue.cc:1035] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:07.356408 23673 tablet_bootstrap.cc:654] T 0399c9d1d7b2408292dc78c185605ee8 P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:07.365346 23643 tablet_bootstrap.cc:492] T 0f953dbfbe0b4e93880c5665003395e0 P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:54:07.365566 23643 ts_tablet_manager.cc:1367] T 0f953dbfbe0b4e93880c5665003395e0 P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.131s user 0.002s sys 0.006s
I20240502 01:54:07.366446 23643 raft_consensus.cc:348] T 0f953dbfbe0b4e93880c5665003395e0 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:07.366724 23643 raft_consensus.cc:374] T 0f953dbfbe0b4e93880c5665003395e0 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:07.366871 23643 raft_consensus.cc:729] T 0f953dbfbe0b4e93880c5665003395e0 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:54:07.367163 23643 consensus_queue.cc:260] T 0f953dbfbe0b4e93880c5665003395e0 P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:07.367993 23643 ts_tablet_manager.cc:1398] T 0f953dbfbe0b4e93880c5665003395e0 P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:54:07.373656 23611 raft_consensus.cc:482] T 0f953dbfbe0b4e93880c5665003395e0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:07.373795 23611 raft_consensus.cc:504] T 0f953dbfbe0b4e93880c5665003395e0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:07.374004 23673 tablet_bootstrap.cc:492] T 0399c9d1d7b2408292dc78c185605ee8 P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:54:07.374132 23673 ts_tablet_manager.cc:1367] T 0399c9d1d7b2408292dc78c185605ee8 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.002s
I20240502 01:54:07.374768 23611 leader_election.cc:288] T 0f953dbfbe0b4e93880c5665003395e0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:54:07.375298 20366 raft_consensus.cc:1261] T d10d76ac7b094fe9b230b1afcbed9ca6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:07.376531 23671 tablet_bootstrap.cc:492] T d25f2ab1a7544cec893dd02c7c89285c P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:54:07.377130 17865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f953dbfbe0b4e93880c5665003395e0" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:54:07.374774 23673 raft_consensus.cc:348] T 0399c9d1d7b2408292dc78c185605ee8 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
W20240502 01:54:07.377946 20303 leader_election.cc:341] T 0f953dbfbe0b4e93880c5665003395e0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:07.378098 23673 raft_consensus.cc:374] T 0399c9d1d7b2408292dc78c185605ee8 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:07.378263 23673 raft_consensus.cc:729] T 0399c9d1d7b2408292dc78c185605ee8 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:54:07.378582 23673 consensus_queue.cc:260] T 0399c9d1d7b2408292dc78c185605ee8 P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:07.378813 23671 tablet_bootstrap.cc:654] T d25f2ab1a7544cec893dd02c7c89285c P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:07.381163 23670 raft_consensus.cc:482] T 0399c9d1d7b2408292dc78c185605ee8 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:07.381322 23670 raft_consensus.cc:504] T 0399c9d1d7b2408292dc78c185605ee8 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:07.382066 23675 tablet_bootstrap.cc:492] T d25f2ab1a7544cec893dd02c7c89285c P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:54:07.382506 20366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0399c9d1d7b2408292dc78c185605ee8" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
W20240502 01:54:07.383128 17542 leader_election.cc:341] T 0399c9d1d7b2408292dc78c185605ee8 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:07.383498 23670 leader_election.cc:288] T 0399c9d1d7b2408292dc78c185605ee8 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:54:07.384006 23675 tablet_bootstrap.cc:654] T d25f2ab1a7544cec893dd02c7c89285c P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:07.384004 17865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0399c9d1d7b2408292dc78c185605ee8" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:54:07.384233 23644 tablet_bootstrap.cc:492] T 0399c9d1d7b2408292dc78c185605ee8 P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:54:07.384418 23644 ts_tablet_manager.cc:1367] T 0399c9d1d7b2408292dc78c185605ee8 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.055s user 0.000s sys 0.018s
I20240502 01:54:07.386247 23343 consensus_queue.cc:1035] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:07.388830 23644 raft_consensus.cc:348] T 0399c9d1d7b2408292dc78c185605ee8 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:07.389129 23644 raft_consensus.cc:374] T 0399c9d1d7b2408292dc78c185605ee8 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:07.389271 23644 raft_consensus.cc:729] T 0399c9d1d7b2408292dc78c185605ee8 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:54:07.389581 23644 consensus_queue.cc:260] T 0399c9d1d7b2408292dc78c185605ee8 P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:07.390209 23644 ts_tablet_manager.cc:1398] T 0399c9d1d7b2408292dc78c185605ee8 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 01:54:07.390568 23644 tablet_bootstrap.cc:492] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:54:07.390849 17734 raft_consensus.cc:1261] T d10d76ac7b094fe9b230b1afcbed9ca6 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:07.392954 17473 raft_consensus.cc:1261] T 28463d45bfde44429d56179c1eec56b0 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEARNER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:07.392971 17865 raft_consensus.cc:1261] T 28463d45bfde44429d56179c1eec56b0 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:07.393275 23611 raft_consensus.cc:482] T 0399c9d1d7b2408292dc78c185605ee8 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:07.393426 23611 raft_consensus.cc:504] T 0399c9d1d7b2408292dc78c185605ee8 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:07.394095 23611 leader_election.cc:288] T 0399c9d1d7b2408292dc78c185605ee8 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:54:07.394681 23644 tablet_bootstrap.cc:654] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:07.394853 23611 consensus_queue.cc:1035] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:07.396019 17865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0399c9d1d7b2408292dc78c185605ee8" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
W20240502 01:54:07.396564 20303 leader_election.cc:341] T 0399c9d1d7b2408292dc78c185605ee8 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:07.396829 23611 consensus_queue.cc:1035] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:07.405112 20628 raft_consensus.cc:1261] T 28463d45bfde44429d56179c1eec56b0 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:07.405768 17866 raft_consensus.cc:1261] T 0d9f168de0d1410da2518abf936035ff P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:07.406517 23644 tablet_bootstrap.cc:492] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:54:07.406651 23644 ts_tablet_manager.cc:1367] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.007s
I20240502 01:54:07.407471 23644 raft_consensus.cc:348] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:07.407646 23644 raft_consensus.cc:374] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:07.407735 23644 raft_consensus.cc:729] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:54:07.407915 23644 consensus_queue.cc:260] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:07.408435 23644 ts_tablet_manager.cc:1398] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:54:07.409511 23591 raft_consensus.cc:1050] T 139304da92ab4f48b8a08fa24ed60c95 P 2f306549407b4bb281ccf2994035e4f1: attempting to promote NON_VOTER bb31bceba76d466db24b9341c04900c3 to VOTER
I20240502 01:54:07.409703 22876 consensus_queue.cc:1035] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:07.410290 23591 consensus_queue.cc:237] T 139304da92ab4f48b8a08fa24ed60c95 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } }
I20240502 01:54:07.411991 17734 raft_consensus.cc:1203] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:07.394690 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0399c9d1d7b2408292dc78c185605ee8" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:54:07.418952 17604 raft_consensus.cc:2454] T 0399c9d1d7b2408292dc78c185605ee8 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 0.
I20240502 01:54:07.419440 20300 leader_election.cc:302] T 0399c9d1d7b2408292dc78c185605ee8 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, 6a36c952c51542c0bf7ea3ec2857a9f9; no voters: bb31bceba76d466db24b9341c04900c3
I20240502 01:54:07.412879 17605 raft_consensus.cc:1203] T 81f335c023444796ab933e19d809cc03 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:07.420352 23410 raft_consensus.cc:2781] T 0399c9d1d7b2408292dc78c185605ee8 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:07.420460 23410 raft_consensus.cc:482] T 0399c9d1d7b2408292dc78c185605ee8 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:07.420542 23410 raft_consensus.cc:3037] T 0399c9d1d7b2408292dc78c185605ee8 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:07.426296 23410 raft_consensus.cc:504] T 0399c9d1d7b2408292dc78c185605ee8 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:07.427397 17605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0399c9d1d7b2408292dc78c185605ee8" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:54:07.427588 17605 raft_consensus.cc:3037] T 0399c9d1d7b2408292dc78c185605ee8 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:07.430203 17605 raft_consensus.cc:2454] T 0399c9d1d7b2408292dc78c185605ee8 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 1.
I20240502 01:54:07.431217 20300 leader_election.cc:302] T 0399c9d1d7b2408292dc78c185605ee8 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, 6a36c952c51542c0bf7ea3ec2857a9f9; no voters:
I20240502 01:54:07.432579 23611 raft_consensus.cc:2781] T 0399c9d1d7b2408292dc78c185605ee8 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:07.432755 23611 raft_consensus.cc:686] T 0399c9d1d7b2408292dc78c185605ee8 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Becoming Leader. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Running, Role: LEADER
I20240502 01:54:07.433112 23611 consensus_queue.cc:237] T 0399c9d1d7b2408292dc78c185605ee8 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:07.434854 23410 leader_election.cc:288] T 0399c9d1d7b2408292dc78c185605ee8 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Requested vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:54:07.435418 23343 consensus_queue.cc:1035] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:07.436293 23687 ts_tablet_manager.cc:902] T 18549019e31b47d899d011cef0637221 P f19cc6cb1e76460aa6711e49a5775091: Initiating tablet copy from peer 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:54:07.436374 17866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0399c9d1d7b2408292dc78c185605ee8" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
W20240502 01:54:07.413429 17543 leader_election.cc:341] T 0399c9d1d7b2408292dc78c185605ee8 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:07.436829 17543 leader_election.cc:302] T 0399c9d1d7b2408292dc78c185605ee8 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29; no voters: 6a36c952c51542c0bf7ea3ec2857a9f9, bb31bceba76d466db24b9341c04900c3
I20240502 01:54:07.437131 23388 raft_consensus.cc:2726] T 0399c9d1d7b2408292dc78c185605ee8 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 01:54:07.437223 20303 leader_election.cc:341] T 0399c9d1d7b2408292dc78c185605ee8 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:07.378481 20230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f953dbfbe0b4e93880c5665003395e0" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:54:07.437986 20230 raft_consensus.cc:2454] T 0f953dbfbe0b4e93880c5665003395e0 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 0.
I20240502 01:54:07.439049 23652 consensus_queue.cc:1035] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:07.439433 20303 leader_election.cc:302] T 0f953dbfbe0b4e93880c5665003395e0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, fdc2ac372492454c85387589a91f63af; no voters: bb31bceba76d466db24b9341c04900c3
I20240502 01:54:07.391716 20233 raft_consensus.cc:1261] T 0d9f168de0d1410da2518abf936035ff P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:07.440307 23410 raft_consensus.cc:2781] T 0f953dbfbe0b4e93880c5665003395e0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:07.440459 23410 raft_consensus.cc:482] T 0f953dbfbe0b4e93880c5665003395e0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:07.440618 23410 raft_consensus.cc:3037] T 0f953dbfbe0b4e93880c5665003395e0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:07.443466 23410 raft_consensus.cc:504] T 0f953dbfbe0b4e93880c5665003395e0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:07.444443 20233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f953dbfbe0b4e93880c5665003395e0" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:54:07.444707 20233 raft_consensus.cc:3037] T 0f953dbfbe0b4e93880c5665003395e0 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:07.447495 20233 raft_consensus.cc:2454] T 0f953dbfbe0b4e93880c5665003395e0 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 1.
I20240502 01:54:07.448029 20303 leader_election.cc:302] T 0f953dbfbe0b4e93880c5665003395e0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:54:07.448429 23681 raft_consensus.cc:2781] T 0f953dbfbe0b4e93880c5665003395e0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:07.448653 23681 raft_consensus.cc:686] T 0f953dbfbe0b4e93880c5665003395e0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Becoming Leader. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Running, Role: LEADER
I20240502 01:54:07.449093 23681 consensus_queue.cc:237] T 0f953dbfbe0b4e93880c5665003395e0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:07.450978 23410 leader_election.cc:288] T 0f953dbfbe0b4e93880c5665003395e0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Requested vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:54:07.451234 17866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f953dbfbe0b4e93880c5665003395e0" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:54:07.451828 23643 tablet_bootstrap.cc:492] T bbf05e3b3ad44354be7600c3c07756fd P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:54:07.451468 23347 raft_consensus.cc:2932] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:07.452441 17330 catalog_manager.cc:5484] T 0399c9d1d7b2408292dc78c185605ee8 P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70). New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } }
W20240502 01:54:07.456876 20303 leader_election.cc:341] T 0f953dbfbe0b4e93880c5665003395e0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:07.465238 20366 raft_consensus.cc:2932] T d10d76ac7b094fe9b230b1afcbed9ca6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:07.466876 17334 catalog_manager.cc:5484] T 0f953dbfbe0b4e93880c5665003395e0 P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70). New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:07.467693 22876 raft_consensus.cc:2932] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:07.469197 23671 tablet_bootstrap.cc:492] T d25f2ab1a7544cec893dd02c7c89285c P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:54:07.469398 23671 ts_tablet_manager.cc:1367] T d25f2ab1a7544cec893dd02c7c89285c P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.093s user 0.003s sys 0.000s
I20240502 01:54:07.470235 23671 raft_consensus.cc:348] T d25f2ab1a7544cec893dd02c7c89285c P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:07.470748 23671 raft_consensus.cc:374] T d25f2ab1a7544cec893dd02c7c89285c P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:07.470860 23671 raft_consensus.cc:729] T d25f2ab1a7544cec893dd02c7c89285c P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:54:07.470928 23638 raft_consensus.cc:1050] T bffb870715a243d992107583d7c17f1b P fdc2ac372492454c85387589a91f63af: attempting to promote NON_VOTER 4da690c0fa13407a85c0689ce8de9ade to VOTER
I20240502 01:54:07.471176 23671 consensus_queue.cc:260] T d25f2ab1a7544cec893dd02c7c89285c P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:07.471732 23638 consensus_queue.cc:237] T bffb870715a243d992107583d7c17f1b P fdc2ac372492454c85387589a91f63af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } }
I20240502 01:54:07.473318 17865 raft_consensus.cc:2932] T 28463d45bfde44429d56179c1eec56b0 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:07.477041 23410 raft_consensus.cc:482] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:07.477239 23643 tablet_bootstrap.cc:654] T bbf05e3b3ad44354be7600c3c07756fd P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:07.477247 23410 raft_consensus.cc:504] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:07.478142 17605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbf05e3b3ad44354be7600c3c07756fd" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
W20240502 01:54:07.478674 20300 leader_election.cc:341] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:07.479039 23531 tablet_copy_client.cc:783] T 246cec994d274786aad7672f2a30e5d8 P bb31bceba76d466db24b9341c04900c3: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:07.479243 23531 tablet_copy_client.cc:647] T 246cec994d274786aad7672f2a30e5d8 P bb31bceba76d466db24b9341c04900c3: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:07.479554 20233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbf05e3b3ad44354be7600c3c07756fd" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
W20240502 01:54:07.480087 20303 leader_election.cc:341] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:07.480203 20303 leader_election.cc:302] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9; no voters: 3fceccefba274b1dbbd8216af8b0bc29, fdc2ac372492454c85387589a91f63af
I20240502 01:54:07.480522 23410 leader_election.cc:288] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:54:07.480712 23531 tablet_copy_client.cc:523] T 246cec994d274786aad7672f2a30e5d8 P bb31bceba76d466db24b9341c04900c3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:07.480885 23410 raft_consensus.cc:2726] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:54:07.484824 23531 tablet_bootstrap.cc:492] T 246cec994d274786aad7672f2a30e5d8 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:54:07.485873 23642 tablet_bootstrap.cc:492] T 0f953dbfbe0b4e93880c5665003395e0 P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:54:07.486070 23642 ts_tablet_manager.cc:1367] T 0f953dbfbe0b4e93880c5665003395e0 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.205s user 0.009s sys 0.023s
I20240502 01:54:07.489493 17474 raft_consensus.cc:1261] T bffb870715a243d992107583d7c17f1b P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEARNER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:07.489717 23675 tablet_bootstrap.cc:492] T d25f2ab1a7544cec893dd02c7c89285c P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:54:07.489910 23675 ts_tablet_manager.cc:1367] T d25f2ab1a7544cec893dd02c7c89285c P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.108s user 0.004s sys 0.000s
I20240502 01:54:07.490562 20630 raft_consensus.cc:1261] T bffb870715a243d992107583d7c17f1b P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:07.490818 23675 raft_consensus.cc:348] T d25f2ab1a7544cec893dd02c7c89285c P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:07.490932 20233 raft_consensus.cc:1261] T 139304da92ab4f48b8a08fa24ed60c95 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:07.491024 23675 raft_consensus.cc:374] T d25f2ab1a7544cec893dd02c7c89285c P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:07.491127 23675 raft_consensus.cc:729] T d25f2ab1a7544cec893dd02c7c89285c P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:54:07.491503 23697 consensus_queue.cc:1035] T 139304da92ab4f48b8a08fa24ed60c95 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:07.491780 23671 ts_tablet_manager.cc:1398] T d25f2ab1a7544cec893dd02c7c89285c P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20240502 01:54:07.491955 23637 consensus_queue.cc:1035] T bffb870715a243d992107583d7c17f1b P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:07.491381 23675 consensus_queue.cc:260] T d25f2ab1a7544cec893dd02c7c89285c P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:07.493126 23642 raft_consensus.cc:348] T 0f953dbfbe0b4e93880c5665003395e0 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:07.493297 23642 raft_consensus.cc:374] T 0f953dbfbe0b4e93880c5665003395e0 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:07.493355 23642 raft_consensus.cc:729] T 0f953dbfbe0b4e93880c5665003395e0 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:54:07.493562 23642 consensus_queue.cc:260] T 0f953dbfbe0b4e93880c5665003395e0 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:07.491415 20630 raft_consensus.cc:1261] T 139304da92ab4f48b8a08fa24ed60c95 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:07.494235 23642 ts_tablet_manager.cc:1398] T 0f953dbfbe0b4e93880c5665003395e0 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 01:54:07.494535 23642 tablet_bootstrap.cc:492] T 0399c9d1d7b2408292dc78c185605ee8 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:54:07.494577 23639 consensus_queue.cc:1035] T bffb870715a243d992107583d7c17f1b P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:07.495950 17605 raft_consensus.cc:1261] T bffb870715a243d992107583d7c17f1b P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:07.496083 23642 tablet_bootstrap.cc:654] T 0399c9d1d7b2408292dc78c185605ee8 P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:07.496541 23638 consensus_queue.cc:1035] T bffb870715a243d992107583d7c17f1b P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:07.497689 23643 tablet_bootstrap.cc:492] T bbf05e3b3ad44354be7600c3c07756fd P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:54:07.497898 23643 ts_tablet_manager.cc:1367] T bbf05e3b3ad44354be7600c3c07756fd P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.001s
I20240502 01:54:07.498715 23643 raft_consensus.cc:348] T bbf05e3b3ad44354be7600c3c07756fd P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:07.499042 23643 raft_consensus.cc:374] T bbf05e3b3ad44354be7600c3c07756fd P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:07.501314 23675 ts_tablet_manager.cc:1398] T d25f2ab1a7544cec893dd02c7c89285c P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240502 01:54:07.501453 23643 raft_consensus.cc:729] T bbf05e3b3ad44354be7600c3c07756fd P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:54:07.501739 23643 consensus_queue.cc:260] T bbf05e3b3ad44354be7600c3c07756fd P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:07.502120 17333 catalog_manager.cc:5484] T 28463d45bfde44429d56179c1eec56b0 P bb31bceba76d466db24b9341c04900c3 reported cstate change: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:07.502501 23643 ts_tablet_manager.cc:1398] T bbf05e3b3ad44354be7600c3c07756fd P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 01:54:07.508179 23642 tablet_bootstrap.cc:492] T 0399c9d1d7b2408292dc78c185605ee8 P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:54:07.508309 23642 ts_tablet_manager.cc:1367] T 0399c9d1d7b2408292dc78c185605ee8 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.001s
I20240502 01:54:07.508949 23642 raft_consensus.cc:348] T 0399c9d1d7b2408292dc78c185605ee8 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:07.509258 23642 raft_consensus.cc:374] T 0399c9d1d7b2408292dc78c185605ee8 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:07.510003 23642 raft_consensus.cc:729] T 0399c9d1d7b2408292dc78c185605ee8 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:54:07.510553 23642 consensus_queue.cc:260] T 0399c9d1d7b2408292dc78c185605ee8 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:07.515519 23642 ts_tablet_manager.cc:1398] T 0399c9d1d7b2408292dc78c185605ee8 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 01:54:07.518493 23180 raft_consensus.cc:1050] T 3ff84ce60ac1438d92b2c944bb27aabb P f19cc6cb1e76460aa6711e49a5775091: attempting to promote NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 to VOTER
I20240502 01:54:07.519081 23642 tablet_bootstrap.cc:492] T d25f2ab1a7544cec893dd02c7c89285c P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:54:07.519361 23180 consensus_queue.cc:237] T 3ff84ce60ac1438d92b2c944bb27aabb P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } }
I20240502 01:54:07.520669 23642 tablet_bootstrap.cc:654] T d25f2ab1a7544cec893dd02c7c89285c P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:07.521570 20630 raft_consensus.cc:2932] T 28463d45bfde44429d56179c1eec56b0 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:07.528439 22876 raft_consensus.cc:1050] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9: attempting to promote NON_VOTER dfa1152987644bd781b46c7e42a7a685 to VOTER
I20240502 01:54:07.529314 22876 consensus_queue.cc:237] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:07.535431 23638 raft_consensus.cc:482] T bbf05e3b3ad44354be7600c3c07756fd P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:07.535600 23638 raft_consensus.cc:504] T bbf05e3b3ad44354be7600c3c07756fd P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:07.536779 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbf05e3b3ad44354be7600c3c07756fd" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
W20240502 01:54:07.537384 20165 leader_election.cc:341] T bbf05e3b3ad44354be7600c3c07756fd P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:07.537901 23638 leader_election.cc:288] T bbf05e3b3ad44354be7600c3c07756fd P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:54:07.538404 20365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbf05e3b3ad44354be7600c3c07756fd" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:54:07.538620 20365 raft_consensus.cc:2454] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 0.
I20240502 01:54:07.539098 20167 leader_election.cc:302] T bbf05e3b3ad44354be7600c3c07756fd P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, fdc2ac372492454c85387589a91f63af; no voters: 3fceccefba274b1dbbd8216af8b0bc29
I20240502 01:54:07.539345 23638 raft_consensus.cc:2781] T bbf05e3b3ad44354be7600c3c07756fd P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:07.539445 23638 raft_consensus.cc:482] T bbf05e3b3ad44354be7600c3c07756fd P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:07.539526 23638 raft_consensus.cc:3037] T bbf05e3b3ad44354be7600c3c07756fd P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:07.542940 23638 raft_consensus.cc:504] T bbf05e3b3ad44354be7600c3c07756fd P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:07.543910 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbf05e3b3ad44354be7600c3c07756fd" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
W20240502 01:54:07.544433 20165 leader_election.cc:341] T bbf05e3b3ad44354be7600c3c07756fd P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:07.544724 23638 leader_election.cc:288] T bbf05e3b3ad44354be7600c3c07756fd P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 election: Requested vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:54:07.545158 20365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbf05e3b3ad44354be7600c3c07756fd" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
I20240502 01:54:07.545388 20365 raft_consensus.cc:3037] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:07.548018 20365 raft_consensus.cc:2454] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 1.
I20240502 01:54:07.549961 17865 raft_consensus.cc:1261] T 139304da92ab4f48b8a08fa24ed60c95 P bb31bceba76d466db24b9341c04900c3 [term 1 LEARNER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:07.553442 20167 leader_election.cc:302] T bbf05e3b3ad44354be7600c3c07756fd P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, fdc2ac372492454c85387589a91f63af; no voters: 3fceccefba274b1dbbd8216af8b0bc29
I20240502 01:54:07.553612 23343 raft_consensus.cc:1050] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1: attempting to promote NON_VOTER 4da690c0fa13407a85c0689ce8de9ade to VOTER
I20240502 01:54:07.553862 23637 raft_consensus.cc:2781] T bbf05e3b3ad44354be7600c3c07756fd P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:07.554013 23637 raft_consensus.cc:686] T bbf05e3b3ad44354be7600c3c07756fd P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Becoming Leader. State: Replica: fdc2ac372492454c85387589a91f63af, State: Running, Role: LEADER
I20240502 01:54:07.554253 23637 consensus_queue.cc:237] T bbf05e3b3ad44354be7600c3c07756fd P fdc2ac372492454c85387589a91f63af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:07.548997 23591 consensus_queue.cc:1035] T 139304da92ab4f48b8a08fa24ed60c95 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20240502 01:54:07.557310 23673 ts_tablet_manager.cc:1398] T 0399c9d1d7b2408292dc78c185605ee8 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.183s user 0.001s sys 0.001s
I20240502 01:54:07.566737 23343 consensus_queue.cc:237] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } }
I20240502 01:54:07.567229 23673 tablet_bootstrap.cc:492] T bbf05e3b3ad44354be7600c3c07756fd P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:54:07.570053 23711 consensus_queue.cc:1035] T 139304da92ab4f48b8a08fa24ed60c95 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:07.573163 23339 raft_consensus.cc:2932] T bffb870715a243d992107583d7c17f1b P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:07.577004 23396 consensus_queue.cc:1035] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 01:54:07.577672 17673 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.42.68:44083, user_credentials={real_user=slave}} blocked reactor thread for 122026us
I20240502 01:54:07.577303 17475 raft_consensus.cc:2932] T bffb870715a243d992107583d7c17f1b P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:07.583628 17330 catalog_manager.cc:5484] T bffb870715a243d992107583d7c17f1b P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fdc2ac372492454c85387589a91f63af" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:07.584537 20629 raft_consensus.cc:2932] T bffb870715a243d992107583d7c17f1b P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:07.589594 17330 catalog_manager.cc:5484] T bbf05e3b3ad44354be7600c3c07756fd P fdc2ac372492454c85387589a91f63af reported cstate change: term changed from 0 to 1, leader changed from <none> to fdc2ac372492454c85387589a91f63af (127.11.42.69). New cstate: current_term: 1 leader_uuid: "fdc2ac372492454c85387589a91f63af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:07.592311 23717 ts_tablet_manager.cc:902] T 377f8b554ad1448d981d2283b865f8fe P 6a36c952c51542c0bf7ea3ec2857a9f9: Initiating tablet copy from peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:54:07.598090 17327 catalog_manager.cc:5484] T d10d76ac7b094fe9b230b1afcbed9ca6 P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:07.602907 17604 raft_consensus.cc:1261] T 3ff84ce60ac1438d92b2c944bb27aabb P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEARNER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:07.603912 20230 raft_consensus.cc:1261] T 3ff84ce60ac1438d92b2c944bb27aabb P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:07.605031 23180 consensus_queue.cc:1035] T 3ff84ce60ac1438d92b2c944bb27aabb P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:07.605999 23569 consensus_queue.cc:1035] T 3ff84ce60ac1438d92b2c944bb27aabb P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:07.605806 17605 raft_consensus.cc:2932] T bffb870715a243d992107583d7c17f1b P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:07.614825 20494 raft_consensus.cc:1261] T 3ff84ce60ac1438d92b2c944bb27aabb P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:07.612567 23711 raft_consensus.cc:2932] T 139304da92ab4f48b8a08fa24ed60c95 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:07.615813 17865 raft_consensus.cc:2932] T 139304da92ab4f48b8a08fa24ed60c95 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:07.619746 23642 tablet_bootstrap.cc:492] T d25f2ab1a7544cec893dd02c7c89285c P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:54:07.619434 20629 raft_consensus.cc:2932] T 139304da92ab4f48b8a08fa24ed60c95 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:07.619917 23642 ts_tablet_manager.cc:1367] T d25f2ab1a7544cec893dd02c7c89285c P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.101s user 0.008s sys 0.006s
I20240502 01:54:07.620667 23642 raft_consensus.cc:348] T d25f2ab1a7544cec893dd02c7c89285c P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:07.620870 23642 raft_consensus.cc:374] T d25f2ab1a7544cec893dd02c7c89285c P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:07.620971 23642 raft_consensus.cc:729] T d25f2ab1a7544cec893dd02c7c89285c P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:54:07.621237 23642 consensus_queue.cc:260] T d25f2ab1a7544cec893dd02c7c89285c P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:07.621805 23642 ts_tablet_manager.cc:1398] T d25f2ab1a7544cec893dd02c7c89285c P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:07.622628 23531 tablet_bootstrap.cc:492] T 246cec994d274786aad7672f2a30e5d8 P bb31bceba76d466db24b9341c04900c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:07.623034 23531 tablet_bootstrap.cc:492] T 246cec994d274786aad7672f2a30e5d8 P bb31bceba76d466db24b9341c04900c3: Bootstrap complete.
I20240502 01:54:07.623270 23531 ts_tablet_manager.cc:1367] T 246cec994d274786aad7672f2a30e5d8 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.139s user 0.009s sys 0.011s
I20240502 01:54:07.623960 23531 raft_consensus.cc:348] T 246cec994d274786aad7672f2a30e5d8 P bb31bceba76d466db24b9341c04900c3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } }
I20240502 01:54:07.624164 23531 raft_consensus.cc:729] T 246cec994d274786aad7672f2a30e5d8 P bb31bceba76d466db24b9341c04900c3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: LEARNER
I20240502 01:54:07.624366 23531 consensus_queue.cc:260] T 246cec994d274786aad7672f2a30e5d8 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } }
I20240502 01:54:07.624926 23531 ts_tablet_manager.cc:1398] T 246cec994d274786aad7672f2a30e5d8 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:54:07.626355 20231 consensus_queue.cc:237] T bffb870715a243d992107583d7c17f1b P fdc2ac372492454c85387589a91f63af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } }
I20240502 01:54:07.626405 20233 raft_consensus.cc:2932] T 139304da92ab4f48b8a08fa24ed60c95 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:07.629071 17605 raft_consensus.cc:1261] T bffb870715a243d992107583d7c17f1b P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:07.629654 23638 consensus_queue.cc:1035] T bffb870715a243d992107583d7c17f1b P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:07.624935 17327 catalog_manager.cc:5484] T 139304da92ab4f48b8a08fa24ed60c95 P bb31bceba76d466db24b9341c04900c3 reported cstate change: config changed from index 2 to 3, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:07.630842 17475 raft_consensus.cc:1261] T bffb870715a243d992107583d7c17f1b P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:07.633605 23638 consensus_queue.cc:1035] T bffb870715a243d992107583d7c17f1b P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 01:54:07.636744 20518 tablet_copy_service.cc:342] P 2f306549407b4bb281ccf2994035e4f1: Request end of tablet copy session bb31bceba76d466db24b9341c04900c3-246cec994d274786aad7672f2a30e5d8 received from {username='slave'} at 127.11.42.68:49559
I20240502 01:54:07.636889 20518 tablet_copy_service.cc:433] P 2f306549407b4bb281ccf2994035e4f1: ending tablet copy session bb31bceba76d466db24b9341c04900c3-246cec994d274786aad7672f2a30e5d8 on tablet 246cec994d274786aad7672f2a30e5d8 with peer bb31bceba76d466db24b9341c04900c3
I20240502 01:54:07.644075 23637 raft_consensus.cc:2932] T bffb870715a243d992107583d7c17f1b P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:07.648461 17322 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bffb870715a243d992107583d7c17f1b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:07.649633 17475 raft_consensus.cc:2932] T bffb870715a243d992107583d7c17f1b P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:07.657601 23723 ts_tablet_manager.cc:902] T 062b9d90fb0444f4be37bf09770c9caf P 4da690c0fa13407a85c0689ce8de9ade: Initiating tablet copy from peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:54:07.659171 17333 catalog_manager.cc:5484] T bffb870715a243d992107583d7c17f1b P fdc2ac372492454c85387589a91f63af reported cstate change: config changed from index 3 to 4, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New cstate: current_term: 1 leader_uuid: "fdc2ac372492454c85387589a91f63af" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:07.660063 17606 raft_consensus.cc:2932] T bffb870715a243d992107583d7c17f1b P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:07.661321 23569 consensus_queue.cc:1035] T 3ff84ce60ac1438d92b2c944bb27aabb P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:07.662505 23396 consensus_queue.cc:1035] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:07.664009 23569 raft_consensus.cc:482] T d25f2ab1a7544cec893dd02c7c89285c P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:07.664175 23569 raft_consensus.cc:504] T d25f2ab1a7544cec893dd02c7c89285c P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:07.665619 23569 leader_election.cc:288] T d25f2ab1a7544cec893dd02c7c89285c P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:54:07.665743 20609 tablet_service.cc:1514] Processing DeleteTablet for tablet bffb870715a243d992107583d7c17f1b with delete_type TABLET_DATA_TOMBSTONED (TS f19cc6cb1e76460aa6711e49a5775091 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33698
I20240502 01:54:07.669265 17865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d25f2ab1a7544cec893dd02c7c89285c" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:54:07.670514 23717 tablet_copy_client.cc:308] T 377f8b554ad1448d981d2283b865f8fe P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.68:44083
I20240502 01:54:07.671519 17864 raft_consensus.cc:1203] T 246cec994d274786aad7672f2a30e5d8 P bb31bceba76d466db24b9341c04900c3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:07.672518 17887 tablet_copy_service.cc:140] P bb31bceba76d466db24b9341c04900c3: Received BeginTabletCopySession request for tablet 377f8b554ad1448d981d2283b865f8fe from peer 6a36c952c51542c0bf7ea3ec2857a9f9 ({username='slave'} at 127.11.42.70:41203)
I20240502 01:54:07.673945 17887 tablet_copy_service.cc:161] P bb31bceba76d466db24b9341c04900c3: Beginning new tablet copy session on tablet 377f8b554ad1448d981d2283b865f8fe from peer 6a36c952c51542c0bf7ea3ec2857a9f9 at {username='slave'} at 127.11.42.70:41203: session id = 6a36c952c51542c0bf7ea3ec2857a9f9-377f8b554ad1448d981d2283b865f8fe
I20240502 01:54:07.676965 17474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d25f2ab1a7544cec893dd02c7c89285c" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:54:07.677211 17474 raft_consensus.cc:2454] T d25f2ab1a7544cec893dd02c7c89285c P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 0.
I20240502 01:54:07.677634 20564 leader_election.cc:302] T d25f2ab1a7544cec893dd02c7c89285c P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:54:07.677949 23569 raft_consensus.cc:2781] T d25f2ab1a7544cec893dd02c7c89285c P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:07.678040 23569 raft_consensus.cc:482] T d25f2ab1a7544cec893dd02c7c89285c P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:07.678145 23569 raft_consensus.cc:3037] T d25f2ab1a7544cec893dd02c7c89285c P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:07.678364 17887 tablet_copy_source_session.cc:205] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:07.679379 17865 raft_consensus.cc:2454] T d25f2ab1a7544cec893dd02c7c89285c P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 0.
I20240502 01:54:07.679808 23717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 377f8b554ad1448d981d2283b865f8fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:07.680758 23569 raft_consensus.cc:504] T d25f2ab1a7544cec893dd02c7c89285c P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:07.681879 17865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d25f2ab1a7544cec893dd02c7c89285c" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:54:07.682054 17865 raft_consensus.cc:3037] T d25f2ab1a7544cec893dd02c7c89285c P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:07.682262 23569 leader_election.cc:288] T d25f2ab1a7544cec893dd02c7c89285c P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 election: Requested vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:54:07.683935 17327 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:54:07.685550 17476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d25f2ab1a7544cec893dd02c7c89285c" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:54:07.685743 17476 raft_consensus.cc:3037] T d25f2ab1a7544cec893dd02c7c89285c P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:07.688339 17476 raft_consensus.cc:2454] T d25f2ab1a7544cec893dd02c7c89285c P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 1.
I20240502 01:54:07.688618 20629 raft_consensus.cc:1261] T a212c17ba64842c99b3d6a78a3b7cc8a P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:07.688855 17865 raft_consensus.cc:2454] T d25f2ab1a7544cec893dd02c7c89285c P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 1.
I20240502 01:54:07.689201 23611 consensus_queue.cc:1035] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:07.689282 20565 leader_election.cc:302] T d25f2ab1a7544cec893dd02c7c89285c P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb31bceba76d466db24b9341c04900c3, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:54:07.689553 23569 raft_consensus.cc:2781] T d25f2ab1a7544cec893dd02c7c89285c P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:07.689769 17476 raft_consensus.cc:2932] T 28463d45bfde44429d56179c1eec56b0 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:07.687068 20233 raft_consensus.cc:2932] T d10d76ac7b094fe9b230b1afcbed9ca6 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:07.692960 23569 raft_consensus.cc:686] T d25f2ab1a7544cec893dd02c7c89285c P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Becoming Leader. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Running, Role: LEADER
I20240502 01:54:07.693394 23569 consensus_queue.cc:237] T d25f2ab1a7544cec893dd02c7c89285c P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:07.694844 23717 tablet_copy_client.cc:783] T 377f8b554ad1448d981d2283b865f8fe P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:07.695158 23717 tablet_copy_client.cc:647] T 377f8b554ad1448d981d2283b865f8fe P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:07.697464 23717 tablet_copy_client.cc:523] T 377f8b554ad1448d981d2283b865f8fe P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:07.700027 22991 raft_consensus.cc:2932] T 3ff84ce60ac1438d92b2c944bb27aabb P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:07.700886 23717 tablet_bootstrap.cc:492] T 377f8b554ad1448d981d2283b865f8fe P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:54:07.705830 17604 raft_consensus.cc:2932] T 3ff84ce60ac1438d92b2c944bb27aabb P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:07.711599 20367 consensus_queue.cc:237] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } }
I20240502 01:54:07.713061 23586 raft_consensus.cc:482] T 0399c9d1d7b2408292dc78c185605ee8 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:07.713207 23586 raft_consensus.cc:504] T 0399c9d1d7b2408292dc78c185605ee8 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:07.714041 23586 leader_election.cc:288] T 0399c9d1d7b2408292dc78c185605ee8 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:54:07.714669 17606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0399c9d1d7b2408292dc78c185605ee8" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:54:07.714936 17606 raft_consensus.cc:2379] T 0399c9d1d7b2408292dc78c185605ee8 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bb31bceba76d466db24b9341c04900c3 in current term 1: Already voted for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in this term.
I20240502 01:54:07.715132 20366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0399c9d1d7b2408292dc78c185605ee8" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:54:07.715994 17476 raft_consensus.cc:1261] T 28463d45bfde44429d56179c1eec56b0 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:07.716118 17800 leader_election.cc:302] T 0399c9d1d7b2408292dc78c185605ee8 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb31bceba76d466db24b9341c04900c3; no voters: 3fceccefba274b1dbbd8216af8b0bc29, 6a36c952c51542c0bf7ea3ec2857a9f9
I20240502 01:54:07.716796 23410 consensus_queue.cc:1035] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:07.717975 23586 raft_consensus.cc:3037] T 0399c9d1d7b2408292dc78c185605ee8 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:07.721149 23586 raft_consensus.cc:2726] T 0399c9d1d7b2408292dc78c185605ee8 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:54:07.721838 17865 raft_consensus.cc:1261] T 28463d45bfde44429d56179c1eec56b0 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:07.722481 23410 consensus_queue.cc:1035] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:07.724889 23410 raft_consensus.cc:2932] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:07.725464 17864 raft_consensus.cc:2932] T 28463d45bfde44429d56179c1eec56b0 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:07.729344 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 28463d45bfde44429d56179c1eec56b0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:07.744719 20233 raft_consensus.cc:2932] T 3ff84ce60ac1438d92b2c944bb27aabb P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:07.745575 17734 raft_consensus.cc:1261] T a212c17ba64842c99b3d6a78a3b7cc8a P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:07.746517 23410 consensus_queue.cc:1035] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:07.747859 23673 tablet_bootstrap.cc:654] T bbf05e3b3ad44354be7600c3c07756fd P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:07.750137 17334 catalog_manager.cc:5484] T 3ff84ce60ac1438d92b2c944bb27aabb P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:07.752821 17474 raft_consensus.cc:1261] T a212c17ba64842c99b3d6a78a3b7cc8a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:07.753351 23410 consensus_queue.cc:1035] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:07.755049 17475 raft_consensus.cc:2932] T 28463d45bfde44429d56179c1eec56b0 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:07.756533 20494 consensus_queue.cc:237] T 139304da92ab4f48b8a08fa24ed60c95 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } }
I20240502 01:54:07.758620 17865 raft_consensus.cc:1261] T 139304da92ab4f48b8a08fa24ed60c95 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:07.759618 23711 consensus_queue.cc:1035] T 139304da92ab4f48b8a08fa24ed60c95 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:07.762066 23711 raft_consensus.cc:2932] T 139304da92ab4f48b8a08fa24ed60c95 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:07.765568 17867 raft_consensus.cc:2932] T 139304da92ab4f48b8a08fa24ed60c95 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:07.768520 23727 tablet_replica.cc:330] T bffb870715a243d992107583d7c17f1b P f19cc6cb1e76460aa6711e49a5775091: stopping tablet replica
I20240502 01:54:07.768736 23727 raft_consensus.cc:2229] T bffb870715a243d992107583d7c17f1b P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:07.768961 23727 raft_consensus.cc:2258] T bffb870715a243d992107583d7c17f1b P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:07.773715 17327 catalog_manager.cc:5484] T 28463d45bfde44429d56179c1eec56b0 P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: config changed from index 3 to 4, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:07.780189 17326 catalog_manager.cc:5484] T 139304da92ab4f48b8a08fa24ed60c95 P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: config changed from index 3 to 4, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:07.781874 23727 ts_tablet_manager.cc:1875] T bffb870715a243d992107583d7c17f1b P f19cc6cb1e76460aa6711e49a5775091: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:07.785210 20212 tablet_service.cc:1514] Processing DeleteTablet for tablet 139304da92ab4f48b8a08fa24ed60c95 with delete_type TABLET_DATA_TOMBSTONED (TS fdc2ac372492454c85387589a91f63af not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44624
I20240502 01:54:07.789052 23611 raft_consensus.cc:2932] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:07.793869 17476 raft_consensus.cc:2932] T a212c17ba64842c99b3d6a78a3b7cc8a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:07.795032 17322 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 139304da92ab4f48b8a08fa24ed60c95 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:07.796226 20629 raft_consensus.cc:2932] T a212c17ba64842c99b3d6a78a3b7cc8a P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:07.800261 17326 catalog_manager.cc:5484] T a212c17ba64842c99b3d6a78a3b7cc8a P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: config changed from index 5 to 6, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:07.801137 20366 raft_consensus.cc:1261] T fac0ba26fabd4643bc38b7559faa471c P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:07.801674 23711 consensus_queue.cc:1035] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:07.813537 20233 raft_consensus.cc:1261] T 0f953dbfbe0b4e93880c5665003395e0 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:07.816126 22876 consensus_queue.cc:1035] T 0f953dbfbe0b4e93880c5665003395e0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:07.818379 23363 raft_consensus.cc:2932] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:07.826644 23727 ts_tablet_manager.cc:1888] T bffb870715a243d992107583d7c17f1b P f19cc6cb1e76460aa6711e49a5775091: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:54:07.827682 17476 raft_consensus.cc:1261] T fac0ba26fabd4643bc38b7559faa471c P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEARNER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:07.830427 23343 consensus_queue.cc:1035] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 01:54:07.836244 17802 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.42.67:36115, user_credentials={real_user=slave}} blocked reactor thread for 76699us
I20240502 01:54:07.845417 17866 raft_consensus.cc:3037] T 0f953dbfbe0b4e93880c5665003395e0 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:07.845919 17734 raft_consensus.cc:2932] T a212c17ba64842c99b3d6a78a3b7cc8a P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:07.851039 17866 raft_consensus.cc:1261] T 0f953dbfbe0b4e93880c5665003395e0 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:07.852406 23742 tablet_replica.cc:330] T 139304da92ab4f48b8a08fa24ed60c95 P fdc2ac372492454c85387589a91f63af: stopping tablet replica
I20240502 01:54:07.852913 23742 raft_consensus.cc:2229] T 139304da92ab4f48b8a08fa24ed60c95 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:07.853196 23742 raft_consensus.cc:2258] T 139304da92ab4f48b8a08fa24ed60c95 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:07.854121 23742 ts_tablet_manager.cc:1875] T 139304da92ab4f48b8a08fa24ed60c95 P fdc2ac372492454c85387589a91f63af: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:07.854940 23727 log.cc:1196] T bffb870715a243d992107583d7c17f1b P f19cc6cb1e76460aa6711e49a5775091: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal/wals/bffb870715a243d992107583d7c17f1b
W20240502 01:54:07.839491 20562 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.42.66:38531, user_credentials={real_user=slave}} blocked reactor thread for 98662.4us
I20240502 01:54:07.857968 20630 raft_consensus.cc:1261] T 139304da92ab4f48b8a08fa24ed60c95 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:07.858279 17321 catalog_manager.cc:4830] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): tablet bffb870715a243d992107583d7c17f1b (table rebalancer_extra_table_32 [id=6cf4b74847bf491c93c43661f9c6d7a5]) successfully deleted
I20240502 01:54:07.859548 20630 raft_consensus.cc:1261] T fac0ba26fabd4643bc38b7559faa471c P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:07.861090 20609 tablet_service.cc:1514] Processing DeleteTablet for tablet 28463d45bfde44429d56179c1eec56b0 with delete_type TABLET_DATA_TOMBSTONED (TS f19cc6cb1e76460aa6711e49a5775091 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33698
I20240502 01:54:07.862116 23694 consensus_queue.cc:1035] T 139304da92ab4f48b8a08fa24ed60c95 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:07.862375 23711 consensus_queue.cc:1035] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:07.863634 23673 tablet_bootstrap.cc:492] T bbf05e3b3ad44354be7600c3c07756fd P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:54:07.864022 23673 ts_tablet_manager.cc:1367] T bbf05e3b3ad44354be7600c3c07756fd P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.297s user 0.000s sys 0.005s
I20240502 01:54:07.864828 23739 raft_consensus.cc:1050] T 81f335c023444796ab933e19d809cc03 P f19cc6cb1e76460aa6711e49a5775091: attempting to promote NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 to VOTER
I20240502 01:54:07.865247 20626 raft_consensus.cc:2932] T 139304da92ab4f48b8a08fa24ed60c95 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:07.865962 23739 consensus_queue.cc:237] T 81f335c023444796ab933e19d809cc03 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } }
I20240502 01:54:07.869315 20231 raft_consensus.cc:2932] T 0d9f168de0d1410da2518abf936035ff P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:07.872267 20494 raft_consensus.cc:2932] T 3ff84ce60ac1438d92b2c944bb27aabb P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:07.872736 17334 catalog_manager.cc:5484] T d25f2ab1a7544cec893dd02c7c89285c P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: term changed from 0 to 1, leader changed from <none> to f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72). New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:07.873519 20629 raft_consensus.cc:2932] T 0d9f168de0d1410da2518abf936035ff P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:07.873854 22876 consensus_queue.cc:1035] T 0399c9d1d7b2408292dc78c185605ee8 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:07.877740 23723 tablet_copy_client.cc:308] T 062b9d90fb0444f4be37bf09770c9caf P 4da690c0fa13407a85c0689ce8de9ade: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.69:34339
I20240502 01:54:07.878409 20253 tablet_copy_service.cc:140] P fdc2ac372492454c85387589a91f63af: Received BeginTabletCopySession request for tablet 062b9d90fb0444f4be37bf09770c9caf from peer 4da690c0fa13407a85c0689ce8de9ade ({username='slave'} at 127.11.42.65:54337)
I20240502 01:54:07.878553 20253 tablet_copy_service.cc:161] P fdc2ac372492454c85387589a91f63af: Beginning new tablet copy session on tablet 062b9d90fb0444f4be37bf09770c9caf from peer 4da690c0fa13407a85c0689ce8de9ade at {username='slave'} at 127.11.42.65:54337: session id = 4da690c0fa13407a85c0689ce8de9ade-062b9d90fb0444f4be37bf09770c9caf
I20240502 01:54:07.882611 20253 tablet_copy_source_session.cc:205] T 062b9d90fb0444f4be37bf09770c9caf P fdc2ac372492454c85387589a91f63af: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:07.884075 20496 consensus_queue.cc:237] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:07.884594 23727 tablet_replica.cc:330] T 28463d45bfde44429d56179c1eec56b0 P f19cc6cb1e76460aa6711e49a5775091: stopping tablet replica
I20240502 01:54:07.884749 23723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 062b9d90fb0444f4be37bf09770c9caf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:07.885967 23727 raft_consensus.cc:2229] T 28463d45bfde44429d56179c1eec56b0 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:07.886058 20231 raft_consensus.cc:1261] T d10d76ac7b094fe9b230b1afcbed9ca6 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:07.886328 23727 raft_consensus.cc:2258] T 28463d45bfde44429d56179c1eec56b0 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:07.888409 23694 consensus_queue.cc:1035] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:07.891952 20231 raft_consensus.cc:2932] T d10d76ac7b094fe9b230b1afcbed9ca6 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:07.893097 20230 raft_consensus.cc:1261] T 81f335c023444796ab933e19d809cc03 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:07.893762 23748 consensus_queue.cc:1035] T 81f335c023444796ab933e19d809cc03 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:07.891781 23694 raft_consensus.cc:2932] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:07.896705 17604 raft_consensus.cc:1261] T 81f335c023444796ab933e19d809cc03 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEARNER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:07.898294 20494 raft_consensus.cc:1261] T 81f335c023444796ab933e19d809cc03 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:07.901224 23569 consensus_queue.cc:1035] T 81f335c023444796ab933e19d809cc03 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:07.903390 23569 consensus_queue.cc:1035] T 81f335c023444796ab933e19d809cc03 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:07.907117 23723 tablet_copy_client.cc:783] T 062b9d90fb0444f4be37bf09770c9caf P 4da690c0fa13407a85c0689ce8de9ade: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:07.907402 23723 tablet_copy_client.cc:647] T 062b9d90fb0444f4be37bf09770c9caf P 4da690c0fa13407a85c0689ce8de9ade: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:07.907640 23711 raft_consensus.cc:2932] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:07.908958 23723 tablet_copy_client.cc:523] T 062b9d90fb0444f4be37bf09770c9caf P 4da690c0fa13407a85c0689ce8de9ade: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:07.912256 17476 raft_consensus.cc:2932] T fac0ba26fabd4643bc38b7559faa471c P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:07.913797 17328 catalog_manager.cc:5484] T 0d9f168de0d1410da2518abf936035ff P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: config changed from index 7 to 8, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:07.915272 23727 ts_tablet_manager.cc:1875] T 28463d45bfde44429d56179c1eec56b0 P f19cc6cb1e76460aa6711e49a5775091: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:07.920161 23723 tablet_bootstrap.cc:492] T 062b9d90fb0444f4be37bf09770c9caf P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:54:07.921388 17330 catalog_manager.cc:5484] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:07.923629 20628 raft_consensus.cc:2932] T fac0ba26fabd4643bc38b7559faa471c P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:07.940125 23739 raft_consensus.cc:2932] T 81f335c023444796ab933e19d809cc03 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:07.940701 20626 consensus_queue.cc:237] T 3ff84ce60ac1438d92b2c944bb27aabb P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } }
I20240502 01:54:07.942678 20231 raft_consensus.cc:1261] T 3ff84ce60ac1438d92b2c944bb27aabb P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:07.943346 22991 consensus_queue.cc:1035] T 3ff84ce60ac1438d92b2c944bb27aabb P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:07.943732 17606 raft_consensus.cc:1261] T 3ff84ce60ac1438d92b2c944bb27aabb P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:07.943804 17604 raft_consensus.cc:2932] T 81f335c023444796ab933e19d809cc03 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
W20240502 01:54:07.945442 17673 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.11.42.126:39453, user_credentials={real_user=slave}} blocked reactor thread for 75766.9us
I20240502 01:54:07.950306 17735 raft_consensus.cc:1261] T d10d76ac7b094fe9b230b1afcbed9ca6 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:07.950832 17734 raft_consensus.cc:2932] T d10d76ac7b094fe9b230b1afcbed9ca6 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:07.953722 17737 consensus_queue.cc:237] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } }
I20240502 01:54:07.952881 23711 consensus_queue.cc:1035] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:07.960484 20366 raft_consensus.cc:2932] T fac0ba26fabd4643bc38b7559faa471c P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:07.963258 23731 consensus_queue.cc:1035] T 3ff84ce60ac1438d92b2c944bb27aabb P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:07.963541 23723 tablet_bootstrap.cc:492] T 062b9d90fb0444f4be37bf09770c9caf P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:07.963946 23723 tablet_bootstrap.cc:492] T 062b9d90fb0444f4be37bf09770c9caf P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap complete.
I20240502 01:54:07.964215 23723 ts_tablet_manager.cc:1367] T 062b9d90fb0444f4be37bf09770c9caf P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.044s user 0.008s sys 0.000s
I20240502 01:54:07.965056 23723 raft_consensus.cc:348] T 062b9d90fb0444f4be37bf09770c9caf P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } }
I20240502 01:54:07.965322 23723 raft_consensus.cc:729] T 062b9d90fb0444f4be37bf09770c9caf P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: LEARNER
I20240502 01:54:07.965523 23723 consensus_queue.cc:260] T 062b9d90fb0444f4be37bf09770c9caf P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } }
I20240502 01:54:07.966512 20496 consensus_queue.cc:237] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } }
I20240502 01:54:07.967131 23748 raft_consensus.cc:2932] T 3ff84ce60ac1438d92b2c944bb27aabb P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:07.967340 17326 catalog_manager.cc:5484] T 81f335c023444796ab933e19d809cc03 P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:07.947149 17865 raft_consensus.cc:2932] T 0d9f168de0d1410da2518abf936035ff P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:07.972419 23723 ts_tablet_manager.cc:1398] T 062b9d90fb0444f4be37bf09770c9caf P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20240502 01:54:07.973471 20253 tablet_copy_service.cc:342] P fdc2ac372492454c85387589a91f63af: Request end of tablet copy session 4da690c0fa13407a85c0689ce8de9ade-062b9d90fb0444f4be37bf09770c9caf received from {username='slave'} at 127.11.42.65:54337
I20240502 01:54:07.973608 20253 tablet_copy_service.cc:433] P fdc2ac372492454c85387589a91f63af: ending tablet copy session 4da690c0fa13407a85c0689ce8de9ade-062b9d90fb0444f4be37bf09770c9caf on tablet 062b9d90fb0444f4be37bf09770c9caf with peer 4da690c0fa13407a85c0689ce8de9ade
I20240502 01:54:07.975214 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3ff84ce60ac1438d92b2c944bb27aabb with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:07.975831 17322 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d10d76ac7b094fe9b230b1afcbed9ca6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:07.978049 17476 raft_consensus.cc:1261] T fac0ba26fabd4643bc38b7559faa471c P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:07.978150 20494 raft_consensus.cc:2932] T 81f335c023444796ab933e19d809cc03 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:07.978781 23347 consensus_queue.cc:1035] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:07.981252 23710 raft_consensus.cc:2932] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:07.982077 20231 raft_consensus.cc:2932] T 3ff84ce60ac1438d92b2c944bb27aabb P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:07.986899 23739 raft_consensus.cc:1050] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P f19cc6cb1e76460aa6711e49a5775091: attempting to promote NON_VOTER dfa1152987644bd781b46c7e42a7a685 to VOTER
I20240502 01:54:07.987749 23739 consensus_queue.cc:237] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:07.987906 17322 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fac0ba26fabd4643bc38b7559faa471c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:07.988543 17326 catalog_manager.cc:5484] T 3ff84ce60ac1438d92b2c944bb27aabb P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: config changed from index 3 to 4, VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) evicted. New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:07.989761 17473 raft_consensus.cc:2932] T fac0ba26fabd4643bc38b7559faa471c P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:07.993927 20476 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ff84ce60ac1438d92b2c944bb27aabb with delete_type TABLET_DATA_TOMBSTONED (TS 2f306549407b4bb281ccf2994035e4f1 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35018
I20240502 01:54:07.994454 17606 raft_consensus.cc:2932] T 3ff84ce60ac1438d92b2c944bb27aabb P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:07.998759 23673 raft_consensus.cc:348] T bbf05e3b3ad44354be7600c3c07756fd P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:07.999073 23673 raft_consensus.cc:374] T bbf05e3b3ad44354be7600c3c07756fd P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:07.999199 23673 raft_consensus.cc:729] T bbf05e3b3ad44354be7600c3c07756fd P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:54:07.999711 23673 consensus_queue.cc:260] T bbf05e3b3ad44354be7600c3c07756fd P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:08.000174 21667 raft_consensus.cc:482] T 0f953dbfbe0b4e93880c5665003395e0 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:08.000317 21667 raft_consensus.cc:504] T 0f953dbfbe0b4e93880c5665003395e0 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:08.001314 21667 leader_election.cc:288] T 0f953dbfbe0b4e93880c5665003395e0 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:54:08.001875 20232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f953dbfbe0b4e93880c5665003395e0" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:54:08.003580 20365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f953dbfbe0b4e93880c5665003395e0" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:54:08.004320 17802 leader_election.cc:302] T 0f953dbfbe0b4e93880c5665003395e0 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb31bceba76d466db24b9341c04900c3; no voters: 6a36c952c51542c0bf7ea3ec2857a9f9, fdc2ac372492454c85387589a91f63af
I20240502 01:54:08.004842 23740 raft_consensus.cc:2726] T 0f953dbfbe0b4e93880c5665003395e0 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:08.008528 23410 consensus_queue.cc:1035] T 0f953dbfbe0b4e93880c5665003395e0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:08.010165 20365 raft_consensus.cc:1261] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:08.012144 17866 raft_consensus.cc:1261] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:08.012777 22991 consensus_queue.cc:1035] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:08.013835 23748 consensus_queue.cc:1035] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:08.014021 17330 catalog_manager.cc:5484] T fac0ba26fabd4643bc38b7559faa471c P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: config changed from index 3 to 4, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:08.017383 17734 raft_consensus.cc:1261] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:08.019197 22991 consensus_queue.cc:1035] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:08.020113 23687 tablet_copy_client.cc:308] T 18549019e31b47d899d011cef0637221 P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.70:39155
I20240502 01:54:08.020125 23639 consensus_queue.cc:1035] T bbf05e3b3ad44354be7600c3c07756fd P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:08.020745 20387 tablet_copy_service.cc:140] P 6a36c952c51542c0bf7ea3ec2857a9f9: Received BeginTabletCopySession request for tablet 18549019e31b47d899d011cef0637221 from peer f19cc6cb1e76460aa6711e49a5775091 ({username='slave'} at 127.11.42.72:36559)
I20240502 01:54:08.020946 20387 tablet_copy_service.cc:161] P 6a36c952c51542c0bf7ea3ec2857a9f9: Beginning new tablet copy session on tablet 18549019e31b47d899d011cef0637221 from peer f19cc6cb1e76460aa6711e49a5775091 at {username='slave'} at 127.11.42.72:36559: session id = f19cc6cb1e76460aa6711e49a5775091-18549019e31b47d899d011cef0637221
I20240502 01:54:08.024083 20387 tablet_copy_source_session.cc:205] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:08.024533 23410 consensus_queue.cc:1035] T 0399c9d1d7b2408292dc78c185605ee8 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:08.026252 23687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18549019e31b47d899d011cef0637221. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:08.031711 17735 raft_consensus.cc:2932] T d10d76ac7b094fe9b230b1afcbed9ca6 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:08.018795 17326 catalog_manager.cc:5484] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: config changed from index 3 to 4, VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) evicted. New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:08.044916 23742 ts_tablet_manager.cc:1888] T 139304da92ab4f48b8a08fa24ed60c95 P fdc2ac372492454c85387589a91f63af: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:54:08.045069 23673 ts_tablet_manager.cc:1398] T bbf05e3b3ad44354be7600c3c07756fd P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.181s user 0.001s sys 0.003s
I20240502 01:54:08.045150 23742 log.cc:1196] T 139304da92ab4f48b8a08fa24ed60c95 P fdc2ac372492454c85387589a91f63af: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal/wals/139304da92ab4f48b8a08fa24ed60c95
I20240502 01:54:08.046088 17322 catalog_manager.cc:4830] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): tablet 139304da92ab4f48b8a08fa24ed60c95 (table rebalancer_extra_table_12 [id=ecb981650f8c425798b67a5194ad793c]) successfully deleted
I20240502 01:54:08.047809 23731 raft_consensus.cc:2932] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:08.048628 17867 raft_consensus.cc:2932] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:08.052040 17736 raft_consensus.cc:2932] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:08.052920 20365 raft_consensus.cc:2932] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:08.059051 17333 catalog_manager.cc:5576] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339) does not have the latest schema for tablet 0f953dbfbe0b4e93880c5665003395e0 (table rebalancer_dynamic_table [id=5b37c5f6d0cc4a34b264799964d1ffe9]). Expected version 1 got 0
I20240502 01:54:08.060320 17334 catalog_manager.cc:5484] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:08.060830 17735 consensus_queue.cc:237] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } }
I20240502 01:54:08.063598 20628 raft_consensus.cc:1261] T 0d9f168de0d1410da2518abf936035ff P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:08.064080 22568 consensus_queue.cc:1035] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:08.066458 22568 raft_consensus.cc:2932] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:08.069564 20367 raft_consensus.cc:1261] T fac0ba26fabd4643bc38b7559faa471c P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:08.070397 20628 raft_consensus.cc:2932] T 0d9f168de0d1410da2518abf936035ff P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:08.073104 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0d9f168de0d1410da2518abf936035ff with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:08.074396 23711 consensus_queue.cc:1035] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:08.081353 17476 raft_consensus.cc:1203] T 062b9d90fb0444f4be37bf09770c9caf P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:08.083003 20230 raft_consensus.cc:1261] T 0d9f168de0d1410da2518abf936035ff P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:08.082809 20233 raft_consensus.cc:2932] T 81f335c023444796ab933e19d809cc03 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:08.078570 23776 tablet.cc:1722] T 0f953dbfbe0b4e93880c5665003395e0 P fdc2ac372492454c85387589a91f63af: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:08.085693 20366 raft_consensus.cc:2932] T fac0ba26fabd4643bc38b7559faa471c P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:08.086710 20609 tablet_service.cc:1514] Processing DeleteTablet for tablet fac0ba26fabd4643bc38b7559faa471c with delete_type TABLET_DATA_TOMBSTONED (TS f19cc6cb1e76460aa6711e49a5775091 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33698
I20240502 01:54:08.087235 23776 tablet.cc:1620] T 0f953dbfbe0b4e93880c5665003395e0 P fdc2ac372492454c85387589a91f63af: MemRowSet was empty: no flush needed.
I20240502 01:54:08.092979 20629 consensus_queue.cc:237] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:08.094753 17867 raft_consensus.cc:1261] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:08.095285 23731 consensus_queue.cc:1035] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:08.097698 23731 raft_consensus.cc:2932] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:08.098011 17328 catalog_manager.cc:5484] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 reported cstate change: config changed from index 8 to 9, VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) evicted. New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:08.100970 17863 raft_consensus.cc:2932] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:08.104604 23396 consensus_queue.cc:1035] T 0d9f168de0d1410da2518abf936035ff P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:08.105484 17846 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d9f168de0d1410da2518abf936035ff with delete_type TABLET_DATA_TOMBSTONED (TS bb31bceba76d466db24b9341c04900c3 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:47226
I20240502 01:54:08.109103 20629 consensus_queue.cc:237] T 81f335c023444796ab933e19d809cc03 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } }
I20240502 01:54:08.111814 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bfd7b1d1002c4b57a14d43ed9c6d9f50 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:08.111887 20626 raft_consensus.cc:1261] T 5df83f83d2674155afba93c2d992af0a P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:08.112460 23363 consensus_queue.cc:1035] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:08.113441 17604 raft_consensus.cc:1261] T 81f335c023444796ab933e19d809cc03 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:08.114391 23765 tablet_replica.cc:330] T 3ff84ce60ac1438d92b2c944bb27aabb P 2f306549407b4bb281ccf2994035e4f1: stopping tablet replica
I20240502 01:54:08.114665 23765 raft_consensus.cc:2229] T 3ff84ce60ac1438d92b2c944bb27aabb P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:08.114807 17476 raft_consensus.cc:1261] T 5df83f83d2674155afba93c2d992af0a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:08.115087 23765 raft_consensus.cc:2258] T 3ff84ce60ac1438d92b2c944bb27aabb P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:08.116137 22568 consensus_queue.cc:1035] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:08.116679 17326 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=5b37c5f6d0cc4a34b264799964d1ffe9] alter complete (version 1)
I20240502 01:54:08.118280 22991 consensus_queue.cc:1035] T 81f335c023444796ab933e19d809cc03 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20240502 01:54:08.118182 17328 catalog_manager.cc:5484] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: config changed from index 3 to 4, VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) evicted. New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:08.119601 23363 raft_consensus.cc:2932] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:54:08.120875 22991 raft_consensus.cc:2932] T 81f335c023444796ab933e19d809cc03 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:08.121598 17476 raft_consensus.cc:2932] T 5df83f83d2674155afba93c2d992af0a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:54:08.122475 17604 raft_consensus.cc:2932] T 81f335c023444796ab933e19d809cc03 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:08.129130 17328 catalog_manager.cc:5484] T 5df83f83d2674155afba93c2d992af0a P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: config changed from index 6 to 7, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:54:08.133348 23766 raft_consensus.cc:482] T bbf05e3b3ad44354be7600c3c07756fd P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:08.133471 23766 raft_consensus.cc:504] T bbf05e3b3ad44354be7600c3c07756fd P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:08.134152 23765 ts_tablet_manager.cc:1875] T 3ff84ce60ac1438d92b2c944bb27aabb P 2f306549407b4bb281ccf2994035e4f1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:08.134214 23766 leader_election.cc:288] T bbf05e3b3ad44354be7600c3c07756fd P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:54:08.135133 20230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbf05e3b3ad44354be7600c3c07756fd" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:54:08.136814 17333 catalog_manager.cc:5484] T 81f335c023444796ab933e19d809cc03 P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: config changed from index 3 to 4, VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) evicted. New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:08.143673 17326 catalog_manager.cc:5576] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083) does not have the latest schema for tablet 0f953dbfbe0b4e93880c5665003395e0 (table rebalancer_dynamic_table [id=5b37c5f6d0cc4a34b264799964d1ffe9]). Expected version 1 got 0
I20240502 01:54:08.148635 20231 raft_consensus.cc:1261] T 81f335c023444796ab933e19d809cc03 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:08.149320 17604 raft_consensus.cc:3037] T bbf05e3b3ad44354be7600c3c07756fd P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:08.150391 17734 raft_consensus.cc:1261] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:08.153424 20476 tablet_service.cc:1514] Processing DeleteTablet for tablet 81f335c023444796ab933e19d809cc03 with delete_type TABLET_DATA_TOMBSTONED (TS 2f306549407b4bb281ccf2994035e4f1 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35018
I20240502 01:54:08.153963 23347 raft_consensus.cc:1050] T 246cec994d274786aad7672f2a30e5d8 P 2f306549407b4bb281ccf2994035e4f1: attempting to promote NON_VOTER bb31bceba76d466db24b9341c04900c3 to VOTER
I20240502 01:54:08.154412 23638 consensus_queue.cc:1035] T bbf05e3b3ad44354be7600c3c07756fd P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:08.154803 23347 consensus_queue.cc:237] T 246cec994d274786aad7672f2a30e5d8 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } }
I20240502 01:54:08.155820 20231 raft_consensus.cc:2932] T 0d9f168de0d1410da2518abf936035ff P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:08.163439 23778 tablet_replica.cc:330] T 0d9f168de0d1410da2518abf936035ff P bb31bceba76d466db24b9341c04900c3: stopping tablet replica
W20240502 01:54:08.164067 17673 consensus_peers.cc:485] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5df83f83d2674155afba93c2d992af0a. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:08.165418 23778 raft_consensus.cc:2229] T 0d9f168de0d1410da2518abf936035ff P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:08.167603 23778 raft_consensus.cc:2258] T 0d9f168de0d1410da2518abf936035ff P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:08.169894 23778 ts_tablet_manager.cc:1875] T 0d9f168de0d1410da2518abf936035ff P bb31bceba76d466db24b9341c04900c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:08.170948 20231 raft_consensus.cc:1261] T 246cec994d274786aad7672f2a30e5d8 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:08.171397 23343 consensus_queue.cc:1035] T 246cec994d274786aad7672f2a30e5d8 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:08.173262 17864 raft_consensus.cc:1261] T 246cec994d274786aad7672f2a30e5d8 P bb31bceba76d466db24b9341c04900c3 [term 1 LEARNER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:08.173372 23687 tablet_copy_client.cc:783] T 18549019e31b47d899d011cef0637221 P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:08.173702 20347 tablet_service.cc:1514] Processing DeleteTablet for tablet d10d76ac7b094fe9b230b1afcbed9ca6 with delete_type TABLET_DATA_TOMBSTONED (TS 6a36c952c51542c0bf7ea3ec2857a9f9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40196
I20240502 01:54:08.173921 23694 consensus_queue.cc:1035] T 246cec994d274786aad7672f2a30e5d8 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:08.173564 17734 raft_consensus.cc:1261] T 246cec994d274786aad7672f2a30e5d8 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:08.174772 23347 consensus_queue.cc:1035] T 246cec994d274786aad7672f2a30e5d8 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:08.187865 23765 ts_tablet_manager.cc:1888] T 3ff84ce60ac1438d92b2c944bb27aabb P 2f306549407b4bb281ccf2994035e4f1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:54:08.188871 23739 consensus_queue.cc:1035] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
I20240502 01:54:08.192806 17737 raft_consensus.cc:2932] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:08.200623 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 81f335c023444796ab933e19d809cc03 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:08.201469 23731 consensus_queue.cc:1035] T 81f335c023444796ab933e19d809cc03 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.038s
I20240502 01:54:08.203081 20231 raft_consensus.cc:2932] T 81f335c023444796ab933e19d809cc03 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:08.207397 23687 tablet_copy_client.cc:647] T 18549019e31b47d899d011cef0637221 P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:08.213368 20629 raft_consensus.cc:2932] T 5df83f83d2674155afba93c2d992af0a P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:54:08.217099 23727 ts_tablet_manager.cc:1888] T 28463d45bfde44429d56179c1eec56b0 P f19cc6cb1e76460aa6711e49a5775091: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:54:08.217238 23727 log.cc:1196] T 28463d45bfde44429d56179c1eec56b0 P f19cc6cb1e76460aa6711e49a5775091: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal/wals/28463d45bfde44429d56179c1eec56b0
I20240502 01:54:08.217895 17321 catalog_manager.cc:4830] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): tablet 28463d45bfde44429d56179c1eec56b0 (table rebalancer_extra_table_30 [id=9c8c7af6d77740e1a130a1e68baee62d]) successfully deleted
I20240502 01:54:08.218052 23727 tablet_replica.cc:330] T fac0ba26fabd4643bc38b7559faa471c P f19cc6cb1e76460aa6711e49a5775091: stopping tablet replica
I20240502 01:54:08.218266 23727 raft_consensus.cc:2229] T fac0ba26fabd4643bc38b7559faa471c P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:08.218484 23727 raft_consensus.cc:2258] T fac0ba26fabd4643bc38b7559faa471c P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:08.219378 23727 ts_tablet_manager.cc:1875] T fac0ba26fabd4643bc38b7559faa471c P f19cc6cb1e76460aa6711e49a5775091: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:08.222483 20365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbf05e3b3ad44354be7600c3c07756fd" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:54:08.223121 17542 leader_election.cc:302] T bbf05e3b3ad44354be7600c3c07756fd P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29; no voters: 6a36c952c51542c0bf7ea3ec2857a9f9, fdc2ac372492454c85387589a91f63af
I20240502 01:54:08.223589 23766 raft_consensus.cc:2726] T bbf05e3b3ad44354be7600c3c07756fd P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:54:08.226202 23765 log.cc:1196] T 3ff84ce60ac1438d92b2c944bb27aabb P 2f306549407b4bb281ccf2994035e4f1: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal/wals/3ff84ce60ac1438d92b2c944bb27aabb
I20240502 01:54:08.196446 20345 tablet_service.cc:1514] Processing DeleteTablet for tablet bfd7b1d1002c4b57a14d43ed9c6d9f50 with delete_type TABLET_DATA_TOMBSTONED (TS 6a36c952c51542c0bf7ea3ec2857a9f9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40196
I20240502 01:54:08.227150 17322 catalog_manager.cc:4830] TS 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): tablet 3ff84ce60ac1438d92b2c944bb27aabb (table rebalancer_extra_table_16 [id=61898cfacac64133a52ca6484799789a]) successfully deleted
I20240502 01:54:08.229561 23765 tablet_replica.cc:330] T 81f335c023444796ab933e19d809cc03 P 2f306549407b4bb281ccf2994035e4f1: stopping tablet replica
I20240502 01:54:08.229794 23765 raft_consensus.cc:2229] T 81f335c023444796ab933e19d809cc03 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:08.230016 23765 raft_consensus.cc:2258] T 81f335c023444796ab933e19d809cc03 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:08.235168 17328 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:54:08.243337 23694 raft_consensus.cc:2932] T 246cec994d274786aad7672f2a30e5d8 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:08.248340 23687 tablet_copy_client.cc:523] T 18549019e31b47d899d011cef0637221 P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:08.251034 20231 raft_consensus.cc:2932] T 246cec994d274786aad7672f2a30e5d8 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:08.256392 17326 catalog_manager.cc:5484] T 246cec994d274786aad7672f2a30e5d8 P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: config changed from index 2 to 3, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:08.260257 17734 raft_consensus.cc:2932] T 246cec994d274786aad7672f2a30e5d8 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:08.269111 23775 tablet.cc:1722] T 0f953dbfbe0b4e93880c5665003395e0 P 6a36c952c51542c0bf7ea3ec2857a9f9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:08.269937 23775 tablet.cc:1620] T 0f953dbfbe0b4e93880c5665003395e0 P 6a36c952c51542c0bf7ea3ec2857a9f9: MemRowSet was empty: no flush needed.
I20240502 01:54:08.270548 20494 consensus_queue.cc:237] T 2ea539ebb37f42f68c166242a06f8d2e P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } }
I20240502 01:54:08.271261 22991 consensus_queue.cc:1035] T d25f2ab1a7544cec893dd02c7c89285c P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20240502 01:54:08.289634 23687 tablet_bootstrap.cc:492] T 18549019e31b47d899d011cef0637221 P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:54:08.299152 22991 consensus_queue.cc:1035] T d25f2ab1a7544cec893dd02c7c89285c P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:08.300767 23786 tablet_replica.cc:330] T d10d76ac7b094fe9b230b1afcbed9ca6 P 6a36c952c51542c0bf7ea3ec2857a9f9: stopping tablet replica
I20240502 01:54:08.301673 23776 tablet.cc:1722] T 0f953dbfbe0b4e93880c5665003395e0 P fdc2ac372492454c85387589a91f63af: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:08.302186 23776 tablet.cc:1620] T 0f953dbfbe0b4e93880c5665003395e0 P fdc2ac372492454c85387589a91f63af: MemRowSet was empty: no flush needed.
I20240502 01:54:08.309610 23727 ts_tablet_manager.cc:1888] T fac0ba26fabd4643bc38b7559faa471c P f19cc6cb1e76460aa6711e49a5775091: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:54:08.309744 23727 log.cc:1196] T fac0ba26fabd4643bc38b7559faa471c P f19cc6cb1e76460aa6711e49a5775091: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal/wals/fac0ba26fabd4643bc38b7559faa471c
I20240502 01:54:08.316344 17328 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=5b37c5f6d0cc4a34b264799964d1ffe9] alter complete (version 2)
I20240502 01:54:08.321466 23775 tablet.cc:1722] T 0f953dbfbe0b4e93880c5665003395e0 P 6a36c952c51542c0bf7ea3ec2857a9f9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:08.326917 17334 catalog_manager.cc:5576] TS 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155) does not have the latest schema for tablet 0f953dbfbe0b4e93880c5665003395e0 (table rebalancer_dynamic_table [id=5b37c5f6d0cc4a34b264799964d1ffe9]). Expected version 2 got 1
I20240502 01:54:08.329214 23775 tablet.cc:1620] T 0f953dbfbe0b4e93880c5665003395e0 P 6a36c952c51542c0bf7ea3ec2857a9f9: MemRowSet was empty: no flush needed.
I20240502 01:54:08.330802 23765 ts_tablet_manager.cc:1875] T 81f335c023444796ab933e19d809cc03 P 2f306549407b4bb281ccf2994035e4f1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:08.338336 23687 tablet_bootstrap.cc:492] T 18549019e31b47d899d011cef0637221 P f19cc6cb1e76460aa6711e49a5775091: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:08.338491 23786 raft_consensus.cc:2229] T d10d76ac7b094fe9b230b1afcbed9ca6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:08.338786 23786 raft_consensus.cc:2258] T d10d76ac7b094fe9b230b1afcbed9ca6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:08.339226 23687 tablet_bootstrap.cc:492] T 18549019e31b47d899d011cef0637221 P f19cc6cb1e76460aa6711e49a5775091: Bootstrap complete.
I20240502 01:54:08.339540 23687 ts_tablet_manager.cc:1367] T 18549019e31b47d899d011cef0637221 P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.059s user 0.011s sys 0.001s
I20240502 01:54:08.344208 23786 ts_tablet_manager.cc:1875] T d10d76ac7b094fe9b230b1afcbed9ca6 P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:08.349480 20626 raft_consensus.cc:1261] T 2ea539ebb37f42f68c166242a06f8d2e P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:08.350068 23694 consensus_queue.cc:1035] T 2ea539ebb37f42f68c166242a06f8d2e P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:08.350836 17476 raft_consensus.cc:1261] T 2ea539ebb37f42f68c166242a06f8d2e P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:08.351378 23694 consensus_queue.cc:1035] T 2ea539ebb37f42f68c166242a06f8d2e P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:54:08.352460 20434 consensus_peers.cc:485] T 2ea539ebb37f42f68c166242a06f8d2e P 2f306549407b4bb281ccf2994035e4f1 -> Peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): Couldn't send request to peer bb31bceba76d466db24b9341c04900c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2ea539ebb37f42f68c166242a06f8d2e. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:08.362123 23778 ts_tablet_manager.cc:1888] T 0d9f168de0d1410da2518abf936035ff P bb31bceba76d466db24b9341c04900c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:54:08.362252 23778 log.cc:1196] T 0d9f168de0d1410da2518abf936035ff P bb31bceba76d466db24b9341c04900c3: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/wals/0d9f168de0d1410da2518abf936035ff
I20240502 01:54:08.362636 23717 tablet_bootstrap.cc:492] T 377f8b554ad1448d981d2283b865f8fe P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:08.363027 23717 tablet_bootstrap.cc:492] T 377f8b554ad1448d981d2283b865f8fe P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap complete.
I20240502 01:54:08.363207 17322 catalog_manager.cc:4830] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): tablet 0d9f168de0d1410da2518abf936035ff (table rebalance_test_table_9 [id=ffe6d29170f24ad48433354f548b9b2a]) successfully deleted
I20240502 01:54:08.363298 23717 ts_tablet_manager.cc:1367] T 377f8b554ad1448d981d2283b865f8fe P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.662s user 0.033s sys 0.020s
I20240502 01:54:08.364087 23717 raft_consensus.cc:348] T 377f8b554ad1448d981d2283b865f8fe P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } }
I20240502 01:54:08.364491 23717 raft_consensus.cc:729] T 377f8b554ad1448d981d2283b865f8fe P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: LEARNER
I20240502 01:54:08.364723 23717 consensus_queue.cc:260] T 377f8b554ad1448d981d2283b865f8fe P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } }
I20240502 01:54:08.368382 17334 catalog_manager.cc:5576] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083) does not have the latest schema for tablet 0f953dbfbe0b4e93880c5665003395e0 (table rebalancer_dynamic_table [id=5b37c5f6d0cc4a34b264799964d1ffe9]). Expected version 2 got 0
I20240502 01:54:08.371388 23793 tablet.cc:1722] T 0f953dbfbe0b4e93880c5665003395e0 P bb31bceba76d466db24b9341c04900c3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:08.374033 23793 tablet.cc:1620] T 0f953dbfbe0b4e93880c5665003395e0 P bb31bceba76d466db24b9341c04900c3: MemRowSet was empty: no flush needed.
I20240502 01:54:08.379362 23765 ts_tablet_manager.cc:1888] T 81f335c023444796ab933e19d809cc03 P 2f306549407b4bb281ccf2994035e4f1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:54:08.379493 23765 log.cc:1196] T 81f335c023444796ab933e19d809cc03 P 2f306549407b4bb281ccf2994035e4f1: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal/wals/81f335c023444796ab933e19d809cc03
I20240502 01:54:08.380393 23717 ts_tablet_manager.cc:1398] T 377f8b554ad1448d981d2283b865f8fe P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.017s user 0.004s sys 0.000s
I20240502 01:54:08.381296 17322 catalog_manager.cc:4830] TS 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): tablet 81f335c023444796ab933e19d809cc03 (table rebalancer_extra_table_18 [id=448fbe791a344997b564da10a31bc5d2]) successfully deleted
I20240502 01:54:08.388185 20494 consensus_queue.cc:237] T 246cec994d274786aad7672f2a30e5d8 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } }
I20240502 01:54:08.388940 17865 raft_consensus.cc:2932] T 246cec994d274786aad7672f2a30e5d8 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:08.392694 17734 raft_consensus.cc:1261] T 246cec994d274786aad7672f2a30e5d8 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:08.393286 23343 consensus_queue.cc:1035] T 246cec994d274786aad7672f2a30e5d8 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:08.395851 17887 tablet_copy_service.cc:342] P bb31bceba76d466db24b9341c04900c3: Request end of tablet copy session 6a36c952c51542c0bf7ea3ec2857a9f9-377f8b554ad1448d981d2283b865f8fe received from {username='slave'} at 127.11.42.70:41203
I20240502 01:54:08.395975 17887 tablet_copy_service.cc:433] P bb31bceba76d466db24b9341c04900c3: ending tablet copy session 6a36c952c51542c0bf7ea3ec2857a9f9-377f8b554ad1448d981d2283b865f8fe on tablet 377f8b554ad1448d981d2283b865f8fe with peer 6a36c952c51542c0bf7ea3ec2857a9f9
I20240502 01:54:08.395737 23343 raft_consensus.cc:2932] T 246cec994d274786aad7672f2a30e5d8 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:08.399459 17862 raft_consensus.cc:1261] T 246cec994d274786aad7672f2a30e5d8 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:08.400403 17737 raft_consensus.cc:2932] T 246cec994d274786aad7672f2a30e5d8 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:08.401222 23347 consensus_queue.cc:1035] T 246cec994d274786aad7672f2a30e5d8 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:08.402527 17862 raft_consensus.cc:2932] T 246cec994d274786aad7672f2a30e5d8 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:08.405933 17322 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 246cec994d274786aad7672f2a30e5d8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:08.411442 17326 catalog_manager.cc:5484] T 246cec994d274786aad7672f2a30e5d8 P bb31bceba76d466db24b9341c04900c3 reported cstate change: config changed from index 3 to 4, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:08.417541 20212 tablet_service.cc:1514] Processing DeleteTablet for tablet 246cec994d274786aad7672f2a30e5d8 with delete_type TABLET_DATA_TOMBSTONED (TS fdc2ac372492454c85387589a91f63af not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44624
I20240502 01:54:08.417872 23742 tablet_replica.cc:330] T 246cec994d274786aad7672f2a30e5d8 P fdc2ac372492454c85387589a91f63af: stopping tablet replica
I20240502 01:54:08.418079 23742 raft_consensus.cc:2229] T 246cec994d274786aad7672f2a30e5d8 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:08.418308 23742 raft_consensus.cc:2258] T 246cec994d274786aad7672f2a30e5d8 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:08.419310 23742 ts_tablet_manager.cc:1875] T 246cec994d274786aad7672f2a30e5d8 P fdc2ac372492454c85387589a91f63af: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:08.422573 23793 tablet.cc:1722] T 0f953dbfbe0b4e93880c5665003395e0 P bb31bceba76d466db24b9341c04900c3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:08.423871 17333 catalog_manager.cc:5576] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083) does not have the latest schema for tablet 0f953dbfbe0b4e93880c5665003395e0 (table rebalancer_dynamic_table [id=5b37c5f6d0cc4a34b264799964d1ffe9]). Expected version 2 got 1
I20240502 01:54:08.425102 23793 tablet.cc:1620] T 0f953dbfbe0b4e93880c5665003395e0 P bb31bceba76d466db24b9341c04900c3: MemRowSet was empty: no flush needed.
I20240502 01:54:08.427744 23742 ts_tablet_manager.cc:1888] T 246cec994d274786aad7672f2a30e5d8 P fdc2ac372492454c85387589a91f63af: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:54:08.427929 23742 log.cc:1196] T 246cec994d274786aad7672f2a30e5d8 P fdc2ac372492454c85387589a91f63af: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal/wals/246cec994d274786aad7672f2a30e5d8
I20240502 01:54:08.428916 17322 catalog_manager.cc:4830] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): tablet 246cec994d274786aad7672f2a30e5d8 (table rebalancer_extra_table_18 [id=448fbe791a344997b564da10a31bc5d2]) successfully deleted
I20240502 01:54:08.429291 17334 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:54:08.436468 23347 raft_consensus.cc:2932] T 2ea539ebb37f42f68c166242a06f8d2e P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } } }
I20240502 01:54:08.437366 23786 ts_tablet_manager.cc:1888] T d10d76ac7b094fe9b230b1afcbed9ca6 P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:54:08.437878 23775 tablet.cc:1722] T 0f953dbfbe0b4e93880c5665003395e0 P 6a36c952c51542c0bf7ea3ec2857a9f9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:08.438719 23776 tablet.cc:1722] T 0f953dbfbe0b4e93880c5665003395e0 P fdc2ac372492454c85387589a91f63af: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:08.443192 17328 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_extra_table_40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:08.448053 17326 catalog_manager.cc:5576] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083) does not have the latest schema for tablet 0f953dbfbe0b4e93880c5665003395e0 (table rebalancer_dynamic_table_new_name [id=5b37c5f6d0cc4a34b264799964d1ffe9]). Expected version 3 got 2
I20240502 01:54:08.449560 23802 tablet.cc:1722] T 0f953dbfbe0b4e93880c5665003395e0 P bb31bceba76d466db24b9341c04900c3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:08.455054 17476 raft_consensus.cc:2932] T 2ea539ebb37f42f68c166242a06f8d2e P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } } }
I20240502 01:54:08.455281 17326 catalog_manager.cc:5484] T 2ea539ebb37f42f68c166242a06f8d2e P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: config changed from index -1 to 2, NON_VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) added. New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:08.456456 17333 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table_new_name [id=5b37c5f6d0cc4a34b264799964d1ffe9] alter complete (version 3)
I20240502 01:54:08.461287 17585 tablet_service.cc:1467] Processing CreateTablet for tablet a86548104dca43bd9e1b3ec4d23ec0e4 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=04407f9191884214866d42214a1a1beb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:08.461896 20476 tablet_service.cc:1467] Processing CreateTablet for tablet a86548104dca43bd9e1b3ec4d23ec0e4 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=04407f9191884214866d42214a1a1beb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:08.462325 20476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a86548104dca43bd9e1b3ec4d23ec0e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:08.465219 17585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a86548104dca43bd9e1b3ec4d23ec0e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:08.465859 17716 tablet_service.cc:1467] Processing CreateTablet for tablet a86548104dca43bd9e1b3ec4d23ec0e4 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=04407f9191884214866d42214a1a1beb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:08.466310 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a86548104dca43bd9e1b3ec4d23ec0e4. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 01:54:08.468152 20565 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.42.68:44083, user_credentials={real_user=slave}} blocked reactor thread for 76024.1us
I20240502 01:54:08.469326 17715 tablet_service.cc:1467] Processing CreateTablet for tablet 38bb8dd915f743f49b26ee2572871375 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=04407f9191884214866d42214a1a1beb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:08.469333 20610 tablet_service.cc:1467] Processing CreateTablet for tablet 38bb8dd915f743f49b26ee2572871375 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=04407f9191884214866d42214a1a1beb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:08.469659 17715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38bb8dd915f743f49b26ee2572871375. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:08.470312 20609 tablet_service.cc:1467] Processing CreateTablet for tablet 6ded7434afab44999620dfa3213c7a82 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=04407f9191884214866d42214a1a1beb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
W20240502 01:54:08.471640 20564 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.42.65:38321, user_credentials={real_user=slave}} blocked reactor thread for 72892.4us
I20240502 01:54:08.473906 17321 catalog_manager.cc:4830] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): tablet fac0ba26fabd4643bc38b7559faa471c (table rebalancer_extra_table_16 [id=61898cfacac64133a52ca6484799789a]) successfully deleted
I20240502 01:54:08.474037 23673 tablet_bootstrap.cc:492] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:54:08.475445 23673 tablet_bootstrap.cc:654] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:08.488255 20477 tablet_service.cc:1467] Processing CreateTablet for tablet 6ded7434afab44999620dfa3213c7a82 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=04407f9191884214866d42214a1a1beb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:08.488655 20477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ded7434afab44999620dfa3213c7a82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:08.488255 20478 tablet_service.cc:1467] Processing CreateTablet for tablet 38bb8dd915f743f49b26ee2572871375 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=04407f9191884214866d42214a1a1beb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:08.488987 20478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38bb8dd915f743f49b26ee2572871375. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:08.489063 20610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38bb8dd915f743f49b26ee2572871375. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:08.489336 23687 raft_consensus.cc:348] T 18549019e31b47d899d011cef0637221 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:54:08.492010 20609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ded7434afab44999620dfa3213c7a82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:08.493072 20629 raft_consensus.cc:2932] T 2ea539ebb37f42f68c166242a06f8d2e P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } } }
I20240502 01:54:08.494179 17585 tablet_service.cc:1467] Processing CreateTablet for tablet 6ded7434afab44999620dfa3213c7a82 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=04407f9191884214866d42214a1a1beb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:08.494551 17585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ded7434afab44999620dfa3213c7a82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:08.524016 23687 raft_consensus.cc:729] T 18549019e31b47d899d011cef0637221 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: LEARNER
I20240502 01:54:08.524284 23687 consensus_queue.cc:260] T 18549019e31b47d899d011cef0637221 P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:54:08.524962 23687 ts_tablet_manager.cc:1398] T 18549019e31b47d899d011cef0637221 P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.181s user 0.001s sys 0.001s
I20240502 01:54:08.528028 23809 tablet_bootstrap.cc:492] T 6ded7434afab44999620dfa3213c7a82 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:54:08.530695 23809 tablet_bootstrap.cc:654] T 6ded7434afab44999620dfa3213c7a82 P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:08.537819 23811 tablet_bootstrap.cc:492] T a86548104dca43bd9e1b3ec4d23ec0e4 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:54:08.539208 23811 tablet_bootstrap.cc:654] T a86548104dca43bd9e1b3ec4d23ec0e4 P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:08.541144 23808 tablet_bootstrap.cc:492] T 38bb8dd915f743f49b26ee2572871375 P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:54:08.542937 23808 tablet_bootstrap.cc:654] T 38bb8dd915f743f49b26ee2572871375 P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:08.550223 23809 tablet_bootstrap.cc:492] T 6ded7434afab44999620dfa3213c7a82 P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:54:08.551357 23809 ts_tablet_manager.cc:1367] T 6ded7434afab44999620dfa3213c7a82 P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.004s
I20240502 01:54:08.557093 20387 tablet_copy_service.cc:342] P 6a36c952c51542c0bf7ea3ec2857a9f9: Request end of tablet copy session f19cc6cb1e76460aa6711e49a5775091-18549019e31b47d899d011cef0637221 received from {username='slave'} at 127.11.42.72:36559
I20240502 01:54:08.557221 20387 tablet_copy_service.cc:433] P 6a36c952c51542c0bf7ea3ec2857a9f9: ending tablet copy session f19cc6cb1e76460aa6711e49a5775091-18549019e31b47d899d011cef0637221 on tablet 18549019e31b47d899d011cef0637221 with peer f19cc6cb1e76460aa6711e49a5775091
I20240502 01:54:08.555469 17333 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:08.558880 17333 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:08.562672 17333 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet 0f953dbfbe0b4e93880c5665003395e0
I20240502 01:54:08.563460 20347 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f953dbfbe0b4e93880c5665003395e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:08 UTC) from {username='slave'} at 127.0.0.1:40196
I20240502 01:54:08.564793 17846 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f953dbfbe0b4e93880c5665003395e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:08 UTC) from {username='slave'} at 127.0.0.1:47226
I20240502 01:54:08.564889 20212 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f953dbfbe0b4e93880c5665003395e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:08 UTC) from {username='slave'} at 127.0.0.1:44624
I20240502 01:54:08.565119 23778 tablet_replica.cc:330] T 0f953dbfbe0b4e93880c5665003395e0 P bb31bceba76d466db24b9341c04900c3: stopping tablet replica
I20240502 01:54:08.565176 23742 tablet_replica.cc:330] T 0f953dbfbe0b4e93880c5665003395e0 P fdc2ac372492454c85387589a91f63af: stopping tablet replica
I20240502 01:54:08.565333 23778 raft_consensus.cc:2229] T 0f953dbfbe0b4e93880c5665003395e0 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:08.565387 23742 raft_consensus.cc:2229] T 0f953dbfbe0b4e93880c5665003395e0 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:08.565572 23778 raft_consensus.cc:2258] T 0f953dbfbe0b4e93880c5665003395e0 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:08.565591 23742 raft_consensus.cc:2258] T 0f953dbfbe0b4e93880c5665003395e0 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:08.566697 23742 ts_tablet_manager.cc:1875] T 0f953dbfbe0b4e93880c5665003395e0 P fdc2ac372492454c85387589a91f63af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:08.566751 23778 ts_tablet_manager.cc:1875] T 0f953dbfbe0b4e93880c5665003395e0 P bb31bceba76d466db24b9341c04900c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:08.572650 23778 ts_tablet_manager.cc:1888] T 0f953dbfbe0b4e93880c5665003395e0 P bb31bceba76d466db24b9341c04900c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:08.572777 23778 log.cc:1196] T 0f953dbfbe0b4e93880c5665003395e0 P bb31bceba76d466db24b9341c04900c3: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/wals/0f953dbfbe0b4e93880c5665003395e0
I20240502 01:54:08.573266 23778 ts_tablet_manager.cc:1909] T 0f953dbfbe0b4e93880c5665003395e0 P bb31bceba76d466db24b9341c04900c3: Deleting consensus metadata
I20240502 01:54:08.569869 23809 raft_consensus.cc:348] T 6ded7434afab44999620dfa3213c7a82 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:08.573688 23809 raft_consensus.cc:374] T 6ded7434afab44999620dfa3213c7a82 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:08.574280 17322 catalog_manager.cc:4830] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): tablet 0f953dbfbe0b4e93880c5665003395e0 (table rebalancer_dynamic_table_new_name [id=5b37c5f6d0cc4a34b264799964d1ffe9]) successfully deleted
I20240502 01:54:08.574577 23809 raft_consensus.cc:729] T 6ded7434afab44999620dfa3213c7a82 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:54:08.575093 23809 consensus_queue.cc:260] T 6ded7434afab44999620dfa3213c7a82 P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:08.575466 23742 ts_tablet_manager.cc:1888] T 0f953dbfbe0b4e93880c5665003395e0 P fdc2ac372492454c85387589a91f63af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:08.575620 23742 log.cc:1196] T 0f953dbfbe0b4e93880c5665003395e0 P fdc2ac372492454c85387589a91f63af: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal/wals/0f953dbfbe0b4e93880c5665003395e0
I20240502 01:54:08.576447 23742 ts_tablet_manager.cc:1909] T 0f953dbfbe0b4e93880c5665003395e0 P fdc2ac372492454c85387589a91f63af: Deleting consensus metadata
I20240502 01:54:08.578583 17322 catalog_manager.cc:4830] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): tablet 0f953dbfbe0b4e93880c5665003395e0 (table rebalancer_dynamic_table_new_name [id=5b37c5f6d0cc4a34b264799964d1ffe9]) successfully deleted
I20240502 01:54:08.437573 23786 log.cc:1196] T d10d76ac7b094fe9b230b1afcbed9ca6 P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal/wals/d10d76ac7b094fe9b230b1afcbed9ca6
I20240502 01:54:08.581661 23809 ts_tablet_manager.cc:1398] T 6ded7434afab44999620dfa3213c7a82 P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.027s user 0.000s sys 0.001s
I20240502 01:54:08.581827 17321 catalog_manager.cc:4830] TS 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): tablet d10d76ac7b094fe9b230b1afcbed9ca6 (table rebalancer_extra_table_20 [id=0e1c9ce53d7c47f7a42d19e60342c557]) successfully deleted
I20240502 01:54:08.583235 23786 tablet_replica.cc:330] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P 6a36c952c51542c0bf7ea3ec2857a9f9: stopping tablet replica
I20240502 01:54:08.583449 23786 raft_consensus.cc:2229] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:08.583695 23786 raft_consensus.cc:2258] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:08.587023 23786 ts_tablet_manager.cc:1875] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:08.589972 23673 tablet_bootstrap.cc:492] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:54:08.590165 23673 ts_tablet_manager.cc:1367] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.116s user 0.005s sys 0.004s
I20240502 01:54:08.590971 23673 raft_consensus.cc:348] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:08.591219 23673 raft_consensus.cc:374] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:08.591586 23673 raft_consensus.cc:729] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:54:08.592228 23808 tablet_bootstrap.cc:492] T 38bb8dd915f743f49b26ee2572871375 P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:54:08.592428 23808 ts_tablet_manager.cc:1367] T 38bb8dd915f743f49b26ee2572871375 P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.051s user 0.000s sys 0.004s
I20240502 01:54:08.593218 23809 tablet_bootstrap.cc:492] T 38bb8dd915f743f49b26ee2572871375 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:54:08.593248 23808 raft_consensus.cc:348] T 38bb8dd915f743f49b26ee2572871375 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:08.593389 23673 consensus_queue.cc:260] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:08.593567 23808 raft_consensus.cc:374] T 38bb8dd915f743f49b26ee2572871375 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:08.593705 23808 raft_consensus.cc:729] T 38bb8dd915f743f49b26ee2572871375 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:54:08.594022 23673 ts_tablet_manager.cc:1398] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:54:08.593991 23808 consensus_queue.cc:260] T 38bb8dd915f743f49b26ee2572871375 P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:08.594607 23808 ts_tablet_manager.cc:1398] T 38bb8dd915f743f49b26ee2572871375 P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:08.599725 23673 tablet_bootstrap.cc:492] T 6ded7434afab44999620dfa3213c7a82 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:54:08.603330 23808 tablet_bootstrap.cc:492] T 6ded7434afab44999620dfa3213c7a82 P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:54:08.604341 23673 tablet_bootstrap.cc:654] T 6ded7434afab44999620dfa3213c7a82 P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:08.605595 23811 tablet_bootstrap.cc:492] T a86548104dca43bd9e1b3ec4d23ec0e4 P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:54:08.608379 23809 tablet_bootstrap.cc:654] T 38bb8dd915f743f49b26ee2572871375 P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:08.608561 23811 ts_tablet_manager.cc:1367] T a86548104dca43bd9e1b3ec4d23ec0e4 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.071s user 0.007s sys 0.000s
I20240502 01:54:08.609468 23811 raft_consensus.cc:348] T a86548104dca43bd9e1b3ec4d23ec0e4 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:08.610850 20366 raft_consensus.cc:1203] T 377f8b554ad1448d981d2283b865f8fe P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 01:54:08.616205 23811 raft_consensus.cc:374] T a86548104dca43bd9e1b3ec4d23ec0e4 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:08.616364 23811 raft_consensus.cc:729] T a86548104dca43bd9e1b3ec4d23ec0e4 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:54:08.616688 23811 consensus_queue.cc:260] T a86548104dca43bd9e1b3ec4d23ec0e4 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:08.618672 23811 ts_tablet_manager.cc:1398] T a86548104dca43bd9e1b3ec4d23ec0e4 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240502 01:54:08.622033 23811 tablet_bootstrap.cc:492] T 38bb8dd915f743f49b26ee2572871375 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:54:08.624982 23786 ts_tablet_manager.cc:1888] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:54:08.625587 23786 log.cc:1196] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal/wals/bfd7b1d1002c4b57a14d43ed9c6d9f50
I20240502 01:54:08.627015 23811 tablet_bootstrap.cc:654] T 38bb8dd915f743f49b26ee2572871375 P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:08.628736 17321 catalog_manager.cc:4830] TS 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): tablet bfd7b1d1002c4b57a14d43ed9c6d9f50 (table rebalancer_extra_table_30 [id=9c8c7af6d77740e1a130a1e68baee62d]) successfully deleted
I20240502 01:54:08.629408 23786 tablet_replica.cc:330] T 0f953dbfbe0b4e93880c5665003395e0 P 6a36c952c51542c0bf7ea3ec2857a9f9: stopping tablet replica
I20240502 01:54:08.629621 23786 raft_consensus.cc:2229] T 0f953dbfbe0b4e93880c5665003395e0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:08.629978 23786 raft_consensus.cc:2258] T 0f953dbfbe0b4e93880c5665003395e0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:08.632330 23786 ts_tablet_manager.cc:1875] T 0f953dbfbe0b4e93880c5665003395e0 P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:08.639735 23673 tablet_bootstrap.cc:492] T 6ded7434afab44999620dfa3213c7a82 P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:54:08.639928 23673 ts_tablet_manager.cc:1367] T 6ded7434afab44999620dfa3213c7a82 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.000s
I20240502 01:54:08.640655 23673 raft_consensus.cc:348] T 6ded7434afab44999620dfa3213c7a82 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:08.640895 23673 raft_consensus.cc:374] T 6ded7434afab44999620dfa3213c7a82 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:08.640990 23673 raft_consensus.cc:729] T 6ded7434afab44999620dfa3213c7a82 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:54:08.641274 23673 consensus_queue.cc:260] T 6ded7434afab44999620dfa3213c7a82 P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:08.642438 23786 ts_tablet_manager.cc:1888] T 0f953dbfbe0b4e93880c5665003395e0 P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:08.642601 23786 log.cc:1196] T 0f953dbfbe0b4e93880c5665003395e0 P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal/wals/0f953dbfbe0b4e93880c5665003395e0
I20240502 01:54:08.643131 23673 ts_tablet_manager.cc:1398] T 6ded7434afab44999620dfa3213c7a82 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:54:08.644578 23815 ts_tablet_manager.cc:902] T 5df83f83d2674155afba93c2d992af0a P fdc2ac372492454c85387589a91f63af: Initiating tablet copy from peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:54:08.645082 23809 tablet_bootstrap.cc:492] T 38bb8dd915f743f49b26ee2572871375 P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:54:08.645243 23809 ts_tablet_manager.cc:1367] T 38bb8dd915f743f49b26ee2572871375 P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.052s user 0.002s sys 0.003s
I20240502 01:54:08.646031 23809 raft_consensus.cc:348] T 38bb8dd915f743f49b26ee2572871375 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:08.646236 23809 raft_consensus.cc:374] T 38bb8dd915f743f49b26ee2572871375 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:08.646332 23809 raft_consensus.cc:729] T 38bb8dd915f743f49b26ee2572871375 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:54:08.646554 23809 consensus_queue.cc:260] T 38bb8dd915f743f49b26ee2572871375 P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:08.647094 23809 ts_tablet_manager.cc:1398] T 38bb8dd915f743f49b26ee2572871375 P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:54:08.647262 23811 tablet_bootstrap.cc:492] T 38bb8dd915f743f49b26ee2572871375 P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:54:08.647399 23809 tablet_bootstrap.cc:492] T a86548104dca43bd9e1b3ec4d23ec0e4 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:54:08.647572 23811 ts_tablet_manager.cc:1367] T 38bb8dd915f743f49b26ee2572871375 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20240502 01:54:08.648663 23811 raft_consensus.cc:348] T 38bb8dd915f743f49b26ee2572871375 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:08.648908 23809 tablet_bootstrap.cc:654] T a86548104dca43bd9e1b3ec4d23ec0e4 P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:08.648914 23811 raft_consensus.cc:374] T 38bb8dd915f743f49b26ee2572871375 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:08.649071 23811 raft_consensus.cc:729] T 38bb8dd915f743f49b26ee2572871375 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:54:08.649358 23811 consensus_queue.cc:260] T 38bb8dd915f743f49b26ee2572871375 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:08.649617 23731 raft_consensus.cc:482] T 38bb8dd915f743f49b26ee2572871375 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:08.656940 23731 raft_consensus.cc:504] T 38bb8dd915f743f49b26ee2572871375 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:08.657485 23811 ts_tablet_manager.cc:1398] T 38bb8dd915f743f49b26ee2572871375 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240502 01:54:08.657682 23731 leader_election.cc:288] T 38bb8dd915f743f49b26ee2572871375 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:54:08.658310 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38bb8dd915f743f49b26ee2572871375" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:54:08.658512 17737 raft_consensus.cc:2454] T 38bb8dd915f743f49b26ee2572871375 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 0.
I20240502 01:54:08.659817 20564 leader_election.cc:302] T 38bb8dd915f743f49b26ee2572871375 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dfa1152987644bd781b46c7e42a7a685, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:54:08.659854 20494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38bb8dd915f743f49b26ee2572871375" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
I20240502 01:54:08.660056 20494 raft_consensus.cc:2454] T 38bb8dd915f743f49b26ee2572871375 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 0.
I20240502 01:54:08.660176 23731 raft_consensus.cc:2781] T 38bb8dd915f743f49b26ee2572871375 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:08.660285 23731 raft_consensus.cc:482] T 38bb8dd915f743f49b26ee2572871375 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:08.660451 23731 raft_consensus.cc:3037] T 38bb8dd915f743f49b26ee2572871375 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:08.662071 23807 raft_consensus.cc:482] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:08.662189 23807 raft_consensus.cc:504] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:08.663002 23807 leader_election.cc:288] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:54:08.663002 23731 raft_consensus.cc:504] T 38bb8dd915f743f49b26ee2572871375 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:08.663755 23808 tablet_bootstrap.cc:654] T 6ded7434afab44999620dfa3213c7a82 P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:08.663993 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a86548104dca43bd9e1b3ec4d23ec0e4" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:54:08.664161 17737 raft_consensus.cc:2454] T a86548104dca43bd9e1b3ec4d23ec0e4 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:54:08.664232 20498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38bb8dd915f743f49b26ee2572871375" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
I20240502 01:54:08.664462 20498 raft_consensus.cc:3037] T 38bb8dd915f743f49b26ee2572871375 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:08.664602 17542 leader_election.cc:302] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:54:08.667598 20498 raft_consensus.cc:2454] T 38bb8dd915f743f49b26ee2572871375 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 1.
I20240502 01:54:08.668190 23786 ts_tablet_manager.cc:1909] T 0f953dbfbe0b4e93880c5665003395e0 P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting consensus metadata
I20240502 01:54:08.668390 20494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a86548104dca43bd9e1b3ec4d23ec0e4" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
I20240502 01:54:08.668794 20565 leader_election.cc:302] T 38bb8dd915f743f49b26ee2572871375 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, f19cc6cb1e76460aa6711e49a5775091; no voters:
W20240502 01:54:08.668850 17543 leader_election.cc:341] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:08.669173 22991 raft_consensus.cc:2781] T 38bb8dd915f743f49b26ee2572871375 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:08.669358 22991 raft_consensus.cc:686] T 38bb8dd915f743f49b26ee2572871375 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Becoming Leader. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Running, Role: LEADER
I20240502 01:54:08.669620 22991 consensus_queue.cc:237] T 38bb8dd915f743f49b26ee2572871375 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:08.670146 23807 raft_consensus.cc:2781] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:08.670234 23807 raft_consensus.cc:482] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:08.670305 23807 raft_consensus.cc:3037] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:08.672677 23807 raft_consensus.cc:504] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:08.674103 20494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a86548104dca43bd9e1b3ec4d23ec0e4" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
I20240502 01:54:08.673466 17326 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20240502 01:54:08.674597 17543 leader_election.cc:341] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:08.674576 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38bb8dd915f743f49b26ee2572871375" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:54:08.674757 17737 raft_consensus.cc:3037] T 38bb8dd915f743f49b26ee2572871375 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:08.675115 23807 leader_election.cc:288] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Requested vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:54:08.676018 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a86548104dca43bd9e1b3ec4d23ec0e4" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:54:08.676191 17735 raft_consensus.cc:3037] T a86548104dca43bd9e1b3ec4d23ec0e4 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:08.678618 17735 raft_consensus.cc:2454] T a86548104dca43bd9e1b3ec4d23ec0e4 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:54:08.679262 17737 raft_consensus.cc:2454] T 38bb8dd915f743f49b26ee2572871375 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 1.
I20240502 01:54:08.680364 23731 leader_election.cc:288] T 38bb8dd915f743f49b26ee2572871375 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 election: Requested vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:54:08.681250 23639 raft_consensus.cc:1050] T 062b9d90fb0444f4be37bf09770c9caf P fdc2ac372492454c85387589a91f63af: attempting to promote NON_VOTER 4da690c0fa13407a85c0689ce8de9ade to VOTER
I20240502 01:54:08.682068 23639 consensus_queue.cc:237] T 062b9d90fb0444f4be37bf09770c9caf P fdc2ac372492454c85387589a91f63af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } }
I20240502 01:54:08.685529 17716 tablet_service.cc:1467] Processing CreateTablet for tablet 62018fa910704cb6a389c53556ca0509 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ec69e0e7508c4ea1970ea90c7ca70f96]), partition=
I20240502 01:54:08.685894 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62018fa910704cb6a389c53556ca0509. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:08.686430 17542 leader_election.cc:302] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, dfa1152987644bd781b46c7e42a7a685; no voters: 2f306549407b4bb281ccf2994035e4f1
I20240502 01:54:08.686981 23766 raft_consensus.cc:2781] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:08.687090 23766 raft_consensus.cc:686] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Becoming Leader. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Running, Role: LEADER
I20240502 01:54:08.687295 17604 consensus_queue.cc:237] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:54:08.687343 23766 consensus_queue.cc:237] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:08.690157 17456 tablet_service.cc:1467] Processing CreateTablet for tablet 62018fa910704cb6a389c53556ca0509 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ec69e0e7508c4ea1970ea90c7ca70f96]), partition=
I20240502 01:54:08.690549 17456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62018fa910704cb6a389c53556ca0509. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:08.693361 20610 tablet_service.cc:1467] Processing CreateTablet for tablet 62018fa910704cb6a389c53556ca0509 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ec69e0e7508c4ea1970ea90c7ca70f96]), partition=
I20240502 01:54:08.693691 20610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62018fa910704cb6a389c53556ca0509. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:08.696249 17476 raft_consensus.cc:1261] T 062b9d90fb0444f4be37bf09770c9caf P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEARNER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:08.696790 23638 consensus_queue.cc:1035] T 062b9d90fb0444f4be37bf09770c9caf P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:08.699179 17321 catalog_manager.cc:4830] TS 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): tablet 0f953dbfbe0b4e93880c5665003395e0 (table rebalancer_dynamic_table_new_name [id=5b37c5f6d0cc4a34b264799964d1ffe9]) successfully deleted
I20240502 01:54:08.702737 20494 raft_consensus.cc:1261] T 062b9d90fb0444f4be37bf09770c9caf P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:08.706360 23638 consensus_queue.cc:1035] T 062b9d90fb0444f4be37bf09770c9caf P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:08.721616 17865 raft_consensus.cc:1261] T 062b9d90fb0444f4be37bf09770c9caf P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:08.722929 23639 consensus_queue.cc:1035] T 062b9d90fb0444f4be37bf09770c9caf P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:08.723919 23811 tablet_bootstrap.cc:492] T 62018fa910704cb6a389c53556ca0509 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:54:08.731251 23811 tablet_bootstrap.cc:654] T 62018fa910704cb6a389c53556ca0509 P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:08.736491 23811 tablet_bootstrap.cc:492] T 62018fa910704cb6a389c53556ca0509 P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:54:08.736680 23811 ts_tablet_manager.cc:1367] T 62018fa910704cb6a389c53556ca0509 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20240502 01:54:08.737980 23811 raft_consensus.cc:348] T 62018fa910704cb6a389c53556ca0509 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:08.738870 23808 tablet_bootstrap.cc:492] T 6ded7434afab44999620dfa3213c7a82 P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:54:08.739104 23808 ts_tablet_manager.cc:1367] T 6ded7434afab44999620dfa3213c7a82 P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.136s user 0.004s sys 0.001s
I20240502 01:54:08.740141 23808 raft_consensus.cc:348] T 6ded7434afab44999620dfa3213c7a82 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:08.740461 23808 raft_consensus.cc:374] T 6ded7434afab44999620dfa3213c7a82 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:08.741366 23808 raft_consensus.cc:729] T 6ded7434afab44999620dfa3213c7a82 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:54:08.743124 23808 consensus_queue.cc:260] T 6ded7434afab44999620dfa3213c7a82 P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:08.747022 23638 raft_consensus.cc:2932] T 062b9d90fb0444f4be37bf09770c9caf P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:08.754976 20494 raft_consensus.cc:2932] T 062b9d90fb0444f4be37bf09770c9caf P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:08.764348 23833 tablet_bootstrap.cc:492] T 62018fa910704cb6a389c53556ca0509 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:54:08.766152 23833 tablet_bootstrap.cc:654] T 62018fa910704cb6a389c53556ca0509 P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:08.767855 17333 catalog_manager.cc:5484] T 062b9d90fb0444f4be37bf09770c9caf P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fdc2ac372492454c85387589a91f63af" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
W20240502 01:54:08.770205 17542 consensus_peers.cc:485] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:08.774897 20629 raft_consensus.cc:1261] T 4aefc61cb95f4e949e7a23a1c677749e P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:08.775436 23766 consensus_queue.cc:1035] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:08.777472 17334 catalog_manager.cc:5484] T 38bb8dd915f743f49b26ee2572871375 P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: term changed from 0 to 1, leader changed from <none> to f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72). New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:08.779639 17328 catalog_manager.cc:5484] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66). New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:08.781505 17865 raft_consensus.cc:1261] T 4aefc61cb95f4e949e7a23a1c677749e P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:08.785477 23766 consensus_queue.cc:1035] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:08.794175 23833 tablet_bootstrap.cc:492] T 62018fa910704cb6a389c53556ca0509 P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:54:08.794397 23833 ts_tablet_manager.cc:1367] T 62018fa910704cb6a389c53556ca0509 P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20240502 01:54:08.795415 23833 raft_consensus.cc:348] T 62018fa910704cb6a389c53556ca0509 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:08.795697 23833 raft_consensus.cc:374] T 62018fa910704cb6a389c53556ca0509 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:08.795799 23833 raft_consensus.cc:729] T 62018fa910704cb6a389c53556ca0509 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:54:08.795974 23807 raft_consensus.cc:2932] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:08.796707 17865 raft_consensus.cc:2932] T 4aefc61cb95f4e949e7a23a1c677749e P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:08.798156 23815 tablet_copy_client.cc:308] T 5df83f83d2674155afba93c2d992af0a P fdc2ac372492454c85387589a91f63af: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.67:36115
I20240502 01:54:08.802970 17862 raft_consensus.cc:2932] T 062b9d90fb0444f4be37bf09770c9caf P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:08.810694 17334 catalog_manager.cc:5484] T 4aefc61cb95f4e949e7a23a1c677749e P bb31bceba76d466db24b9341c04900c3 reported cstate change: config changed from index 6 to 7, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:08.827359 23833 consensus_queue.cc:260] T 62018fa910704cb6a389c53556ca0509 P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:08.828649 23833 ts_tablet_manager.cc:1398] T 62018fa910704cb6a389c53556ca0509 P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.034s user 0.002s sys 0.000s
I20240502 01:54:08.830282 17474 raft_consensus.cc:2932] T 062b9d90fb0444f4be37bf09770c9caf P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:08.839903 20231 consensus_queue.cc:237] T 062b9d90fb0444f4be37bf09770c9caf P fdc2ac372492454c85387589a91f63af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } }
I20240502 01:54:08.841868 17474 raft_consensus.cc:1261] T 062b9d90fb0444f4be37bf09770c9caf P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:08.844413 23638 consensus_queue.cc:1035] T 062b9d90fb0444f4be37bf09770c9caf P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:08.845580 17862 raft_consensus.cc:1261] T 062b9d90fb0444f4be37bf09770c9caf P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:08.850059 23639 consensus_queue.cc:1035] T 062b9d90fb0444f4be37bf09770c9caf P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:08.854598 17862 raft_consensus.cc:2932] T 062b9d90fb0444f4be37bf09770c9caf P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:08.855031 23808 ts_tablet_manager.cc:1398] T 6ded7434afab44999620dfa3213c7a82 P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:54:08.869683 20628 raft_consensus.cc:1203] T 18549019e31b47d899d011cef0637221 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:08.870720 17473 raft_consensus.cc:2932] T 062b9d90fb0444f4be37bf09770c9caf P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:08.879722 17334 catalog_manager.cc:5484] T 062b9d90fb0444f4be37bf09770c9caf P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: config changed from index 3 to 4, VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) evicted. New cstate: current_term: 1 leader_uuid: "fdc2ac372492454c85387589a91f63af" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:08.886607 20477 tablet_service.cc:1514] Processing DeleteTablet for tablet 062b9d90fb0444f4be37bf09770c9caf with delete_type TABLET_DATA_TOMBSTONED (TS 2f306549407b4bb281ccf2994035e4f1 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35018
I20240502 01:54:08.903868 23847 tablet_replica.cc:330] T 062b9d90fb0444f4be37bf09770c9caf P 2f306549407b4bb281ccf2994035e4f1: stopping tablet replica
I20240502 01:54:08.904260 23847 raft_consensus.cc:2229] T 062b9d90fb0444f4be37bf09770c9caf P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:08.904668 23847 raft_consensus.cc:2258] T 062b9d90fb0444f4be37bf09770c9caf P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:08.904955 23808 tablet_bootstrap.cc:492] T 62018fa910704cb6a389c53556ca0509 P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:54:08.905691 23847 ts_tablet_manager.cc:1875] T 062b9d90fb0444f4be37bf09770c9caf P 2f306549407b4bb281ccf2994035e4f1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:08.799245 17757 tablet_copy_service.cc:140] P dfa1152987644bd781b46c7e42a7a685: Received BeginTabletCopySession request for tablet 5df83f83d2674155afba93c2d992af0a from peer fdc2ac372492454c85387589a91f63af ({username='slave'} at 127.11.42.69:44021)
I20240502 01:54:08.907680 17757 tablet_copy_service.cc:161] P dfa1152987644bd781b46c7e42a7a685: Beginning new tablet copy session on tablet 5df83f83d2674155afba93c2d992af0a from peer fdc2ac372492454c85387589a91f63af at {username='slave'} at 127.11.42.69:44021: session id = fdc2ac372492454c85387589a91f63af-5df83f83d2674155afba93c2d992af0a
I20240502 01:54:08.910630 17757 tablet_copy_source_session.cc:205] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:08.912137 23815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5df83f83d2674155afba93c2d992af0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:08.913216 23811 raft_consensus.cc:374] T 62018fa910704cb6a389c53556ca0509 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:08.913333 23811 raft_consensus.cc:729] T 62018fa910704cb6a389c53556ca0509 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:54:08.914801 23809 tablet_bootstrap.cc:492] T a86548104dca43bd9e1b3ec4d23ec0e4 P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:54:08.915539 23343 raft_consensus.cc:482] T 6ded7434afab44999620dfa3213c7a82 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:08.915695 23343 raft_consensus.cc:504] T 6ded7434afab44999620dfa3213c7a82 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:08.915956 23637 raft_consensus.cc:2932] T 062b9d90fb0444f4be37bf09770c9caf P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:08.916440 23343 leader_election.cc:288] T 6ded7434afab44999620dfa3213c7a82 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:54:08.917589 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ded7434afab44999620dfa3213c7a82" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:54:08.917804 17604 raft_consensus.cc:2454] T 6ded7434afab44999620dfa3213c7a82 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 0.
I20240502 01:54:08.917810 20628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ded7434afab44999620dfa3213c7a82" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091" is_pre_election: true
I20240502 01:54:08.918037 20628 raft_consensus.cc:2454] T 6ded7434afab44999620dfa3213c7a82 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 0.
I20240502 01:54:08.918241 20431 leader_election.cc:302] T 6ded7434afab44999620dfa3213c7a82 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 3fceccefba274b1dbbd8216af8b0bc29; no voters:
I20240502 01:54:08.919129 23343 raft_consensus.cc:2781] T 6ded7434afab44999620dfa3213c7a82 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:08.919253 23343 raft_consensus.cc:482] T 6ded7434afab44999620dfa3213c7a82 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:08.919359 23343 raft_consensus.cc:3037] T 6ded7434afab44999620dfa3213c7a82 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:08.920399 23809 ts_tablet_manager.cc:1367] T a86548104dca43bd9e1b3ec4d23ec0e4 P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.268s user 0.049s sys 0.071s
I20240502 01:54:08.921223 23809 raft_consensus.cc:348] T a86548104dca43bd9e1b3ec4d23ec0e4 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:08.921476 23809 raft_consensus.cc:374] T a86548104dca43bd9e1b3ec4d23ec0e4 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:08.921587 23809 raft_consensus.cc:729] T a86548104dca43bd9e1b3ec4d23ec0e4 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:54:08.921835 23809 consensus_queue.cc:260] T a86548104dca43bd9e1b3ec4d23ec0e4 P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:08.922451 23809 ts_tablet_manager.cc:1398] T a86548104dca43bd9e1b3ec4d23ec0e4 P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:08.923187 23343 raft_consensus.cc:504] T 6ded7434afab44999620dfa3213c7a82 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:08.924111 23847 ts_tablet_manager.cc:1888] T 062b9d90fb0444f4be37bf09770c9caf P 2f306549407b4bb281ccf2994035e4f1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:54:08.924276 23847 log.cc:1196] T 062b9d90fb0444f4be37bf09770c9caf P 2f306549407b4bb281ccf2994035e4f1: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal/wals/062b9d90fb0444f4be37bf09770c9caf
I20240502 01:54:08.924867 20628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ded7434afab44999620dfa3213c7a82" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091"
I20240502 01:54:08.924932 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ded7434afab44999620dfa3213c7a82" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:54:08.925110 20628 raft_consensus.cc:3037] T 6ded7434afab44999620dfa3213c7a82 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:08.925124 17604 raft_consensus.cc:3037] T 6ded7434afab44999620dfa3213c7a82 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:08.925241 17322 catalog_manager.cc:4830] TS 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): tablet 062b9d90fb0444f4be37bf09770c9caf (table rebalancer_extra_table_24 [id=6077652d45454150b5c7d7072780d895]) successfully deleted
I20240502 01:54:08.927268 17604 raft_consensus.cc:2454] T 6ded7434afab44999620dfa3213c7a82 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 1.
I20240502 01:54:08.927503 23343 leader_election.cc:288] T 6ded7434afab44999620dfa3213c7a82 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 election: Requested vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:54:08.927718 20431 leader_election.cc:302] T 6ded7434afab44999620dfa3213c7a82 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 3fceccefba274b1dbbd8216af8b0bc29; no voters:
I20240502 01:54:08.927834 20628 raft_consensus.cc:2454] T 6ded7434afab44999620dfa3213c7a82 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 1.
I20240502 01:54:08.928006 23343 raft_consensus.cc:2781] T 6ded7434afab44999620dfa3213c7a82 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:08.928229 23343 raft_consensus.cc:686] T 6ded7434afab44999620dfa3213c7a82 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Running, Role: LEADER
I20240502 01:54:08.928485 23343 consensus_queue.cc:237] T 6ded7434afab44999620dfa3213c7a82 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:08.928989 23815 tablet_copy_client.cc:783] T 5df83f83d2674155afba93c2d992af0a P fdc2ac372492454c85387589a91f63af: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:08.929347 23815 tablet_copy_client.cc:647] T 5df83f83d2674155afba93c2d992af0a P fdc2ac372492454c85387589a91f63af: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:08.931582 23662 raft_consensus.cc:482] T 62018fa910704cb6a389c53556ca0509 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:08.931751 23662 raft_consensus.cc:504] T 62018fa910704cb6a389c53556ca0509 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:08.932873 23662 leader_election.cc:288] T 62018fa910704cb6a389c53556ca0509 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:54:08.934300 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62018fa910704cb6a389c53556ca0509" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:54:08.936385 23815 tablet_copy_client.cc:523] T 5df83f83d2674155afba93c2d992af0a P fdc2ac372492454c85387589a91f63af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:08.938206 20628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62018fa910704cb6a389c53556ca0509" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091" is_pre_election: true
W20240502 01:54:08.939241 17411 leader_election.cc:341] T 62018fa910704cb6a389c53556ca0509 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:08.941776 17862 consensus_queue.cc:237] T 2152c07e448b4582abff22b034363c70 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } }
I20240502 01:54:08.942847 23815 tablet_bootstrap.cc:492] T 5df83f83d2674155afba93c2d992af0a P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:54:08.945740 23842 ts_tablet_manager.cc:902] T 2ea539ebb37f42f68c166242a06f8d2e P bb31bceba76d466db24b9341c04900c3: Initiating tablet copy from peer 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:54:08.946691 23811 consensus_queue.cc:260] T 62018fa910704cb6a389c53556ca0509 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:08.949067 17322 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 062b9d90fb0444f4be37bf09770c9caf with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:08.950379 17737 raft_consensus.cc:2454] T 62018fa910704cb6a389c53556ca0509 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 0.
I20240502 01:54:08.951174 17411 leader_election.cc:302] T 62018fa910704cb6a389c53556ca0509 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, dfa1152987644bd781b46c7e42a7a685; no voters: f19cc6cb1e76460aa6711e49a5775091
I20240502 01:54:08.952513 23662 raft_consensus.cc:2781] T 62018fa910704cb6a389c53556ca0509 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:08.952689 23662 raft_consensus.cc:482] T 62018fa910704cb6a389c53556ca0509 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:08.952776 23662 raft_consensus.cc:3037] T 62018fa910704cb6a389c53556ca0509 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:08.954036 20626 raft_consensus.cc:1261] T 2152c07e448b4582abff22b034363c70 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:08.954633 23586 consensus_queue.cc:1035] T 2152c07e448b4582abff22b034363c70 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:08.955736 23662 raft_consensus.cc:504] T 62018fa910704cb6a389c53556ca0509 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:08.956758 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62018fa910704cb6a389c53556ca0509" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:54:08.956969 17737 raft_consensus.cc:3037] T 62018fa910704cb6a389c53556ca0509 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:08.959817 17737 raft_consensus.cc:2454] T 62018fa910704cb6a389c53556ca0509 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
I20240502 01:54:08.960544 17411 leader_election.cc:302] T 62018fa910704cb6a389c53556ca0509 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:54:08.962801 23662 leader_election.cc:288] T 62018fa910704cb6a389c53556ca0509 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Requested vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:54:08.962977 20366 raft_consensus.cc:1261] T 2152c07e448b4582abff22b034363c70 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:54:08.963697 17802 consensus_peers.cc:485] T 2152c07e448b4582abff22b034363c70 P bb31bceba76d466db24b9341c04900c3 -> Peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Couldn't send request to peer 4da690c0fa13407a85c0689ce8de9ade. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2152c07e448b4582abff22b034363c70. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:08.963866 23586 consensus_queue.cc:1035] T 2152c07e448b4582abff22b034363c70 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:08.963639 23662 raft_consensus.cc:2781] T 62018fa910704cb6a389c53556ca0509 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:08.965588 23662 raft_consensus.cc:686] T 62018fa910704cb6a389c53556ca0509 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Becoming Leader. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Running, Role: LEADER
I20240502 01:54:08.966096 23662 consensus_queue.cc:237] T 62018fa910704cb6a389c53556ca0509 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:08.977871 23586 raft_consensus.cc:2932] T 2152c07e448b4582abff22b034363c70 P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } } }
I20240502 01:54:08.982770 20364 raft_consensus.cc:2932] T 2152c07e448b4582abff22b034363c70 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } } }
I20240502 01:54:08.990129 17334 catalog_manager.cc:5484] T 2152c07e448b4582abff22b034363c70 P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: config changed from index -1 to 2, NON_VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) added. New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } } }
I20240502 01:54:09.008081 23811 ts_tablet_manager.cc:1398] T 62018fa910704cb6a389c53556ca0509 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.271s user 0.002s sys 0.000s
I20240502 01:54:09.016414 23808 tablet_bootstrap.cc:654] T 62018fa910704cb6a389c53556ca0509 P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:09.021390 20629 raft_consensus.cc:2932] T 4aefc61cb95f4e949e7a23a1c677749e P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
W20240502 01:54:09.029980 17411 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.42.67:36115, user_credentials={real_user=slave}} blocked reactor thread for 69479.1us
I20240502 01:54:09.030938 20628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62018fa910704cb6a389c53556ca0509" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091"
I20240502 01:54:09.031456 17333 catalog_manager.cc:5484] T 62018fa910704cb6a389c53556ca0509 P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: term changed from 0 to 1, leader changed from <none> to 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65). New cstate: current_term: 1 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:09.036237 17334 catalog_manager.cc:5484] T 6ded7434afab44999620dfa3213c7a82 P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71). New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } }
W20240502 01:54:09.036806 17411 leader_election.cc:341] T 62018fa910704cb6a389c53556ca0509 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:09.043738 23808 tablet_bootstrap.cc:492] T 62018fa910704cb6a389c53556ca0509 P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:54:09.043946 23808 ts_tablet_manager.cc:1367] T 62018fa910704cb6a389c53556ca0509 P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.139s user 0.000s sys 0.005s
I20240502 01:54:09.045475 23808 raft_consensus.cc:348] T 62018fa910704cb6a389c53556ca0509 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:09.046710 20628 raft_consensus.cc:2932] T 2152c07e448b4582abff22b034363c70 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } } }
I20240502 01:54:09.047798 23808 raft_consensus.cc:374] T 62018fa910704cb6a389c53556ca0509 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:09.047902 23808 raft_consensus.cc:729] T 62018fa910704cb6a389c53556ca0509 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:54:09.048107 23808 consensus_queue.cc:260] T 62018fa910704cb6a389c53556ca0509 P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:09.050159 23808 ts_tablet_manager.cc:1398] T 62018fa910704cb6a389c53556ca0509 P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:54:09.078696 20498 raft_consensus.cc:3037] T a86548104dca43bd9e1b3ec4d23ec0e4 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:09.082269 23766 consensus_queue.cc:1035] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:09.090130 23842 tablet_copy_client.cc:308] T 2ea539ebb37f42f68c166242a06f8d2e P bb31bceba76d466db24b9341c04900c3: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.71:38623
I20240502 01:54:09.090792 20518 tablet_copy_service.cc:140] P 2f306549407b4bb281ccf2994035e4f1: Received BeginTabletCopySession request for tablet 2ea539ebb37f42f68c166242a06f8d2e from peer bb31bceba76d466db24b9341c04900c3 ({username='slave'} at 127.11.42.68:49559)
I20240502 01:54:09.090983 20518 tablet_copy_service.cc:161] P 2f306549407b4bb281ccf2994035e4f1: Beginning new tablet copy session on tablet 2ea539ebb37f42f68c166242a06f8d2e from peer bb31bceba76d466db24b9341c04900c3 at {username='slave'} at 127.11.42.68:49559: session id = bb31bceba76d466db24b9341c04900c3-2ea539ebb37f42f68c166242a06f8d2e
I20240502 01:54:09.094034 20518 tablet_copy_source_session.cc:205] T 2ea539ebb37f42f68c166242a06f8d2e P 2f306549407b4bb281ccf2994035e4f1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:09.095562 23842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ea539ebb37f42f68c166242a06f8d2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.104878 23842 tablet_copy_client.cc:783] T 2ea539ebb37f42f68c166242a06f8d2e P bb31bceba76d466db24b9341c04900c3: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:09.105391 23842 tablet_copy_client.cc:647] T 2ea539ebb37f42f68c166242a06f8d2e P bb31bceba76d466db24b9341c04900c3: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:09.106963 23842 tablet_copy_client.cc:523] T 2ea539ebb37f42f68c166242a06f8d2e P bb31bceba76d466db24b9341c04900c3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:09.109321 23842 tablet_bootstrap.cc:492] T 2ea539ebb37f42f68c166242a06f8d2e P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:54:09.112974 23815 tablet_bootstrap.cc:492] T 5df83f83d2674155afba93c2d992af0a P fdc2ac372492454c85387589a91f63af: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:09.115934 23815 tablet_bootstrap.cc:492] T 5df83f83d2674155afba93c2d992af0a P fdc2ac372492454c85387589a91f63af: Bootstrap complete.
I20240502 01:54:09.116333 23815 ts_tablet_manager.cc:1367] T 5df83f83d2674155afba93c2d992af0a P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.174s user 0.034s sys 0.018s
I20240502 01:54:09.118032 23815 raft_consensus.cc:348] T 5df83f83d2674155afba93c2d992af0a P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } }
I20240502 01:54:09.121388 23815 raft_consensus.cc:729] T 5df83f83d2674155afba93c2d992af0a P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: LEARNER
I20240502 01:54:09.122349 23815 consensus_queue.cc:260] T 5df83f83d2674155afba93c2d992af0a P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } }
I20240502 01:54:09.123878 23815 ts_tablet_manager.cc:1398] T 5df83f83d2674155afba93c2d992af0a P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:54:09.126461 17757 tablet_copy_service.cc:342] P dfa1152987644bd781b46c7e42a7a685: Request end of tablet copy session fdc2ac372492454c85387589a91f63af-5df83f83d2674155afba93c2d992af0a received from {username='slave'} at 127.11.42.69:44021
I20240502 01:54:09.126585 17757 tablet_copy_service.cc:433] P dfa1152987644bd781b46c7e42a7a685: ending tablet copy session fdc2ac372492454c85387589a91f63af-5df83f83d2674155afba93c2d992af0a on tablet 5df83f83d2674155afba93c2d992af0a with peer fdc2ac372492454c85387589a91f63af
I20240502 01:54:09.131378 23842 tablet_bootstrap.cc:492] T 2ea539ebb37f42f68c166242a06f8d2e P bb31bceba76d466db24b9341c04900c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:09.131721 23842 tablet_bootstrap.cc:492] T 2ea539ebb37f42f68c166242a06f8d2e P bb31bceba76d466db24b9341c04900c3: Bootstrap complete.
I20240502 01:54:09.131942 23842 ts_tablet_manager.cc:1367] T 2ea539ebb37f42f68c166242a06f8d2e P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.004s
I20240502 01:54:09.132541 23842 raft_consensus.cc:348] T 2ea539ebb37f42f68c166242a06f8d2e P bb31bceba76d466db24b9341c04900c3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } }
I20240502 01:54:09.132721 23842 raft_consensus.cc:729] T 2ea539ebb37f42f68c166242a06f8d2e P bb31bceba76d466db24b9341c04900c3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: LEARNER
I20240502 01:54:09.132922 23842 consensus_queue.cc:260] T 2ea539ebb37f42f68c166242a06f8d2e P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } }
I20240502 01:54:09.133620 23842 ts_tablet_manager.cc:1398] T 2ea539ebb37f42f68c166242a06f8d2e P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:09.134804 20518 tablet_copy_service.cc:342] P 2f306549407b4bb281ccf2994035e4f1: Request end of tablet copy session bb31bceba76d466db24b9341c04900c3-2ea539ebb37f42f68c166242a06f8d2e received from {username='slave'} at 127.11.42.68:49559
I20240502 01:54:09.134918 20518 tablet_copy_service.cc:433] P 2f306549407b4bb281ccf2994035e4f1: ending tablet copy session bb31bceba76d466db24b9341c04900c3-2ea539ebb37f42f68c166242a06f8d2e on tablet 2ea539ebb37f42f68c166242a06f8d2e with peer bb31bceba76d466db24b9341c04900c3
I20240502 01:54:09.135128 20629 consensus_queue.cc:237] T da06571e020948059fb06f65f26c831d P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
I20240502 01:54:09.135345 23748 consensus_queue.cc:1035] T 38bb8dd915f743f49b26ee2572871375 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:09.139286 17737 raft_consensus.cc:1261] T da06571e020948059fb06f65f26c831d P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:09.140093 23731 consensus_queue.cc:1035] T da06571e020948059fb06f65f26c831d P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:09.141340 20230 raft_consensus.cc:1261] T da06571e020948059fb06f65f26c831d P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:54:09.142083 20565 consensus_peers.cc:485] T da06571e020948059fb06f65f26c831d P f19cc6cb1e76460aa6711e49a5775091 -> Peer 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): Couldn't send request to peer 2f306549407b4bb281ccf2994035e4f1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: da06571e020948059fb06f65f26c831d. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:09.142865 23731 consensus_queue.cc:1035] T da06571e020948059fb06f65f26c831d P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:09.129339 23767 consensus_queue.cc:1035] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.020s
I20240502 01:54:09.152020 23731 consensus_queue.cc:1035] T 38bb8dd915f743f49b26ee2572871375 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:09.152519 23740 raft_consensus.cc:1050] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3: attempting to promote NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 to VOTER
I20240502 01:54:09.153327 23740 consensus_queue.cc:237] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } }
I20240502 01:54:09.155854 20233 raft_consensus.cc:1203] T 5df83f83d2674155afba93c2d992af0a P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 01:54:09.172570 17326 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:54:09.179080 22991 raft_consensus.cc:2932] T da06571e020948059fb06f65f26c831d P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:54:09.182300 17606 raft_consensus.cc:1261] T 377f8b554ad1448d981d2283b865f8fe P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:09.183619 17737 raft_consensus.cc:2932] T da06571e020948059fb06f65f26c831d P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:54:09.185393 20496 raft_consensus.cc:1261] T 377f8b554ad1448d981d2283b865f8fe P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:09.186065 21667 consensus_queue.cc:1035] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:09.186101 20233 raft_consensus.cc:2932] T da06571e020948059fb06f65f26c831d P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:54:09.186636 21667 consensus_queue.cc:1035] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:09.192234 20366 raft_consensus.cc:1261] T 377f8b554ad1448d981d2283b865f8fe P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:09.194973 23586 consensus_queue.cc:1035] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:09.196710 17334 catalog_manager.cc:5484] T da06571e020948059fb06f65f26c831d P fdc2ac372492454c85387589a91f63af reported cstate change: config changed from index -1 to 2, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:54:09.208837 20628 raft_consensus.cc:3037] T 62018fa910704cb6a389c53556ca0509 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:09.211730 20628 raft_consensus.cc:1261] T 62018fa910704cb6a389c53556ca0509 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:09.212591 23852 consensus_queue.cc:1035] T 62018fa910704cb6a389c53556ca0509 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:09.225767 17737 raft_consensus.cc:1261] T 62018fa910704cb6a389c53556ca0509 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:09.226439 23852 consensus_queue.cc:1035] T 62018fa910704cb6a389c53556ca0509 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:09.228536 17862 raft_consensus.cc:1203] T 2ea539ebb37f42f68c166242a06f8d2e P bb31bceba76d466db24b9341c04900c3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:09.255240 21667 raft_consensus.cc:2932] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:54:09.255698 17606 raft_consensus.cc:2932] T 377f8b554ad1448d981d2283b865f8fe P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:54:09.261610 17333 catalog_manager.cc:5576] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115) does not have the latest schema for tablet 62018fa910704cb6a389c53556ca0509 (table rebalancer_dynamic_table [id=ec69e0e7508c4ea1970ea90c7ca70f96]). Expected version 1 got 0
I20240502 01:54:09.264191 20496 raft_consensus.cc:2932] T 377f8b554ad1448d981d2283b865f8fe P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:54:09.275419 17333 catalog_manager.cc:5484] T 377f8b554ad1448d981d2283b865f8fe P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: config changed from index 7 to 8, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:54:09.280172 23895 tablet.cc:1722] T 62018fa910704cb6a389c53556ca0509 P dfa1152987644bd781b46c7e42a7a685: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:09.282344 23895 tablet.cc:1620] T 62018fa910704cb6a389c53556ca0509 P dfa1152987644bd781b46c7e42a7a685: MemRowSet was empty: no flush needed.
I20240502 01:54:09.282955 17326 catalog_manager.cc:5576] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115) does not have the latest schema for tablet 62018fa910704cb6a389c53556ca0509 (table rebalancer_dynamic_table [id=ec69e0e7508c4ea1970ea90c7ca70f96]). Expected version 1 got 0
I20240502 01:54:09.290418 20366 raft_consensus.cc:2932] T 377f8b554ad1448d981d2283b865f8fe P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:54:09.294824 17862 consensus_queue.cc:237] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } }
I20240502 01:54:09.298246 23894 tablet.cc:1722] T 62018fa910704cb6a389c53556ca0509 P 4da690c0fa13407a85c0689ce8de9ade: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:09.298854 23894 tablet.cc:1620] T 62018fa910704cb6a389c53556ca0509 P 4da690c0fa13407a85c0689ce8de9ade: MemRowSet was empty: no flush needed.
I20240502 01:54:09.302371 17328 catalog_manager.cc:5576] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321) does not have the latest schema for tablet 62018fa910704cb6a389c53556ca0509 (table rebalancer_dynamic_table [id=ec69e0e7508c4ea1970ea90c7ca70f96]). Expected version 1 got 0
I20240502 01:54:09.304208 20362 raft_consensus.cc:1261] T 377f8b554ad1448d981d2283b865f8fe P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:09.305935 21667 consensus_queue.cc:1035] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:09.306659 20496 raft_consensus.cc:1261] T 377f8b554ad1448d981d2283b865f8fe P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:09.307475 21667 consensus_queue.cc:1035] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:09.310135 23740 raft_consensus.cc:2932] T 377f8b554ad1448d981d2283b865f8fe P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:54:09.312000 17333 catalog_manager.cc:5576] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411) does not have the latest schema for tablet 62018fa910704cb6a389c53556ca0509 (table rebalancer_dynamic_table [id=ec69e0e7508c4ea1970ea90c7ca70f96]). Expected version 1 got 0
I20240502 01:54:09.314357 17322 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 377f8b554ad1448d981d2283b865f8fe with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:09.316840 20498 raft_consensus.cc:2932] T 377f8b554ad1448d981d2283b865f8fe P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:54:09.322332 23897 tablet.cc:1722] T 62018fa910704cb6a389c53556ca0509 P f19cc6cb1e76460aa6711e49a5775091: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:09.324142 23897 tablet.cc:1620] T 62018fa910704cb6a389c53556ca0509 P f19cc6cb1e76460aa6711e49a5775091: MemRowSet was empty: no flush needed.
I20240502 01:54:09.327736 23347 consensus_queue.cc:1035] T 6ded7434afab44999620dfa3213c7a82 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:09.328545 17326 catalog_manager.cc:5576] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411) does not have the latest schema for tablet 62018fa910704cb6a389c53556ca0509 (table rebalancer_dynamic_table [id=ec69e0e7508c4ea1970ea90c7ca70f96]). Expected version 1 got 0
I20240502 01:54:09.353984 20362 raft_consensus.cc:2932] T 377f8b554ad1448d981d2283b865f8fe P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:54:09.356552 23343 consensus_queue.cc:1035] T 6ded7434afab44999620dfa3213c7a82 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:09.362272 17333 catalog_manager.cc:5484] T 377f8b554ad1448d981d2283b865f8fe P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: config changed from index 8 to 9, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:54:09.364650 17328 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=ec69e0e7508c4ea1970ea90c7ca70f96] alter complete (version 1)
I20240502 01:54:09.369447 23900 ts_tablet_manager.cc:902] T 4aefc61cb95f4e949e7a23a1c677749e P dfa1152987644bd781b46c7e42a7a685: Initiating tablet copy from peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:54:09.374047 17585 tablet_service.cc:1514] Processing DeleteTablet for tablet 377f8b554ad1448d981d2283b865f8fe with delete_type TABLET_DATA_TOMBSTONED (TS 3fceccefba274b1dbbd8216af8b0bc29 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:55232
I20240502 01:54:09.401070 23611 raft_consensus.cc:1050] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9: attempting to promote NON_VOTER f19cc6cb1e76460aa6711e49a5775091 to VOTER
I20240502 01:54:09.401932 23611 consensus_queue.cc:237] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } }
I20240502 01:54:09.411409 20628 raft_consensus.cc:1261] T 18549019e31b47d899d011cef0637221 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:09.412164 17606 raft_consensus.cc:1261] T 18549019e31b47d899d011cef0637221 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:09.412240 22876 consensus_queue.cc:1035] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:09.415441 22876 consensus_queue.cc:1035] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:09.421628 20498 raft_consensus.cc:1261] T 18549019e31b47d899d011cef0637221 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:09.422396 23611 consensus_queue.cc:1035] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:09.435837 23905 tablet_replica.cc:330] T 377f8b554ad1448d981d2283b865f8fe P 3fceccefba274b1dbbd8216af8b0bc29: stopping tablet replica
I20240502 01:54:09.436533 23905 raft_consensus.cc:2229] T 377f8b554ad1448d981d2283b865f8fe P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:09.437259 23905 raft_consensus.cc:2258] T 377f8b554ad1448d981d2283b865f8fe P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:09.438977 23905 ts_tablet_manager.cc:1875] T 377f8b554ad1448d981d2283b865f8fe P 3fceccefba274b1dbbd8216af8b0bc29: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:09.442721 23908 raft_consensus.cc:2932] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:09.446579 20498 raft_consensus.cc:2932] T 18549019e31b47d899d011cef0637221 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:09.448905 23905 ts_tablet_manager.cc:1888] T 377f8b554ad1448d981d2283b865f8fe P 3fceccefba274b1dbbd8216af8b0bc29: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:54:09.449085 23905 log.cc:1196] T 377f8b554ad1448d981d2283b865f8fe P 3fceccefba274b1dbbd8216af8b0bc29: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/wals/377f8b554ad1448d981d2283b865f8fe
I20240502 01:54:09.449007 20628 raft_consensus.cc:2932] T 18549019e31b47d899d011cef0637221 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:09.450353 17319 catalog_manager.cc:4830] TS 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): tablet 377f8b554ad1448d981d2283b865f8fe (table rebalance_test_table_5 [id=1567095c8de34d6a98c56d3bb926b668]) successfully deleted
I20240502 01:54:09.452517 23900 tablet_copy_client.cc:308] T 4aefc61cb95f4e949e7a23a1c677749e P dfa1152987644bd781b46c7e42a7a685: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.66:38531
I20240502 01:54:09.453207 17627 tablet_copy_service.cc:140] P 3fceccefba274b1dbbd8216af8b0bc29: Received BeginTabletCopySession request for tablet 4aefc61cb95f4e949e7a23a1c677749e from peer dfa1152987644bd781b46c7e42a7a685 ({username='slave'} at 127.11.42.67:57179)
I20240502 01:54:09.453380 17627 tablet_copy_service.cc:161] P 3fceccefba274b1dbbd8216af8b0bc29: Beginning new tablet copy session on tablet 4aefc61cb95f4e949e7a23a1c677749e from peer dfa1152987644bd781b46c7e42a7a685 at {username='slave'} at 127.11.42.67:57179: session id = dfa1152987644bd781b46c7e42a7a685-4aefc61cb95f4e949e7a23a1c677749e
I20240502 01:54:09.455056 17606 raft_consensus.cc:2932] T 18549019e31b47d899d011cef0637221 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:09.455289 17333 catalog_manager.cc:5484] T 18549019e31b47d899d011cef0637221 P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: config changed from index 2 to 3, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:09.456073 17627 tablet_copy_source_session.cc:205] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:09.460263 23900 ts_tablet_manager.cc:1875] T 4aefc61cb95f4e949e7a23a1c677749e P dfa1152987644bd781b46c7e42a7a685: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 01:54:09.464648 20362 consensus_queue.cc:237] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } }
I20240502 01:54:09.465991 23900 ts_tablet_manager.cc:1888] T 4aefc61cb95f4e949e7a23a1c677749e P dfa1152987644bd781b46c7e42a7a685: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 01:54:09.466277 23900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aefc61cb95f4e949e7a23a1c677749e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.467059 20628 raft_consensus.cc:1261] T 18549019e31b47d899d011cef0637221 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:09.467876 22876 consensus_queue.cc:1035] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:09.469123 17604 raft_consensus.cc:1261] T 18549019e31b47d899d011cef0637221 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:09.469363 23900 tablet_copy_client.cc:783] T 4aefc61cb95f4e949e7a23a1c677749e P dfa1152987644bd781b46c7e42a7a685: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:09.469594 23900 tablet_copy_client.cc:647] T 4aefc61cb95f4e949e7a23a1c677749e P dfa1152987644bd781b46c7e42a7a685: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:09.472604 23921 consensus_queue.cc:1035] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:09.473515 23611 raft_consensus.cc:2932] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:09.474179 20628 raft_consensus.cc:2932] T 18549019e31b47d899d011cef0637221 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:09.478133 23900 tablet_copy_client.cc:523] T 4aefc61cb95f4e949e7a23a1c677749e P dfa1152987644bd781b46c7e42a7a685: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:09.478757 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 18549019e31b47d899d011cef0637221 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:09.480763 17334 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:54:09.482736 17333 catalog_manager.cc:5484] T 18549019e31b47d899d011cef0637221 P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: config changed from index 3 to 4, VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) evicted. New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:09.482949 17604 raft_consensus.cc:2932] T 18549019e31b47d899d011cef0637221 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:09.492225 20477 tablet_service.cc:1514] Processing DeleteTablet for tablet 18549019e31b47d899d011cef0637221 with delete_type TABLET_DATA_TOMBSTONED (TS 2f306549407b4bb281ccf2994035e4f1 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35018
I20240502 01:54:09.499220 23894 tablet.cc:1722] T 62018fa910704cb6a389c53556ca0509 P 4da690c0fa13407a85c0689ce8de9ade: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:09.499730 23894 tablet.cc:1620] T 62018fa910704cb6a389c53556ca0509 P 4da690c0fa13407a85c0689ce8de9ade: MemRowSet was empty: no flush needed.
I20240502 01:54:09.502094 23895 tablet.cc:1722] T 62018fa910704cb6a389c53556ca0509 P dfa1152987644bd781b46c7e42a7a685: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:09.502568 23895 tablet.cc:1620] T 62018fa910704cb6a389c53556ca0509 P dfa1152987644bd781b46c7e42a7a685: MemRowSet was empty: no flush needed.
I20240502 01:54:09.509445 23928 tablet_replica.cc:330] T 18549019e31b47d899d011cef0637221 P 2f306549407b4bb281ccf2994035e4f1: stopping tablet replica
I20240502 01:54:09.509665 23928 raft_consensus.cc:2229] T 18549019e31b47d899d011cef0637221 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:09.510058 23928 raft_consensus.cc:2258] T 18549019e31b47d899d011cef0637221 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:09.511598 23928 ts_tablet_manager.cc:1875] T 18549019e31b47d899d011cef0637221 P 2f306549407b4bb281ccf2994035e4f1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:09.515200 17334 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=ec69e0e7508c4ea1970ea90c7ca70f96] alter complete (version 2)
I20240502 01:54:09.520181 23931 ts_tablet_manager.cc:902] T 2152c07e448b4582abff22b034363c70 P 4da690c0fa13407a85c0689ce8de9ade: Initiating tablet copy from peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:54:09.520232 23897 tablet.cc:1722] T 62018fa910704cb6a389c53556ca0509 P f19cc6cb1e76460aa6711e49a5775091: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:09.520685 23897 tablet.cc:1620] T 62018fa910704cb6a389c53556ca0509 P f19cc6cb1e76460aa6711e49a5775091: MemRowSet was empty: no flush needed.
I20240502 01:54:09.522820 23931 tablet_copy_client.cc:308] T 2152c07e448b4582abff22b034363c70 P 4da690c0fa13407a85c0689ce8de9ade: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.68:44083
I20240502 01:54:09.523444 17887 tablet_copy_service.cc:140] P bb31bceba76d466db24b9341c04900c3: Received BeginTabletCopySession request for tablet 2152c07e448b4582abff22b034363c70 from peer 4da690c0fa13407a85c0689ce8de9ade ({username='slave'} at 127.11.42.65:36787)
I20240502 01:54:09.523615 17887 tablet_copy_service.cc:161] P bb31bceba76d466db24b9341c04900c3: Beginning new tablet copy session on tablet 2152c07e448b4582abff22b034363c70 from peer 4da690c0fa13407a85c0689ce8de9ade at {username='slave'} at 127.11.42.65:36787: session id = 4da690c0fa13407a85c0689ce8de9ade-2152c07e448b4582abff22b034363c70
I20240502 01:54:09.525393 23928 ts_tablet_manager.cc:1888] T 18549019e31b47d899d011cef0637221 P 2f306549407b4bb281ccf2994035e4f1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:54:09.525504 23928 log.cc:1196] T 18549019e31b47d899d011cef0637221 P 2f306549407b4bb281ccf2994035e4f1: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal/wals/18549019e31b47d899d011cef0637221
I20240502 01:54:09.526330 17322 catalog_manager.cc:4830] TS 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): tablet 18549019e31b47d899d011cef0637221 (table rebalancer_extra_table_14 [id=54db4901eae145fbbc5c66c0719703f1]) successfully deleted
I20240502 01:54:09.526515 17887 tablet_copy_source_session.cc:205] T 2152c07e448b4582abff22b034363c70 P bb31bceba76d466db24b9341c04900c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:09.528147 23931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2152c07e448b4582abff22b034363c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.529706 23900 tablet_bootstrap.cc:492] T 4aefc61cb95f4e949e7a23a1c677749e P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:54:09.536136 23931 tablet_copy_client.cc:783] T 2152c07e448b4582abff22b034363c70 P 4da690c0fa13407a85c0689ce8de9ade: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:09.536338 23931 tablet_copy_client.cc:647] T 2152c07e448b4582abff22b034363c70 P 4da690c0fa13407a85c0689ce8de9ade: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:09.537921 23931 tablet_copy_client.cc:523] T 2152c07e448b4582abff22b034363c70 P 4da690c0fa13407a85c0689ce8de9ade: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:09.540565 23931 tablet_bootstrap.cc:492] T 2152c07e448b4582abff22b034363c70 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:54:09.541546 23937 ts_tablet_manager.cc:902] T da06571e020948059fb06f65f26c831d P 2f306549407b4bb281ccf2994035e4f1: Initiating tablet copy from peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:54:09.558403 17334 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_extra_table_42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:09.573087 20212 tablet_service.cc:1467] Processing CreateTablet for tablet f537e32b31264def980094906bab350c (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=643b09798a754a0ea9786f4dd0d2bc18]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:09.573406 20477 tablet_service.cc:1467] Processing CreateTablet for tablet f537e32b31264def980094906bab350c (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=643b09798a754a0ea9786f4dd0d2bc18]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:09.573534 20212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f537e32b31264def980094906bab350c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.573853 20477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f537e32b31264def980094906bab350c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.573987 17846 tablet_service.cc:1467] Processing CreateTablet for tablet 2544ba74e9d146ef963fd92a1a6f8a98 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=643b09798a754a0ea9786f4dd0d2bc18]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:09.574363 17846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2544ba74e9d146ef963fd92a1a6f8a98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.577442 20347 tablet_service.cc:1467] Processing CreateTablet for tablet f537e32b31264def980094906bab350c (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=643b09798a754a0ea9786f4dd0d2bc18]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:09.577886 20347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f537e32b31264def980094906bab350c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.578370 20476 tablet_service.cc:1467] Processing CreateTablet for tablet 2544ba74e9d146ef963fd92a1a6f8a98 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=643b09798a754a0ea9786f4dd0d2bc18]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:09.578733 20476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2544ba74e9d146ef963fd92a1a6f8a98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.578691 20211 tablet_service.cc:1467] Processing CreateTablet for tablet 2544ba74e9d146ef963fd92a1a6f8a98 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=643b09798a754a0ea9786f4dd0d2bc18]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:09.579066 20211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2544ba74e9d146ef963fd92a1a6f8a98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.581439 20478 tablet_service.cc:1467] Processing CreateTablet for tablet 8e21dfeaf732445098f5b17a0615bb8b (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=643b09798a754a0ea9786f4dd0d2bc18]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:09.581776 20478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e21dfeaf732445098f5b17a0615bb8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.583120 20610 tablet_service.cc:1467] Processing CreateTablet for tablet 8e21dfeaf732445098f5b17a0615bb8b (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=643b09798a754a0ea9786f4dd0d2bc18]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:09.583523 20610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e21dfeaf732445098f5b17a0615bb8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.587922 20345 tablet_service.cc:1467] Processing CreateTablet for tablet 8e21dfeaf732445098f5b17a0615bb8b (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=643b09798a754a0ea9786f4dd0d2bc18]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:09.588349 20345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e21dfeaf732445098f5b17a0615bb8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.619004 17334 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:54:09.619302 23944 tablet_bootstrap.cc:492] T 2544ba74e9d146ef963fd92a1a6f8a98 P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:54:09.619627 23396 raft_consensus.cc:1050] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685: attempting to promote NON_VOTER fdc2ac372492454c85387589a91f63af to VOTER
I20240502 01:54:09.620472 23396 consensus_queue.cc:237] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } }
I20240502 01:54:09.622637 23944 tablet_bootstrap.cc:654] T 2544ba74e9d146ef963fd92a1a6f8a98 P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:09.623653 23945 tablet_bootstrap.cc:492] T 8e21dfeaf732445098f5b17a0615bb8b P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:54:09.625671 23945 tablet_bootstrap.cc:654] T 8e21dfeaf732445098f5b17a0615bb8b P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:09.616793 23946 tablet_bootstrap.cc:492] T 2544ba74e9d146ef963fd92a1a6f8a98 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:54:09.628929 23946 tablet_bootstrap.cc:654] T 2544ba74e9d146ef963fd92a1a6f8a98 P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:09.630762 23894 tablet.cc:1722] T 62018fa910704cb6a389c53556ca0509 P 4da690c0fa13407a85c0689ce8de9ade: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:09.643661 23895 tablet.cc:1722] T 62018fa910704cb6a389c53556ca0509 P dfa1152987644bd781b46c7e42a7a685: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:09.645308 17333 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table_new_name [id=ec69e0e7508c4ea1970ea90c7ca70f96] alter complete (version 3)
I20240502 01:54:09.646492 23343 raft_consensus.cc:1050] T 2ea539ebb37f42f68c166242a06f8d2e P 2f306549407b4bb281ccf2994035e4f1: attempting to promote NON_VOTER bb31bceba76d466db24b9341c04900c3 to VOTER
I20240502 01:54:09.647342 23343 consensus_queue.cc:237] T 2ea539ebb37f42f68c166242a06f8d2e P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } }
I20240502 01:54:09.647958 23946 tablet_bootstrap.cc:492] T 2544ba74e9d146ef963fd92a1a6f8a98 P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:54:09.648253 23946 ts_tablet_manager.cc:1367] T 2544ba74e9d146ef963fd92a1a6f8a98 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.000s
I20240502 01:54:09.650415 23947 tablet_bootstrap.cc:492] T 8e21dfeaf732445098f5b17a0615bb8b P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:54:09.651208 23931 tablet_bootstrap.cc:492] T 2152c07e448b4582abff22b034363c70 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:09.651960 23931 tablet_bootstrap.cc:492] T 2152c07e448b4582abff22b034363c70 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap complete.
I20240502 01:54:09.652208 23931 ts_tablet_manager.cc:1367] T 2152c07e448b4582abff22b034363c70 P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.112s user 0.011s sys 0.000s
I20240502 01:54:09.652936 23947 tablet_bootstrap.cc:654] T 8e21dfeaf732445098f5b17a0615bb8b P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:09.652861 23931 raft_consensus.cc:348] T 2152c07e448b4582abff22b034363c70 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } }
I20240502 01:54:09.653111 23931 raft_consensus.cc:729] T 2152c07e448b4582abff22b034363c70 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: LEARNER
I20240502 01:54:09.653291 23931 consensus_queue.cc:260] T 2152c07e448b4582abff22b034363c70 P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } }
I20240502 01:54:09.653928 23931 ts_tablet_manager.cc:1398] T 2152c07e448b4582abff22b034363c70 P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:09.654551 17887 tablet_copy_service.cc:342] P bb31bceba76d466db24b9341c04900c3: Request end of tablet copy session 4da690c0fa13407a85c0689ce8de9ade-2152c07e448b4582abff22b034363c70 received from {username='slave'} at 127.11.42.65:36787
I20240502 01:54:09.654680 17887 tablet_copy_service.cc:433] P bb31bceba76d466db24b9341c04900c3: ending tablet copy session 4da690c0fa13407a85c0689ce8de9ade-2152c07e448b4582abff22b034363c70 on tablet 2152c07e448b4582abff22b034363c70 with peer 4da690c0fa13407a85c0689ce8de9ade
I20240502 01:54:09.655182 23948 tablet_bootstrap.cc:492] T f537e32b31264def980094906bab350c P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:54:09.655484 23946 raft_consensus.cc:348] T 2544ba74e9d146ef963fd92a1a6f8a98 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:09.657083 23946 raft_consensus.cc:374] T 2544ba74e9d146ef963fd92a1a6f8a98 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:09.657234 23946 raft_consensus.cc:729] T 2544ba74e9d146ef963fd92a1a6f8a98 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:54:09.657229 23897 tablet.cc:1722] T 62018fa910704cb6a389c53556ca0509 P f19cc6cb1e76460aa6711e49a5775091: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:09.659358 23948 tablet_bootstrap.cc:654] T f537e32b31264def980094906bab350c P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:09.660109 20628 raft_consensus.cc:1261] T 5df83f83d2674155afba93c2d992af0a P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:09.660877 23946 consensus_queue.cc:260] T 2544ba74e9d146ef963fd92a1a6f8a98 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:09.662463 17476 raft_consensus.cc:1261] T 5df83f83d2674155afba93c2d992af0a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:09.662227 23947 tablet_bootstrap.cc:492] T 8e21dfeaf732445098f5b17a0615bb8b P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:54:09.662945 23947 ts_tablet_manager.cc:1367] T 8e21dfeaf732445098f5b17a0615bb8b P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240502 01:54:09.665350 23947 raft_consensus.cc:348] T 8e21dfeaf732445098f5b17a0615bb8b P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:09.665547 23947 raft_consensus.cc:374] T 8e21dfeaf732445098f5b17a0615bb8b P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:09.665639 23947 raft_consensus.cc:729] T 8e21dfeaf732445098f5b17a0615bb8b P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:54:09.665975 23947 consensus_queue.cc:260] T 8e21dfeaf732445098f5b17a0615bb8b P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:09.667984 23944 tablet_bootstrap.cc:492] T 2544ba74e9d146ef963fd92a1a6f8a98 P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:54:09.668787 23944 ts_tablet_manager.cc:1367] T 2544ba74e9d146ef963fd92a1a6f8a98 P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.010s
I20240502 01:54:09.670302 23944 raft_consensus.cc:348] T 2544ba74e9d146ef963fd92a1a6f8a98 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:09.671530 23944 raft_consensus.cc:374] T 2544ba74e9d146ef963fd92a1a6f8a98 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:09.671775 23944 raft_consensus.cc:729] T 2544ba74e9d146ef963fd92a1a6f8a98 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:54:09.672165 23944 consensus_queue.cc:260] T 2544ba74e9d146ef963fd92a1a6f8a98 P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:09.674505 23944 ts_tablet_manager.cc:1398] T 2544ba74e9d146ef963fd92a1a6f8a98 P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240502 01:54:09.674536 23947 ts_tablet_manager.cc:1398] T 8e21dfeaf732445098f5b17a0615bb8b P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:54:09.675164 23946 ts_tablet_manager.cc:1398] T 2544ba74e9d146ef963fd92a1a6f8a98 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20240502 01:54:09.675488 23944 tablet_bootstrap.cc:492] T f537e32b31264def980094906bab350c P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:54:09.680763 23944 tablet_bootstrap.cc:654] T f537e32b31264def980094906bab350c P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:09.683730 20231 raft_consensus.cc:1261] T 5df83f83d2674155afba93c2d992af0a P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:09.684893 23363 consensus_queue.cc:1035] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:09.694697 23945 tablet_bootstrap.cc:492] T 8e21dfeaf732445098f5b17a0615bb8b P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:54:09.695020 23945 ts_tablet_manager.cc:1367] T 8e21dfeaf732445098f5b17a0615bb8b P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.071s user 0.001s sys 0.003s
I20240502 01:54:09.695894 23945 raft_consensus.cc:348] T 8e21dfeaf732445098f5b17a0615bb8b P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:09.696194 23945 raft_consensus.cc:374] T 8e21dfeaf732445098f5b17a0615bb8b P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:09.696331 23945 raft_consensus.cc:729] T 8e21dfeaf732445098f5b17a0615bb8b P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:54:09.693113 22568 consensus_queue.cc:1035] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.011s
I20240502 01:54:09.696674 23945 consensus_queue.cc:260] T 8e21dfeaf732445098f5b17a0615bb8b P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:09.697203 22568 consensus_queue.cc:1035] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:09.697367 23944 tablet_bootstrap.cc:492] T f537e32b31264def980094906bab350c P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:54:09.697515 23944 ts_tablet_manager.cc:1367] T f537e32b31264def980094906bab350c P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.002s
I20240502 01:54:09.698221 23944 raft_consensus.cc:348] T f537e32b31264def980094906bab350c P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:09.702029 23944 raft_consensus.cc:374] T f537e32b31264def980094906bab350c P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:09.702315 23937 tablet_copy_client.cc:308] T da06571e020948059fb06f65f26c831d P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.72:42411
I20240502 01:54:09.702483 23944 raft_consensus.cc:729] T f537e32b31264def980094906bab350c P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:54:09.703151 23944 consensus_queue.cc:260] T f537e32b31264def980094906bab350c P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:09.707983 23944 ts_tablet_manager.cc:1398] T f537e32b31264def980094906bab350c P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 01:54:09.714267 20650 tablet_copy_service.cc:140] P f19cc6cb1e76460aa6711e49a5775091: Received BeginTabletCopySession request for tablet da06571e020948059fb06f65f26c831d from peer 2f306549407b4bb281ccf2994035e4f1 ({username='slave'} at 127.11.42.71:50809)
I20240502 01:54:09.714463 20650 tablet_copy_service.cc:161] P f19cc6cb1e76460aa6711e49a5775091: Beginning new tablet copy session on tablet da06571e020948059fb06f65f26c831d from peer 2f306549407b4bb281ccf2994035e4f1 at {username='slave'} at 127.11.42.71:50809: session id = 2f306549407b4bb281ccf2994035e4f1-da06571e020948059fb06f65f26c831d
I20240502 01:54:09.718047 20650 tablet_copy_source_session.cc:205] T da06571e020948059fb06f65f26c831d P f19cc6cb1e76460aa6711e49a5775091: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:09.724277 23937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da06571e020948059fb06f65f26c831d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.737412 23945 ts_tablet_manager.cc:1398] T 8e21dfeaf732445098f5b17a0615bb8b P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.042s user 0.000s sys 0.002s
I20240502 01:54:09.737749 23900 tablet_bootstrap.cc:492] T 4aefc61cb95f4e949e7a23a1c677749e P dfa1152987644bd781b46c7e42a7a685: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:09.738164 23900 tablet_bootstrap.cc:492] T 4aefc61cb95f4e949e7a23a1c677749e P dfa1152987644bd781b46c7e42a7a685: Bootstrap complete.
I20240502 01:54:09.738757 23937 tablet_copy_client.cc:783] T da06571e020948059fb06f65f26c831d P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:09.739368 23937 tablet_copy_client.cc:647] T da06571e020948059fb06f65f26c831d P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:09.740474 23900 ts_tablet_manager.cc:1367] T 4aefc61cb95f4e949e7a23a1c677749e P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.211s user 0.040s sys 0.012s
I20240502 01:54:09.741314 23900 raft_consensus.cc:348] T 4aefc61cb95f4e949e7a23a1c677749e P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:54:09.741575 23900 raft_consensus.cc:729] T 4aefc61cb95f4e949e7a23a1c677749e P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: LEARNER
I20240502 01:54:09.741767 23900 consensus_queue.cc:260] T 4aefc61cb95f4e949e7a23a1c677749e P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:54:09.742933 23937 tablet_copy_client.cc:523] T da06571e020948059fb06f65f26c831d P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:09.743292 23945 tablet_bootstrap.cc:492] T 2544ba74e9d146ef963fd92a1a6f8a98 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:54:09.748293 17862 raft_consensus.cc:1261] T 2ea539ebb37f42f68c166242a06f8d2e P bb31bceba76d466db24b9341c04900c3 [term 1 LEARNER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:09.748535 23937 tablet_bootstrap.cc:492] T da06571e020948059fb06f65f26c831d P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:54:09.748883 23694 consensus_queue.cc:1035] T 2ea539ebb37f42f68c166242a06f8d2e P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:09.749979 23945 tablet_bootstrap.cc:654] T 2544ba74e9d146ef963fd92a1a6f8a98 P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
W20240502 01:54:09.753235 17672 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.42.65:38321, user_credentials={real_user=slave}} blocked reactor thread for 68290.8us
I20240502 01:54:09.762251 23948 tablet_bootstrap.cc:492] T f537e32b31264def980094906bab350c P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:54:09.762420 23948 ts_tablet_manager.cc:1367] T f537e32b31264def980094906bab350c P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.108s user 0.009s sys 0.019s
I20240502 01:54:09.763844 23948 raft_consensus.cc:348] T f537e32b31264def980094906bab350c P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:09.764060 23948 raft_consensus.cc:374] T f537e32b31264def980094906bab350c P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:09.764151 23948 raft_consensus.cc:729] T f537e32b31264def980094906bab350c P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:54:09.764400 23948 consensus_queue.cc:260] T f537e32b31264def980094906bab350c P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:09.765121 17330 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:09.765383 17330 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:09.766561 23948 ts_tablet_manager.cc:1398] T f537e32b31264def980094906bab350c P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20240502 01:54:09.766930 23948 tablet_bootstrap.cc:492] T 8e21dfeaf732445098f5b17a0615bb8b P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:54:09.774278 17330 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet 62018fa910704cb6a389c53556ca0509
I20240502 01:54:09.775022 17716 tablet_service.cc:1514] Processing DeleteTablet for tablet 62018fa910704cb6a389c53556ca0509 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:09 UTC) from {username='slave'} at 127.0.0.1:52198
I20240502 01:54:09.775187 20610 tablet_service.cc:1514] Processing DeleteTablet for tablet 62018fa910704cb6a389c53556ca0509 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:09 UTC) from {username='slave'} at 127.0.0.1:33698
I20240502 01:54:09.775651 17456 tablet_service.cc:1514] Processing DeleteTablet for tablet 62018fa910704cb6a389c53556ca0509 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:09 UTC) from {username='slave'} at 127.0.0.1:54520
I20240502 01:54:09.777446 17476 raft_consensus.cc:1261] T 2ea539ebb37f42f68c166242a06f8d2e P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:09.778009 23363 raft_consensus.cc:2932] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:09.788074 20233 raft_consensus.cc:2932] T 5df83f83d2674155afba93c2d992af0a P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:09.792635 17476 raft_consensus.cc:2932] T 5df83f83d2674155afba93c2d992af0a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:09.793452 17330 catalog_manager.cc:5484] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 reported cstate change: config changed from index 7 to 8, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:09.798801 20628 raft_consensus.cc:1261] T 2ea539ebb37f42f68c166242a06f8d2e P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:09.799939 20628 raft_consensus.cc:2932] T 5df83f83d2674155afba93c2d992af0a P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:09.804360 23347 consensus_queue.cc:1035] T 2ea539ebb37f42f68c166242a06f8d2e P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:09.806847 23347 consensus_queue.cc:1035] T 2ea539ebb37f42f68c166242a06f8d2e P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20240502 01:54:09.812029 23611 raft_consensus.cc:482] T f537e32b31264def980094906bab350c P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:09.812741 23611 raft_consensus.cc:504] T f537e32b31264def980094906bab350c P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:09.813452 23611 leader_election.cc:288] T f537e32b31264def980094906bab350c P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:54:09.815074 20233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f537e32b31264def980094906bab350c" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:54:09.815286 20233 raft_consensus.cc:2454] T f537e32b31264def980094906bab350c P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 0.
I20240502 01:54:09.816198 20303 leader_election.cc:302] T f537e32b31264def980094906bab350c P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:54:09.816819 23611 raft_consensus.cc:2781] T f537e32b31264def980094906bab350c P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:09.816908 23611 raft_consensus.cc:482] T f537e32b31264def980094906bab350c P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:09.816988 23611 raft_consensus.cc:3037] T f537e32b31264def980094906bab350c P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:09.819646 23611 raft_consensus.cc:504] T f537e32b31264def980094906bab350c P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:09.820323 20494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f537e32b31264def980094906bab350c" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
I20240502 01:54:09.820598 20233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f537e32b31264def980094906bab350c" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:54:09.820799 20233 raft_consensus.cc:3037] T f537e32b31264def980094906bab350c P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:09.821463 20498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f537e32b31264def980094906bab350c" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
I20240502 01:54:09.823423 20233 raft_consensus.cc:2454] T f537e32b31264def980094906bab350c P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 1.
W20240502 01:54:09.821924 20303 leader_election.cc:341] T f537e32b31264def980094906bab350c P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:09.824316 23945 tablet_bootstrap.cc:492] T 2544ba74e9d146ef963fd92a1a6f8a98 P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
W20240502 01:54:09.824433 20303 leader_election.cc:341] T f537e32b31264def980094906bab350c P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:09.824527 23945 ts_tablet_manager.cc:1367] T 2544ba74e9d146ef963fd92a1a6f8a98 P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.082s user 0.007s sys 0.002s
I20240502 01:54:09.824640 20303 leader_election.cc:302] T f537e32b31264def980094906bab350c P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, fdc2ac372492454c85387589a91f63af; no voters: 2f306549407b4bb281ccf2994035e4f1
I20240502 01:54:09.824889 23908 raft_consensus.cc:2781] T f537e32b31264def980094906bab350c P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:09.825035 23908 raft_consensus.cc:686] T f537e32b31264def980094906bab350c P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Becoming Leader. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Running, Role: LEADER
I20240502 01:54:09.825294 23908 consensus_queue.cc:237] T f537e32b31264def980094906bab350c P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:09.825398 23945 raft_consensus.cc:348] T 2544ba74e9d146ef963fd92a1a6f8a98 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:09.825660 23945 raft_consensus.cc:374] T 2544ba74e9d146ef963fd92a1a6f8a98 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:09.825798 23945 raft_consensus.cc:729] T 2544ba74e9d146ef963fd92a1a6f8a98 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:54:09.826057 23945 consensus_queue.cc:260] T 2544ba74e9d146ef963fd92a1a6f8a98 P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:09.820843 23611 leader_election.cc:288] T f537e32b31264def980094906bab350c P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Requested vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:54:09.834705 23948 tablet_bootstrap.cc:654] T 8e21dfeaf732445098f5b17a0615bb8b P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:09.838413 23945 ts_tablet_manager.cc:1398] T 2544ba74e9d146ef963fd92a1a6f8a98 P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.014s user 0.003s sys 0.000s
I20240502 01:54:09.839552 23900 ts_tablet_manager.cc:1398] T 4aefc61cb95f4e949e7a23a1c677749e P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.099s user 0.002s sys 0.000s
I20240502 01:54:09.841456 17627 tablet_copy_service.cc:342] P 3fceccefba274b1dbbd8216af8b0bc29: Request end of tablet copy session dfa1152987644bd781b46c7e42a7a685-4aefc61cb95f4e949e7a23a1c677749e received from {username='slave'} at 127.11.42.67:57179
I20240502 01:54:09.841564 17627 tablet_copy_service.cc:433] P 3fceccefba274b1dbbd8216af8b0bc29: ending tablet copy session dfa1152987644bd781b46c7e42a7a685-4aefc61cb95f4e949e7a23a1c677749e on tablet 4aefc61cb95f4e949e7a23a1c677749e with peer dfa1152987644bd781b46c7e42a7a685
I20240502 01:54:09.852272 23945 tablet_bootstrap.cc:492] T f537e32b31264def980094906bab350c P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:54:09.855752 23979 tablet_replica.cc:330] T 62018fa910704cb6a389c53556ca0509 P 4da690c0fa13407a85c0689ce8de9ade: stopping tablet replica
I20240502 01:54:09.856031 23979 raft_consensus.cc:2229] T 62018fa910704cb6a389c53556ca0509 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:09.856084 23945 tablet_bootstrap.cc:654] T f537e32b31264def980094906bab350c P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:09.856505 23979 raft_consensus.cc:2258] T 62018fa910704cb6a389c53556ca0509 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:09.862244 17328 catalog_manager.cc:5484] T f537e32b31264def980094906bab350c P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70). New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:09.864315 23948 tablet_bootstrap.cc:492] T 8e21dfeaf732445098f5b17a0615bb8b P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:54:09.864894 23978 tablet_replica.cc:330] T 62018fa910704cb6a389c53556ca0509 P f19cc6cb1e76460aa6711e49a5775091: stopping tablet replica
I20240502 01:54:09.867300 23945 tablet_bootstrap.cc:492] T f537e32b31264def980094906bab350c P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:54:09.867444 23945 ts_tablet_manager.cc:1367] T f537e32b31264def980094906bab350c P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.002s
I20240502 01:54:09.867120 23948 ts_tablet_manager.cc:1367] T 8e21dfeaf732445098f5b17a0615bb8b P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.100s user 0.001s sys 0.003s
I20240502 01:54:09.867918 23977 tablet_replica.cc:330] T 62018fa910704cb6a389c53556ca0509 P dfa1152987644bd781b46c7e42a7a685: stopping tablet replica
I20240502 01:54:09.868064 23978 raft_consensus.cc:2229] T 62018fa910704cb6a389c53556ca0509 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:09.868324 23977 raft_consensus.cc:2229] T 62018fa910704cb6a389c53556ca0509 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:09.868708 23977 raft_consensus.cc:2258] T 62018fa910704cb6a389c53556ca0509 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:09.868921 23948 raft_consensus.cc:348] T 8e21dfeaf732445098f5b17a0615bb8b P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:09.869166 23948 raft_consensus.cc:374] T 8e21dfeaf732445098f5b17a0615bb8b P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:09.869236 23948 raft_consensus.cc:729] T 8e21dfeaf732445098f5b17a0615bb8b P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:54:09.869501 23948 consensus_queue.cc:260] T 8e21dfeaf732445098f5b17a0615bb8b P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:09.869972 23977 ts_tablet_manager.cc:1875] T 62018fa910704cb6a389c53556ca0509 P dfa1152987644bd781b46c7e42a7a685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:09.870280 23937 tablet_bootstrap.cc:492] T da06571e020948059fb06f65f26c831d P 2f306549407b4bb281ccf2994035e4f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:09.871688 23945 raft_consensus.cc:348] T f537e32b31264def980094906bab350c P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:09.871908 23945 raft_consensus.cc:374] T f537e32b31264def980094906bab350c P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:09.874094 23979 ts_tablet_manager.cc:1875] T 62018fa910704cb6a389c53556ca0509 P 4da690c0fa13407a85c0689ce8de9ade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:09.874382 23978 raft_consensus.cc:2258] T 62018fa910704cb6a389c53556ca0509 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:09.875303 23978 ts_tablet_manager.cc:1875] T 62018fa910704cb6a389c53556ca0509 P f19cc6cb1e76460aa6711e49a5775091: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:09.877547 17328 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:54:09.878439 20628 raft_consensus.cc:2932] T 2ea539ebb37f42f68c166242a06f8d2e P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:09.883448 23844 raft_consensus.cc:482] T 2544ba74e9d146ef963fd92a1a6f8a98 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:09.883601 23844 raft_consensus.cc:504] T 2544ba74e9d146ef963fd92a1a6f8a98 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:09.884306 23844 leader_election.cc:288] T 2544ba74e9d146ef963fd92a1a6f8a98 P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:54:09.884974 17473 raft_consensus.cc:2932] T 2ea539ebb37f42f68c166242a06f8d2e P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:09.885465 17862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2544ba74e9d146ef963fd92a1a6f8a98" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:54:09.885679 17862 raft_consensus.cc:2454] T 2544ba74e9d146ef963fd92a1a6f8a98 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 0.
I20240502 01:54:09.887169 20168 leader_election.cc:302] T 2544ba74e9d146ef963fd92a1a6f8a98 P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb31bceba76d466db24b9341c04900c3, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:54:09.887694 23844 raft_consensus.cc:2781] T 2544ba74e9d146ef963fd92a1a6f8a98 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:09.887811 23844 raft_consensus.cc:482] T 2544ba74e9d146ef963fd92a1a6f8a98 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:09.887888 23844 raft_consensus.cc:3037] T 2544ba74e9d146ef963fd92a1a6f8a98 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:09.890450 23844 raft_consensus.cc:504] T 2544ba74e9d146ef963fd92a1a6f8a98 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:09.891229 23844 leader_election.cc:288] T 2544ba74e9d146ef963fd92a1a6f8a98 P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 election: Requested vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:54:09.891680 23948 ts_tablet_manager.cc:1398] T 8e21dfeaf732445098f5b17a0615bb8b P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20240502 01:54:09.891896 17326 catalog_manager.cc:5484] T 2ea539ebb37f42f68c166242a06f8d2e P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: config changed from index 2 to 3, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:09.894505 17862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2544ba74e9d146ef963fd92a1a6f8a98" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:54:09.894685 17585 tablet_service.cc:1467] Processing CreateTablet for tablet aa5a653bc143469c8c0e891e1bd58207 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=49e09fff43434444b9f484c9d0d5562f]), partition=
I20240502 01:54:09.894735 20347 tablet_service.cc:1467] Processing CreateTablet for tablet aa5a653bc143469c8c0e891e1bd58207 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=49e09fff43434444b9f484c9d0d5562f]), partition=
I20240502 01:54:09.894752 17862 raft_consensus.cc:3037] T 2544ba74e9d146ef963fd92a1a6f8a98 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:09.895018 17585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa5a653bc143469c8c0e891e1bd58207. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.895063 20347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa5a653bc143469c8c0e891e1bd58207. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.897636 20211 tablet_service.cc:1467] Processing CreateTablet for tablet aa5a653bc143469c8c0e891e1bd58207 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=49e09fff43434444b9f484c9d0d5562f]), partition=
I20240502 01:54:09.898588 17862 raft_consensus.cc:2454] T 2544ba74e9d146ef963fd92a1a6f8a98 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 1.
I20240502 01:54:09.899006 23347 raft_consensus.cc:482] T 8e21dfeaf732445098f5b17a0615bb8b P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:09.899000 20168 leader_election.cc:302] T 2544ba74e9d146ef963fd92a1a6f8a98 P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb31bceba76d466db24b9341c04900c3, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:54:09.899398 23844 raft_consensus.cc:2781] T 2544ba74e9d146ef963fd92a1a6f8a98 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:09.899715 23844 raft_consensus.cc:686] T 2544ba74e9d146ef963fd92a1a6f8a98 P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Becoming Leader. State: Replica: fdc2ac372492454c85387589a91f63af, State: Running, Role: LEADER
I20240502 01:54:09.900075 23844 consensus_queue.cc:237] T 2544ba74e9d146ef963fd92a1a6f8a98 P fdc2ac372492454c85387589a91f63af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:09.905681 23987 tablet_bootstrap.cc:492] T aa5a653bc143469c8c0e891e1bd58207 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:54:09.907063 23987 tablet_bootstrap.cc:654] T aa5a653bc143469c8c0e891e1bd58207 P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:09.907370 23948 tablet_bootstrap.cc:492] T aa5a653bc143469c8c0e891e1bd58207 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:54:09.908830 23948 tablet_bootstrap.cc:654] T aa5a653bc143469c8c0e891e1bd58207 P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:09.909261 23945 raft_consensus.cc:729] T f537e32b31264def980094906bab350c P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:54:09.909366 23694 raft_consensus.cc:2932] T 2ea539ebb37f42f68c166242a06f8d2e P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:09.912472 20211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa5a653bc143469c8c0e891e1bd58207. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.915585 20498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2544ba74e9d146ef963fd92a1a6f8a98" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
I20240502 01:54:09.915810 20498 raft_consensus.cc:2454] T 2544ba74e9d146ef963fd92a1a6f8a98 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 0.
I20240502 01:54:09.916205 20494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2544ba74e9d146ef963fd92a1a6f8a98" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
I20240502 01:54:09.916409 20494 raft_consensus.cc:3037] T 2544ba74e9d146ef963fd92a1a6f8a98 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:09.919481 23944 tablet_bootstrap.cc:492] T aa5a653bc143469c8c0e891e1bd58207 P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:54:09.920959 23944 tablet_bootstrap.cc:654] T aa5a653bc143469c8c0e891e1bd58207 P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:09.920641 17862 raft_consensus.cc:2932] T 2ea539ebb37f42f68c166242a06f8d2e P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:09.922263 23347 raft_consensus.cc:504] T 8e21dfeaf732445098f5b17a0615bb8b P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:09.922927 23347 leader_election.cc:288] T 8e21dfeaf732445098f5b17a0615bb8b P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:54:09.924089 20628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e21dfeaf732445098f5b17a0615bb8b" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091" is_pre_election: true
I20240502 01:54:09.923569 20362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e21dfeaf732445098f5b17a0615bb8b" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:54:09.924316 20628 raft_consensus.cc:2454] T 8e21dfeaf732445098f5b17a0615bb8b P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 0.
I20240502 01:54:09.924491 20362 raft_consensus.cc:2454] T 8e21dfeaf732445098f5b17a0615bb8b P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 0.
I20240502 01:54:09.924898 20433 leader_election.cc:302] T 8e21dfeaf732445098f5b17a0615bb8b P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 6a36c952c51542c0bf7ea3ec2857a9f9; no voters:
I20240502 01:54:09.925241 23347 raft_consensus.cc:2781] T 8e21dfeaf732445098f5b17a0615bb8b P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:09.925347 23347 raft_consensus.cc:482] T 8e21dfeaf732445098f5b17a0615bb8b P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:09.925465 23347 raft_consensus.cc:3037] T 8e21dfeaf732445098f5b17a0615bb8b P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:09.927753 20494 raft_consensus.cc:2454] T 2544ba74e9d146ef963fd92a1a6f8a98 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 1.
I20240502 01:54:09.928140 23347 raft_consensus.cc:504] T 8e21dfeaf732445098f5b17a0615bb8b P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:09.930173 23937 tablet_bootstrap.cc:492] T da06571e020948059fb06f65f26c831d P 2f306549407b4bb281ccf2994035e4f1: Bootstrap complete.
I20240502 01:54:09.930424 23937 ts_tablet_manager.cc:1367] T da06571e020948059fb06f65f26c831d P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.182s user 0.004s sys 0.005s
I20240502 01:54:09.930953 20362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e21dfeaf732445098f5b17a0615bb8b" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
I20240502 01:54:09.931150 20362 raft_consensus.cc:3037] T 8e21dfeaf732445098f5b17a0615bb8b P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:09.931224 23937 raft_consensus.cc:348] T da06571e020948059fb06f65f26c831d P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
I20240502 01:54:09.931419 23937 raft_consensus.cc:729] T da06571e020948059fb06f65f26c831d P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: LEARNER
I20240502 01:54:09.932041 23937 consensus_queue.cc:260] T da06571e020948059fb06f65f26c831d P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
I20240502 01:54:09.932735 23937 ts_tablet_manager.cc:1398] T da06571e020948059fb06f65f26c831d P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:54:09.932893 23944 tablet_bootstrap.cc:492] T aa5a653bc143469c8c0e891e1bd58207 P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:54:09.933046 23944 ts_tablet_manager.cc:1367] T aa5a653bc143469c8c0e891e1bd58207 P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.001s
I20240502 01:54:09.933866 23944 raft_consensus.cc:348] T aa5a653bc143469c8c0e891e1bd58207 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:09.934111 23944 raft_consensus.cc:374] T aa5a653bc143469c8c0e891e1bd58207 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:09.934326 23944 raft_consensus.cc:729] T aa5a653bc143469c8c0e891e1bd58207 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:54:09.934723 23944 consensus_queue.cc:260] T aa5a653bc143469c8c0e891e1bd58207 P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:09.935333 20494 consensus_queue.cc:237] T 2ea539ebb37f42f68c166242a06f8d2e P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } }
I20240502 01:54:09.935649 23944 ts_tablet_manager.cc:1398] T aa5a653bc143469c8c0e891e1bd58207 P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:54:09.936707 20650 tablet_copy_service.cc:342] P f19cc6cb1e76460aa6711e49a5775091: Request end of tablet copy session 2f306549407b4bb281ccf2994035e4f1-da06571e020948059fb06f65f26c831d received from {username='slave'} at 127.11.42.71:50809
I20240502 01:54:09.936843 20650 tablet_copy_service.cc:433] P f19cc6cb1e76460aa6711e49a5775091: ending tablet copy session 2f306549407b4bb281ccf2994035e4f1-da06571e020948059fb06f65f26c831d on tablet da06571e020948059fb06f65f26c831d with peer 2f306549407b4bb281ccf2994035e4f1
I20240502 01:54:09.938275 17476 raft_consensus.cc:1261] T 2ea539ebb37f42f68c166242a06f8d2e P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:09.938582 20628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e21dfeaf732445098f5b17a0615bb8b" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091"
I20240502 01:54:09.938789 20628 raft_consensus.cc:3037] T 8e21dfeaf732445098f5b17a0615bb8b P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:09.940938 23343 consensus_queue.cc:1035] T 2ea539ebb37f42f68c166242a06f8d2e P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:09.941110 23347 leader_election.cc:288] T 8e21dfeaf732445098f5b17a0615bb8b P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 election: Requested vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:54:09.942322 20362 raft_consensus.cc:2454] T 8e21dfeaf732445098f5b17a0615bb8b P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 1.
I20240502 01:54:09.942898 20433 leader_election.cc:302] T 8e21dfeaf732445098f5b17a0615bb8b P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 6a36c952c51542c0bf7ea3ec2857a9f9; no voters:
I20240502 01:54:09.943781 20628 raft_consensus.cc:2454] T 8e21dfeaf732445098f5b17a0615bb8b P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 1.
I20240502 01:54:09.944150 23343 raft_consensus.cc:2781] T 8e21dfeaf732445098f5b17a0615bb8b P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:09.944319 23343 raft_consensus.cc:686] T 8e21dfeaf732445098f5b17a0615bb8b P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Running, Role: LEADER
I20240502 01:54:09.944548 23343 consensus_queue.cc:237] T 8e21dfeaf732445098f5b17a0615bb8b P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:09.947485 17862 raft_consensus.cc:1261] T 2ea539ebb37f42f68c166242a06f8d2e P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:09.948107 23694 consensus_queue.cc:1035] T 2ea539ebb37f42f68c166242a06f8d2e P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:09.949913 23987 tablet_bootstrap.cc:492] T aa5a653bc143469c8c0e891e1bd58207 P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:54:09.950098 23987 ts_tablet_manager.cc:1367] T aa5a653bc143469c8c0e891e1bd58207 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.000s
I20240502 01:54:09.955111 23347 raft_consensus.cc:2932] T 2ea539ebb37f42f68c166242a06f8d2e P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:09.956871 23978 ts_tablet_manager.cc:1888] T 62018fa910704cb6a389c53556ca0509 P f19cc6cb1e76460aa6711e49a5775091: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:09.956848 17476 raft_consensus.cc:2932] T 2ea539ebb37f42f68c166242a06f8d2e P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:09.959849 23987 raft_consensus.cc:348] T aa5a653bc143469c8c0e891e1bd58207 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:09.960426 23987 raft_consensus.cc:374] T aa5a653bc143469c8c0e891e1bd58207 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:09.960557 23987 raft_consensus.cc:729] T aa5a653bc143469c8c0e891e1bd58207 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:54:09.963987 17334 catalog_manager.cc:5484] T 2ea539ebb37f42f68c166242a06f8d2e P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: config changed from index 3 to 4, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:09.965514 23979 ts_tablet_manager.cc:1888] T 62018fa910704cb6a389c53556ca0509 P 4da690c0fa13407a85c0689ce8de9ade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:09.965667 23979 log.cc:1196] T 62018fa910704cb6a389c53556ca0509 P 4da690c0fa13407a85c0689ce8de9ade: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/62018fa910704cb6a389c53556ca0509
I20240502 01:54:09.966056 23979 ts_tablet_manager.cc:1909] T 62018fa910704cb6a389c53556ca0509 P 4da690c0fa13407a85c0689ce8de9ade: Deleting consensus metadata
I20240502 01:54:09.960898 23987 consensus_queue.cc:260] T aa5a653bc143469c8c0e891e1bd58207 P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:09.966354 23807 raft_consensus.cc:482] T aa5a653bc143469c8c0e891e1bd58207 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:09.966463 23807 raft_consensus.cc:504] T aa5a653bc143469c8c0e891e1bd58207 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:09.967269 23807 leader_election.cc:288] T aa5a653bc143469c8c0e891e1bd58207 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:54:09.968174 20362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5a653bc143469c8c0e891e1bd58207" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:54:09.968326 20231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5a653bc143469c8c0e891e1bd58207" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:54:09.968535 20231 raft_consensus.cc:2454] T aa5a653bc143469c8c0e891e1bd58207 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
W20240502 01:54:09.969198 17542 leader_election.cc:341] T aa5a653bc143469c8c0e891e1bd58207 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:09.969527 17543 leader_election.cc:302] T aa5a653bc143469c8c0e891e1bd58207 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, fdc2ac372492454c85387589a91f63af; no voters: 6a36c952c51542c0bf7ea3ec2857a9f9
I20240502 01:54:09.978965 23807 raft_consensus.cc:2781] T aa5a653bc143469c8c0e891e1bd58207 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:09.979065 23807 raft_consensus.cc:482] T aa5a653bc143469c8c0e891e1bd58207 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:09.979135 23807 raft_consensus.cc:3037] T aa5a653bc143469c8c0e891e1bd58207 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:09.987053 17322 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2ea539ebb37f42f68c166242a06f8d2e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:09.987110 23807 raft_consensus.cc:504] T aa5a653bc143469c8c0e891e1bd58207 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:09.988184 20231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5a653bc143469c8c0e891e1bd58207" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:54:09.988389 20231 raft_consensus.cc:3037] T aa5a653bc143469c8c0e891e1bd58207 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:09.988433 20366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5a653bc143469c8c0e891e1bd58207" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
W20240502 01:54:09.988947 17542 leader_election.cc:341] T aa5a653bc143469c8c0e891e1bd58207 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:09.989110 23807 leader_election.cc:288] T aa5a653bc143469c8c0e891e1bd58207 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Requested vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:54:09.991230 20231 raft_consensus.cc:2454] T aa5a653bc143469c8c0e891e1bd58207 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:54:09.991535 17543 leader_election.cc:302] T aa5a653bc143469c8c0e891e1bd58207 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, fdc2ac372492454c85387589a91f63af; no voters: 6a36c952c51542c0bf7ea3ec2857a9f9
I20240502 01:54:09.992704 23807 raft_consensus.cc:2781] T aa5a653bc143469c8c0e891e1bd58207 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:09.993609 23807 raft_consensus.cc:686] T aa5a653bc143469c8c0e891e1bd58207 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Becoming Leader. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Running, Role: LEADER
I20240502 01:54:09.993880 23807 consensus_queue.cc:237] T aa5a653bc143469c8c0e891e1bd58207 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:09.997247 23948 tablet_bootstrap.cc:492] T aa5a653bc143469c8c0e891e1bd58207 P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:54:09.997421 23948 ts_tablet_manager.cc:1367] T aa5a653bc143469c8c0e891e1bd58207 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.090s user 0.003s sys 0.000s
I20240502 01:54:09.998817 23948 raft_consensus.cc:348] T aa5a653bc143469c8c0e891e1bd58207 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:09.999042 23948 raft_consensus.cc:374] T aa5a653bc143469c8c0e891e1bd58207 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:09.999148 23948 raft_consensus.cc:729] T aa5a653bc143469c8c0e891e1bd58207 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:54:09.999344 23948 consensus_queue.cc:260] T aa5a653bc143469c8c0e891e1bd58207 P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:09.909636 23945 consensus_queue.cc:260] T f537e32b31264def980094906bab350c P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:10.002251 23948 ts_tablet_manager.cc:1398] T aa5a653bc143469c8c0e891e1bd58207 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:54:10.002872 23987 ts_tablet_manager.cc:1398] T aa5a653bc143469c8c0e891e1bd58207 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.051s user 0.002s sys 0.000s
I20240502 01:54:09.989398 20610 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ea539ebb37f42f68c166242a06f8d2e with delete_type TABLET_DATA_TOMBSTONED (TS f19cc6cb1e76460aa6711e49a5775091 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33698
I20240502 01:54:10.003922 17326 catalog_manager.cc:5484] T aa5a653bc143469c8c0e891e1bd58207 P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66). New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:10.004642 17321 catalog_manager.cc:4830] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): tablet 62018fa910704cb6a389c53556ca0509 (table rebalancer_dynamic_table_new_name [id=ec69e0e7508c4ea1970ea90c7ca70f96]) successfully deleted
I20240502 01:54:10.005129 23978 log.cc:1196] T 62018fa910704cb6a389c53556ca0509 P f19cc6cb1e76460aa6711e49a5775091: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal/wals/62018fa910704cb6a389c53556ca0509
I20240502 01:54:10.005419 23945 ts_tablet_manager.cc:1398] T f537e32b31264def980094906bab350c P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.138s user 0.002s sys 0.001s
I20240502 01:54:10.006012 23978 ts_tablet_manager.cc:1909] T 62018fa910704cb6a389c53556ca0509 P f19cc6cb1e76460aa6711e49a5775091: Deleting consensus metadata
I20240502 01:54:10.010440 17321 catalog_manager.cc:4830] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): tablet 62018fa910704cb6a389c53556ca0509 (table rebalancer_dynamic_table_new_name [id=ec69e0e7508c4ea1970ea90c7ca70f96]) successfully deleted
I20240502 01:54:10.012034 23977 ts_tablet_manager.cc:1888] T 62018fa910704cb6a389c53556ca0509 P dfa1152987644bd781b46c7e42a7a685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:10.013341 23977 log.cc:1196] T 62018fa910704cb6a389c53556ca0509 P dfa1152987644bd781b46c7e42a7a685: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/wals/62018fa910704cb6a389c53556ca0509
I20240502 01:54:10.020416 23977 ts_tablet_manager.cc:1909] T 62018fa910704cb6a389c53556ca0509 P dfa1152987644bd781b46c7e42a7a685: Deleting consensus metadata
I20240502 01:54:10.024130 23978 tablet_replica.cc:330] T 2ea539ebb37f42f68c166242a06f8d2e P f19cc6cb1e76460aa6711e49a5775091: stopping tablet replica
I20240502 01:54:10.025838 23978 raft_consensus.cc:2229] T 2ea539ebb37f42f68c166242a06f8d2e P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:10.026858 23978 raft_consensus.cc:2258] T 2ea539ebb37f42f68c166242a06f8d2e P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:10.026414 17862 raft_consensus.cc:2932] T 2ea539ebb37f42f68c166242a06f8d2e P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:10.029933 23978 ts_tablet_manager.cc:1875] T 2ea539ebb37f42f68c166242a06f8d2e P f19cc6cb1e76460aa6711e49a5775091: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:10.040477 17321 catalog_manager.cc:4830] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): tablet 62018fa910704cb6a389c53556ca0509 (table rebalancer_dynamic_table_new_name [id=ec69e0e7508c4ea1970ea90c7ca70f96]) successfully deleted
I20240502 01:54:10.038033 17328 catalog_manager.cc:5484] T 8e21dfeaf732445098f5b17a0615bb8b P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71). New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:10.041435 22876 raft_consensus.cc:482] T aa5a653bc143469c8c0e891e1bd58207 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:10.041589 22876 raft_consensus.cc:504] T aa5a653bc143469c8c0e891e1bd58207 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:10.042196 22876 leader_election.cc:288] T aa5a653bc143469c8c0e891e1bd58207 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:54:10.042865 20231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5a653bc143469c8c0e891e1bd58207" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:54:10.043126 20231 raft_consensus.cc:2379] T aa5a653bc143469c8c0e891e1bd58207 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in current term 1: Already voted for candidate 3fceccefba274b1dbbd8216af8b0bc29 in this term.
I20240502 01:54:10.043337 17606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5a653bc143469c8c0e891e1bd58207" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:54:10.045055 20300 leader_election.cc:302] T aa5a653bc143469c8c0e891e1bd58207 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9; no voters: 3fceccefba274b1dbbd8216af8b0bc29, fdc2ac372492454c85387589a91f63af
I20240502 01:54:10.045506 22876 raft_consensus.cc:3037] T aa5a653bc143469c8c0e891e1bd58207 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:10.048349 22876 raft_consensus.cc:2726] T aa5a653bc143469c8c0e891e1bd58207 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:54:10.058619 17328 catalog_manager.cc:5484] T 2544ba74e9d146ef963fd92a1a6f8a98 P fdc2ac372492454c85387589a91f63af reported cstate change: term changed from 0 to 1, leader changed from <none> to fdc2ac372492454c85387589a91f63af (127.11.42.69). New cstate: current_term: 1 leader_uuid: "fdc2ac372492454c85387589a91f63af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:10.068954 23978 ts_tablet_manager.cc:1888] T 2ea539ebb37f42f68c166242a06f8d2e P f19cc6cb1e76460aa6711e49a5775091: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:54:10.069146 23978 log.cc:1196] T 2ea539ebb37f42f68c166242a06f8d2e P f19cc6cb1e76460aa6711e49a5775091: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal/wals/2ea539ebb37f42f68c166242a06f8d2e
I20240502 01:54:10.073915 17321 catalog_manager.cc:4830] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): tablet 2ea539ebb37f42f68c166242a06f8d2e (table rebalancer_extra_table_26 [id=43fc79d9e659415194e6983d21d5384a]) successfully deleted
I20240502 01:54:10.125393 17476 raft_consensus.cc:1203] T 2152c07e448b4582abff22b034363c70 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:10.139220 20494 raft_consensus.cc:1203] T da06571e020948059fb06f65f26c831d P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:10.141726 17330 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:54:10.158941 23694 raft_consensus.cc:482] T f537e32b31264def980094906bab350c P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:10.159094 23694 raft_consensus.cc:504] T f537e32b31264def980094906bab350c P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:10.159768 23694 leader_election.cc:288] T f537e32b31264def980094906bab350c P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:54:10.160053 20231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f537e32b31264def980094906bab350c" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:54:10.160368 20231 raft_consensus.cc:2379] T f537e32b31264def980094906bab350c P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2f306549407b4bb281ccf2994035e4f1 in current term 1: Already voted for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in this term.
I20240502 01:54:10.160548 20362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f537e32b31264def980094906bab350c" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:54:10.161080 20433 leader_election.cc:302] T f537e32b31264def980094906bab350c P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1; no voters: 6a36c952c51542c0bf7ea3ec2857a9f9, fdc2ac372492454c85387589a91f63af
I20240502 01:54:10.162282 20231 raft_consensus.cc:1261] T aa5a653bc143469c8c0e891e1bd58207 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:10.162379 20362 raft_consensus.cc:1261] T aa5a653bc143469c8c0e891e1bd58207 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:10.162662 23694 raft_consensus.cc:3037] T f537e32b31264def980094906bab350c P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:10.162802 23766 consensus_queue.cc:1035] T aa5a653bc143469c8c0e891e1bd58207 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:10.163650 23766 consensus_queue.cc:1035] T aa5a653bc143469c8c0e891e1bd58207 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:10.168373 23694 raft_consensus.cc:2726] T f537e32b31264def980094906bab350c P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:54:10.186861 17330 catalog_manager.cc:5576] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339) does not have the latest schema for tablet aa5a653bc143469c8c0e891e1bd58207 (table rebalancer_dynamic_table [id=49e09fff43434444b9f484c9d0d5562f]). Expected version 1 got 0
I20240502 01:54:10.191751 17330 catalog_manager.cc:5576] TS 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155) does not have the latest schema for tablet aa5a653bc143469c8c0e891e1bd58207 (table rebalancer_dynamic_table [id=49e09fff43434444b9f484c9d0d5562f]). Expected version 1 got 0
I20240502 01:54:10.214339 24024 tablet.cc:1722] T aa5a653bc143469c8c0e891e1bd58207 P 6a36c952c51542c0bf7ea3ec2857a9f9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:10.216612 24024 tablet.cc:1620] T aa5a653bc143469c8c0e891e1bd58207 P 6a36c952c51542c0bf7ea3ec2857a9f9: MemRowSet was empty: no flush needed.
I20240502 01:54:10.213974 24021 tablet.cc:1722] T aa5a653bc143469c8c0e891e1bd58207 P fdc2ac372492454c85387589a91f63af: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:10.217521 24021 tablet.cc:1620] T aa5a653bc143469c8c0e891e1bd58207 P fdc2ac372492454c85387589a91f63af: MemRowSet was empty: no flush needed.
I20240502 01:54:10.213531 24020 tablet.cc:1722] T aa5a653bc143469c8c0e891e1bd58207 P 3fceccefba274b1dbbd8216af8b0bc29: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:10.218223 17328 catalog_manager.cc:5576] TS 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531) does not have the latest schema for tablet aa5a653bc143469c8c0e891e1bd58207 (table rebalancer_dynamic_table [id=49e09fff43434444b9f484c9d0d5562f]). Expected version 1 got 0
I20240502 01:54:10.218318 24020 tablet.cc:1620] T aa5a653bc143469c8c0e891e1bd58207 P 3fceccefba274b1dbbd8216af8b0bc29: MemRowSet was empty: no flush needed.
I20240502 01:54:10.218796 17330 catalog_manager.cc:5576] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339) does not have the latest schema for tablet aa5a653bc143469c8c0e891e1bd58207 (table rebalancer_dynamic_table [id=49e09fff43434444b9f484c9d0d5562f]). Expected version 1 got 0
I20240502 01:54:10.230374 17334 catalog_manager.cc:5576] TS 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155) does not have the latest schema for tablet aa5a653bc143469c8c0e891e1bd58207 (table rebalancer_dynamic_table [id=49e09fff43434444b9f484c9d0d5562f]). Expected version 1 got 0
I20240502 01:54:10.234217 17328 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=49e09fff43434444b9f484c9d0d5562f] alter complete (version 1)
I20240502 01:54:10.257404 22876 consensus_queue.cc:1035] T f537e32b31264def980094906bab350c P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:10.265868 22876 consensus_queue.cc:1035] T f537e32b31264def980094906bab350c P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:10.277405 23844 consensus_queue.cc:1035] T 2544ba74e9d146ef963fd92a1a6f8a98 P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:10.286141 23844 consensus_queue.cc:1035] T 2544ba74e9d146ef963fd92a1a6f8a98 P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:10.289508 17737 raft_consensus.cc:1203] T 4aefc61cb95f4e949e7a23a1c677749e P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 01:54:10.346698 17330 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:54:10.349287 20366 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a212c17ba64842c99b3d6a78a3b7cc8a"
dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49558
I20240502 01:54:10.349572 20366 raft_consensus.cc:595] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:54:10.355296 24020 tablet.cc:1722] T aa5a653bc143469c8c0e891e1bd58207 P 3fceccefba274b1dbbd8216af8b0bc29: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:10.355808 24020 tablet.cc:1620] T aa5a653bc143469c8c0e891e1bd58207 P 3fceccefba274b1dbbd8216af8b0bc29: MemRowSet was empty: no flush needed.
I20240502 01:54:10.357344 24024 tablet.cc:1722] T aa5a653bc143469c8c0e891e1bd58207 P 6a36c952c51542c0bf7ea3ec2857a9f9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:10.357856 24024 tablet.cc:1620] T aa5a653bc143469c8c0e891e1bd58207 P 6a36c952c51542c0bf7ea3ec2857a9f9: MemRowSet was empty: no flush needed.
I20240502 01:54:10.358654 24021 tablet.cc:1722] T aa5a653bc143469c8c0e891e1bd58207 P fdc2ac372492454c85387589a91f63af: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:10.359122 24021 tablet.cc:1620] T aa5a653bc143469c8c0e891e1bd58207 P fdc2ac372492454c85387589a91f63af: MemRowSet was empty: no flush needed.
I20240502 01:54:10.369604 17330 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=49e09fff43434444b9f484c9d0d5562f] alter complete (version 2)
I20240502 01:54:10.386973 23694 consensus_queue.cc:1035] T 8e21dfeaf732445098f5b17a0615bb8b P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:10.398262 23694 consensus_queue.cc:1035] T 8e21dfeaf732445098f5b17a0615bb8b P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:10.410003 22876 raft_consensus.cc:979] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9: : Instructing follower dfa1152987644bd781b46c7e42a7a685 to start an election
I20240502 01:54:10.410181 22876 raft_consensus.cc:1067] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Signalling peer dfa1152987644bd781b46c7e42a7a685 to start an election
I20240502 01:54:10.412673 17737 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a212c17ba64842c99b3d6a78a3b7cc8a"
dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
from {username='slave'} at 127.11.42.70:55001
I20240502 01:54:10.413118 17737 raft_consensus.cc:482] T a212c17ba64842c99b3d6a78a3b7cc8a P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:10.413277 17737 raft_consensus.cc:3037] T a212c17ba64842c99b3d6a78a3b7cc8a P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:10.416558 17737 raft_consensus.cc:504] T a212c17ba64842c99b3d6a78a3b7cc8a P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:10.418499 17473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a212c17ba64842c99b3d6a78a3b7cc8a" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:54:10.418706 17473 raft_consensus.cc:3037] T a212c17ba64842c99b3d6a78a3b7cc8a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:10.421965 17473 raft_consensus.cc:2454] T a212c17ba64842c99b3d6a78a3b7cc8a P 4da690c0fa13407a85c0689ce8de9ade [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 2.
I20240502 01:54:10.422895 17737 leader_election.cc:288] T a212c17ba64842c99b3d6a78a3b7cc8a P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 2 election: Requested vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:54:10.423290 20366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a212c17ba64842c99b3d6a78a3b7cc8a" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
I20240502 01:54:10.423344 20628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a212c17ba64842c99b3d6a78a3b7cc8a" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "f19cc6cb1e76460aa6711e49a5775091"
I20240502 01:54:10.423499 20366 raft_consensus.cc:3032] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:10.423556 20628 raft_consensus.cc:3037] T a212c17ba64842c99b3d6a78a3b7cc8a P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:10.423588 20366 raft_consensus.cc:729] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Running, Role: LEADER
I20240502 01:54:10.423904 20366 consensus_queue.cc:260] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:10.424258 20366 raft_consensus.cc:3037] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:10.426223 20628 raft_consensus.cc:2454] T a212c17ba64842c99b3d6a78a3b7cc8a P f19cc6cb1e76460aa6711e49a5775091 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 2.
I20240502 01:54:10.427093 17672 leader_election.cc:302] T a212c17ba64842c99b3d6a78a3b7cc8a P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, dfa1152987644bd781b46c7e42a7a685, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:54:10.427390 23967 raft_consensus.cc:2781] T a212c17ba64842c99b3d6a78a3b7cc8a P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:10.427543 23967 raft_consensus.cc:686] T a212c17ba64842c99b3d6a78a3b7cc8a P dfa1152987644bd781b46c7e42a7a685 [term 2 LEADER]: Becoming Leader. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Running, Role: LEADER
I20240502 01:54:10.427726 20366 raft_consensus.cc:2454] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 2.
I20240502 01:54:10.427829 23967 consensus_queue.cc:237] T a212c17ba64842c99b3d6a78a3b7cc8a P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:10.435020 17328 catalog_manager.cc:5484] T a212c17ba64842c99b3d6a78a3b7cc8a P dfa1152987644bd781b46c7e42a7a685 reported cstate change: term changed from 1 to 2, leader changed from 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) to dfa1152987644bd781b46c7e42a7a685 (127.11.42.67). New cstate: current_term: 2 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:10.470395 17862 consensus_queue.cc:237] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
I20240502 01:54:10.471124 17328 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:54:10.474769 20628 raft_consensus.cc:1261] T 4bb42883b1a0475b863d008bd01fa70a P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:10.474903 17606 raft_consensus.cc:1261] T 4bb42883b1a0475b863d008bd01fa70a P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 01:54:10.475219 17803 consensus_peers.cc:485] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 -> Peer 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): Couldn't send request to peer 2f306549407b4bb281ccf2994035e4f1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4bb42883b1a0475b863d008bd01fa70a. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:10.475591 23898 consensus_queue.cc:1035] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:10.475880 21667 consensus_queue.cc:1035] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:10.480162 21667 raft_consensus.cc:2932] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:54:10.480998 24020 tablet.cc:1722] T aa5a653bc143469c8c0e891e1bd58207 P 3fceccefba274b1dbbd8216af8b0bc29: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:10.481343 20628 raft_consensus.cc:2932] T 4bb42883b1a0475b863d008bd01fa70a P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:54:10.484774 17606 raft_consensus.cc:2932] T 4bb42883b1a0475b863d008bd01fa70a P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:54:10.485152 24021 tablet.cc:1722] T aa5a653bc143469c8c0e891e1bd58207 P fdc2ac372492454c85387589a91f63af: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:10.487109 17328 catalog_manager.cc:5484] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 reported cstate change: config changed from index 6 to 7, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:10.496081 17330 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table_new_name [id=49e09fff43434444b9f484c9d0d5562f] alter complete (version 3)
I20240502 01:54:10.497413 24024 tablet.cc:1722] T aa5a653bc143469c8c0e891e1bd58207 P 6a36c952c51542c0bf7ea3ec2857a9f9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:10.514222 20366 consensus_queue.cc:237] T 472d7827b3074236a81a3ede098f6c0a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } }
I20240502 01:54:10.516739 20233 raft_consensus.cc:1261] T 472d7827b3074236a81a3ede098f6c0a P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:54:10.516922 20300 consensus_peers.cc:485] T 472d7827b3074236a81a3ede098f6c0a P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): Couldn't send request to peer 3fceccefba274b1dbbd8216af8b0bc29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 472d7827b3074236a81a3ede098f6c0a. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:10.517263 23611 consensus_queue.cc:1035] T 472d7827b3074236a81a3ede098f6c0a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:10.517678 17473 raft_consensus.cc:1261] T 472d7827b3074236a81a3ede098f6c0a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:10.518229 23611 consensus_queue.cc:1035] T 472d7827b3074236a81a3ede098f6c0a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:10.520021 23908 raft_consensus.cc:2932] T 472d7827b3074236a81a3ede098f6c0a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } } }
I20240502 01:54:10.520623 20233 raft_consensus.cc:2932] T 472d7827b3074236a81a3ede098f6c0a P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } } }
I20240502 01:54:10.521178 17473 raft_consensus.cc:2932] T 472d7827b3074236a81a3ede098f6c0a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } } }
I20240502 01:54:10.526578 17330 catalog_manager.cc:5484] T 472d7827b3074236a81a3ede098f6c0a P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: config changed from index -1 to 2, NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) added. New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:10.541628 20628 consensus_queue.cc:237] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } }
I20240502 01:54:10.544018 17737 raft_consensus.cc:1261] T cf0237a4c65b495ba892871dd850bed9 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 01:54:10.544205 20565 consensus_peers.cc:485] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 -> Peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): Couldn't send request to peer bb31bceba76d466db24b9341c04900c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cf0237a4c65b495ba892871dd850bed9. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:10.544737 24007 consensus_queue.cc:1035] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:10.544840 20498 raft_consensus.cc:1261] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:10.545428 24010 consensus_queue.cc:1035] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:10.547642 24007 raft_consensus.cc:2932] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } } }
I20240502 01:54:10.549633 17737 raft_consensus.cc:2932] T cf0237a4c65b495ba892871dd850bed9 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } } }
I20240502 01:54:10.554327 20494 raft_consensus.cc:2932] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } } }
I20240502 01:54:10.554425 17334 catalog_manager.cc:5484] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: config changed from index 4 to 5, NON_VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) added. New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:10.557423 24010 raft_consensus.cc:1050] T da06571e020948059fb06f65f26c831d P f19cc6cb1e76460aa6711e49a5775091: attempting to promote NON_VOTER 2f306549407b4bb281ccf2994035e4f1 to VOTER
I20240502 01:54:10.558305 24010 consensus_queue.cc:237] T da06571e020948059fb06f65f26c831d P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } }
I20240502 01:54:10.562191 20231 raft_consensus.cc:1261] T da06571e020948059fb06f65f26c831d P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:10.563319 17737 raft_consensus.cc:1261] T da06571e020948059fb06f65f26c831d P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:10.563357 20498 raft_consensus.cc:1261] T da06571e020948059fb06f65f26c831d P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:10.563911 24007 consensus_queue.cc:1035] T da06571e020948059fb06f65f26c831d P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:10.564445 24007 consensus_queue.cc:1035] T da06571e020948059fb06f65f26c831d P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:10.590351 24007 consensus_queue.cc:1035] T da06571e020948059fb06f65f26c831d P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:10.603413 17326 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:10.603704 17326 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:10.606392 24125 raft_consensus.cc:2932] T da06571e020948059fb06f65f26c831d P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:10.608063 17326 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet aa5a653bc143469c8c0e891e1bd58207
I20240502 01:54:10.608915 17585 tablet_service.cc:1514] Processing DeleteTablet for tablet aa5a653bc143469c8c0e891e1bd58207 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:10 UTC) from {username='slave'} at 127.0.0.1:55232
I20240502 01:54:10.609138 20347 tablet_service.cc:1514] Processing DeleteTablet for tablet aa5a653bc143469c8c0e891e1bd58207 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:10 UTC) from {username='slave'} at 127.0.0.1:40196
I20240502 01:54:10.609138 17737 raft_consensus.cc:2932] T da06571e020948059fb06f65f26c831d P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:10.610625 20211 tablet_service.cc:1514] Processing DeleteTablet for tablet aa5a653bc143469c8c0e891e1bd58207 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:10 UTC) from {username='slave'} at 127.0.0.1:44624
I20240502 01:54:10.616616 20494 raft_consensus.cc:2932] T da06571e020948059fb06f65f26c831d P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:10.619115 17326 catalog_manager.cc:5484] T da06571e020948059fb06f65f26c831d P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: config changed from index 2 to 3, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:10.622064 20231 raft_consensus.cc:2932] T da06571e020948059fb06f65f26c831d P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:10.626351 20628 consensus_queue.cc:237] T da06571e020948059fb06f65f26c831d P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } }
I20240502 01:54:10.628909 20494 raft_consensus.cc:1261] T da06571e020948059fb06f65f26c831d P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:10.630137 24010 consensus_queue.cc:1035] T da06571e020948059fb06f65f26c831d P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:10.633947 17735 raft_consensus.cc:1261] T da06571e020948059fb06f65f26c831d P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:10.634176 24135 tablet_replica.cc:330] T aa5a653bc143469c8c0e891e1bd58207 P 6a36c952c51542c0bf7ea3ec2857a9f9: stopping tablet replica
I20240502 01:54:10.634565 24135 raft_consensus.cc:2229] T aa5a653bc143469c8c0e891e1bd58207 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:10.634856 24135 raft_consensus.cc:2258] T aa5a653bc143469c8c0e891e1bd58207 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:10.634832 24127 consensus_queue.cc:1035] T da06571e020948059fb06f65f26c831d P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:10.635339 24134 tablet_replica.cc:330] T aa5a653bc143469c8c0e891e1bd58207 P 3fceccefba274b1dbbd8216af8b0bc29: stopping tablet replica
I20240502 01:54:10.636296 24135 ts_tablet_manager.cc:1875] T aa5a653bc143469c8c0e891e1bd58207 P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:10.636590 24134 raft_consensus.cc:2229] T aa5a653bc143469c8c0e891e1bd58207 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:10.637054 24134 raft_consensus.cc:2258] T aa5a653bc143469c8c0e891e1bd58207 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:10.639755 24125 raft_consensus.cc:2932] T da06571e020948059fb06f65f26c831d P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:10.641073 17735 raft_consensus.cc:2932] T da06571e020948059fb06f65f26c831d P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:10.644459 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet da06571e020948059fb06f65f26c831d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:10.645244 20494 raft_consensus.cc:2932] T da06571e020948059fb06f65f26c831d P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:10.647666 17330 catalog_manager.cc:5484] T da06571e020948059fb06f65f26c831d P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: config changed from index 3 to 4, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:10.652156 20212 tablet_service.cc:1514] Processing DeleteTablet for tablet da06571e020948059fb06f65f26c831d with delete_type TABLET_DATA_TOMBSTONED (TS fdc2ac372492454c85387589a91f63af not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44624
I20240502 01:54:10.653843 24134 ts_tablet_manager.cc:1875] T aa5a653bc143469c8c0e891e1bd58207 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:10.658605 24136 tablet_replica.cc:330] T aa5a653bc143469c8c0e891e1bd58207 P fdc2ac372492454c85387589a91f63af: stopping tablet replica
I20240502 01:54:10.658989 24136 raft_consensus.cc:2229] T aa5a653bc143469c8c0e891e1bd58207 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:10.660197 24136 raft_consensus.cc:2258] T aa5a653bc143469c8c0e891e1bd58207 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:10.662618 24136 ts_tablet_manager.cc:1875] T aa5a653bc143469c8c0e891e1bd58207 P fdc2ac372492454c85387589a91f63af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:10.664112 17330 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_extra_table_44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:10.674048 24134 ts_tablet_manager.cc:1888] T aa5a653bc143469c8c0e891e1bd58207 P 3fceccefba274b1dbbd8216af8b0bc29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:10.678550 24135 ts_tablet_manager.cc:1888] T aa5a653bc143469c8c0e891e1bd58207 P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:10.679270 17862 consensus_queue.cc:237] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } }
I20240502 01:54:10.679747 24135 log.cc:1196] T aa5a653bc143469c8c0e891e1bd58207 P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal/wals/aa5a653bc143469c8c0e891e1bd58207
I20240502 01:54:10.682868 24135 ts_tablet_manager.cc:1909] T aa5a653bc143469c8c0e891e1bd58207 P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting consensus metadata
I20240502 01:54:10.683614 24134 log.cc:1196] T aa5a653bc143469c8c0e891e1bd58207 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/wals/aa5a653bc143469c8c0e891e1bd58207
I20240502 01:54:10.684159 24134 ts_tablet_manager.cc:1909] T aa5a653bc143469c8c0e891e1bd58207 P 3fceccefba274b1dbbd8216af8b0bc29: Deleting consensus metadata
I20240502 01:54:10.685456 17321 catalog_manager.cc:4830] TS 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): tablet aa5a653bc143469c8c0e891e1bd58207 (table rebalancer_dynamic_table_new_name [id=49e09fff43434444b9f484c9d0d5562f]) successfully deleted
I20240502 01:54:10.686237 17319 catalog_manager.cc:4830] TS 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): tablet aa5a653bc143469c8c0e891e1bd58207 (table rebalancer_dynamic_table_new_name [id=49e09fff43434444b9f484c9d0d5562f]) successfully deleted
I20240502 01:54:10.687398 24136 ts_tablet_manager.cc:1888] T aa5a653bc143469c8c0e891e1bd58207 P fdc2ac372492454c85387589a91f63af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:10.687613 24136 log.cc:1196] T aa5a653bc143469c8c0e891e1bd58207 P fdc2ac372492454c85387589a91f63af: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal/wals/aa5a653bc143469c8c0e891e1bd58207
I20240502 01:54:10.687839 20233 raft_consensus.cc:1261] T ed1522ec150a46f28fd553f9b563651f P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:10.688258 24136 ts_tablet_manager.cc:1909] T aa5a653bc143469c8c0e891e1bd58207 P fdc2ac372492454c85387589a91f63af: Deleting consensus metadata
I20240502 01:54:10.689121 23898 consensus_queue.cc:1035] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:10.689383 24136 tablet_replica.cc:330] T da06571e020948059fb06f65f26c831d P fdc2ac372492454c85387589a91f63af: stopping tablet replica
I20240502 01:54:10.689589 24136 raft_consensus.cc:2229] T da06571e020948059fb06f65f26c831d P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:10.689783 24136 raft_consensus.cc:2258] T da06571e020948059fb06f65f26c831d P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:10.690400 17322 catalog_manager.cc:4830] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): tablet aa5a653bc143469c8c0e891e1bd58207 (table rebalancer_dynamic_table_new_name [id=49e09fff43434444b9f484c9d0d5562f]) successfully deleted
I20240502 01:54:10.690912 24136 ts_tablet_manager.cc:1875] T da06571e020948059fb06f65f26c831d P fdc2ac372492454c85387589a91f63af: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 01:54:10.693784 17802 consensus_peers.cc:485] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 -> Peer 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): Couldn't send request to peer 6a36c952c51542c0bf7ea3ec2857a9f9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ed1522ec150a46f28fd553f9b563651f. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:10.695015 17476 raft_consensus.cc:1261] T ed1522ec150a46f28fd553f9b563651f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:10.696396 21667 consensus_queue.cc:1035] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:10.699172 24136 ts_tablet_manager.cc:1888] T da06571e020948059fb06f65f26c831d P fdc2ac372492454c85387589a91f63af: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:54:10.699373 24136 log.cc:1196] T da06571e020948059fb06f65f26c831d P fdc2ac372492454c85387589a91f63af: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal/wals/da06571e020948059fb06f65f26c831d
I20240502 01:54:10.699615 23586 raft_consensus.cc:2932] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } } }
I20240502 01:54:10.701076 17322 catalog_manager.cc:4830] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): tablet da06571e020948059fb06f65f26c831d (table rebalancer_extra_table_32 [id=6cf4b74847bf491c93c43661f9c6d7a5]) successfully deleted
I20240502 01:54:10.701274 20233 raft_consensus.cc:2932] T ed1522ec150a46f28fd553f9b563651f P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } } }
I20240502 01:54:10.709771 17326 catalog_manager.cc:5484] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 reported cstate change: config changed from index 6 to 7, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:10.711858 17328 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:54:10.720696 17585 tablet_service.cc:1467] Processing CreateTablet for tablet 164e04f7c9454c189f6907b319023d99 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=bc3f32f6efe0449a852946df8151762f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:10.721247 17585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 164e04f7c9454c189f6907b319023d99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.721415 17716 tablet_service.cc:1467] Processing CreateTablet for tablet 8c129f9289024ec9898fbbc06b2fb2cd (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=bc3f32f6efe0449a852946df8151762f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:10.721788 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c129f9289024ec9898fbbc06b2fb2cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.724005 20211 tablet_service.cc:1467] Processing CreateTablet for tablet 164e04f7c9454c189f6907b319023d99 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=bc3f32f6efe0449a852946df8151762f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:10.724009 20213 tablet_service.cc:1467] Processing CreateTablet for tablet f6cac9478db94a21a9799b80d24c8dc9 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=bc3f32f6efe0449a852946df8151762f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:10.724464 20213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6cac9478db94a21a9799b80d24c8dc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.725411 20212 tablet_service.cc:1467] Processing CreateTablet for tablet 8c129f9289024ec9898fbbc06b2fb2cd (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=bc3f32f6efe0449a852946df8151762f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:10.725819 20212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c129f9289024ec9898fbbc06b2fb2cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.727303 21667 raft_consensus.cc:1050] T 2152c07e448b4582abff22b034363c70 P bb31bceba76d466db24b9341c04900c3: attempting to promote NON_VOTER 4da690c0fa13407a85c0689ce8de9ade to VOTER
I20240502 01:54:10.728152 21667 consensus_queue.cc:237] T 2152c07e448b4582abff22b034363c70 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } }
I20240502 01:54:10.729382 17846 tablet_service.cc:1467] Processing CreateTablet for tablet 164e04f7c9454c189f6907b319023d99 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=bc3f32f6efe0449a852946df8151762f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:10.729779 17846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 164e04f7c9454c189f6907b319023d99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.730203 17587 tablet_service.cc:1467] Processing CreateTablet for tablet f6cac9478db94a21a9799b80d24c8dc9 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=bc3f32f6efe0449a852946df8151762f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:10.731324 17587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6cac9478db94a21a9799b80d24c8dc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.732847 17847 tablet_service.cc:1467] Processing CreateTablet for tablet 8c129f9289024ec9898fbbc06b2fb2cd (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=bc3f32f6efe0449a852946df8151762f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:10.733623 17847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c129f9289024ec9898fbbc06b2fb2cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.733681 20610 tablet_service.cc:1467] Processing CreateTablet for tablet f6cac9478db94a21a9799b80d24c8dc9 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=bc3f32f6efe0449a852946df8151762f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:10.734134 20610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6cac9478db94a21a9799b80d24c8dc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.739053 20211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 164e04f7c9454c189f6907b319023d99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.742662 23766 raft_consensus.cc:1050] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29: attempting to promote NON_VOTER dfa1152987644bd781b46c7e42a7a685 to VOTER
I20240502 01:54:10.743494 23766 consensus_queue.cc:237] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:10.746487 17735 raft_consensus.cc:1261] T 4aefc61cb95f4e949e7a23a1c677749e P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:10.747467 20628 raft_consensus.cc:1261] T 4aefc61cb95f4e949e7a23a1c677749e P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:10.748520 23807 consensus_queue.cc:1035] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:10.749053 23807 consensus_queue.cc:1035] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:10.750069 17845 tablet_service.cc:1467] Processing CreateTablet for tablet 59ce542c545f4ba590e41690e8df5fc6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=eb791abbdca04194aa499cb7e38e1dc6]), partition=
I20240502 01:54:10.750299 17862 raft_consensus.cc:1261] T 4aefc61cb95f4e949e7a23a1c677749e P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:10.750422 17845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59ce542c545f4ba590e41690e8df5fc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.750993 23767 consensus_queue.cc:1035] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:10.760174 17456 tablet_service.cc:1467] Processing CreateTablet for tablet 59ce542c545f4ba590e41690e8df5fc6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=eb791abbdca04194aa499cb7e38e1dc6]), partition=
I20240502 01:54:10.760512 17456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59ce542c545f4ba590e41690e8df5fc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.769603 20211 tablet_service.cc:1467] Processing CreateTablet for tablet 59ce542c545f4ba590e41690e8df5fc6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=eb791abbdca04194aa499cb7e38e1dc6]), partition=
I20240502 01:54:10.770648 20211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59ce542c545f4ba590e41690e8df5fc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.784242 23807 raft_consensus.cc:2932] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:10.784802 17473 raft_consensus.cc:1261] T 2152c07e448b4582abff22b034363c70 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEARNER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:10.785856 23898 consensus_queue.cc:1035] T 2152c07e448b4582abff22b034363c70 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:10.790450 20366 raft_consensus.cc:1261] T 2152c07e448b4582abff22b034363c70 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:10.791051 23898 consensus_queue.cc:1035] T 2152c07e448b4582abff22b034363c70 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:10.794267 20625 raft_consensus.cc:1261] T 2152c07e448b4582abff22b034363c70 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:10.795100 20628 raft_consensus.cc:2932] T 4aefc61cb95f4e949e7a23a1c677749e P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:10.796779 17737 raft_consensus.cc:2932] T 4aefc61cb95f4e949e7a23a1c677749e P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:10.799728 17865 raft_consensus.cc:2932] T 4aefc61cb95f4e949e7a23a1c677749e P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:10.800961 21667 consensus_queue.cc:1035] T 2152c07e448b4582abff22b034363c70 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:10.802436 24143 tablet_bootstrap.cc:492] T 8c129f9289024ec9898fbbc06b2fb2cd P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:54:10.802665 24144 tablet_bootstrap.cc:492] T 8c129f9289024ec9898fbbc06b2fb2cd P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:54:10.804206 24143 tablet_bootstrap.cc:654] T 8c129f9289024ec9898fbbc06b2fb2cd P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:10.798918 17328 catalog_manager.cc:5484] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: config changed from index 7 to 8, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:10.819518 24144 tablet_bootstrap.cc:654] T 8c129f9289024ec9898fbbc06b2fb2cd P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:10.821478 24141 tablet_bootstrap.cc:492] T 164e04f7c9454c189f6907b319023d99 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:54:10.828029 24141 tablet_bootstrap.cc:654] T 164e04f7c9454c189f6907b319023d99 P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:10.829562 24144 tablet_bootstrap.cc:492] T 8c129f9289024ec9898fbbc06b2fb2cd P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:54:10.830034 24144 ts_tablet_manager.cc:1367] T 8c129f9289024ec9898fbbc06b2fb2cd P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.003s
I20240502 01:54:10.831537 24144 raft_consensus.cc:348] T 8c129f9289024ec9898fbbc06b2fb2cd P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:10.832738 24144 raft_consensus.cc:374] T 8c129f9289024ec9898fbbc06b2fb2cd P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:10.832895 24144 raft_consensus.cc:729] T 8c129f9289024ec9898fbbc06b2fb2cd P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:54:10.833225 24144 consensus_queue.cc:260] T 8c129f9289024ec9898fbbc06b2fb2cd P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:10.834944 24143 tablet_bootstrap.cc:492] T 8c129f9289024ec9898fbbc06b2fb2cd P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:54:10.835172 24143 ts_tablet_manager.cc:1367] T 8c129f9289024ec9898fbbc06b2fb2cd P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.000s
I20240502 01:54:10.835894 24143 raft_consensus.cc:348] T 8c129f9289024ec9898fbbc06b2fb2cd P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:10.836220 24143 raft_consensus.cc:374] T 8c129f9289024ec9898fbbc06b2fb2cd P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:10.836325 24143 raft_consensus.cc:729] T 8c129f9289024ec9898fbbc06b2fb2cd P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:54:10.836716 24143 consensus_queue.cc:260] T 8c129f9289024ec9898fbbc06b2fb2cd P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:10.838671 24143 ts_tablet_manager.cc:1398] T 8c129f9289024ec9898fbbc06b2fb2cd P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:54:10.839280 24144 ts_tablet_manager.cc:1398] T 8c129f9289024ec9898fbbc06b2fb2cd P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240502 01:54:10.841336 24144 tablet_bootstrap.cc:492] T 164e04f7c9454c189f6907b319023d99 P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:54:10.844553 24144 tablet_bootstrap.cc:654] T 164e04f7c9454c189f6907b319023d99 P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:10.849850 24141 tablet_bootstrap.cc:492] T 164e04f7c9454c189f6907b319023d99 P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:54:10.850069 24141 ts_tablet_manager.cc:1367] T 164e04f7c9454c189f6907b319023d99 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.003s
I20240502 01:54:10.850893 24141 raft_consensus.cc:348] T 164e04f7c9454c189f6907b319023d99 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:10.851493 24141 raft_consensus.cc:374] T 164e04f7c9454c189f6907b319023d99 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:10.851712 24141 raft_consensus.cc:729] T 164e04f7c9454c189f6907b319023d99 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:54:10.852046 24141 consensus_queue.cc:260] T 164e04f7c9454c189f6907b319023d99 P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:10.855952 24141 ts_tablet_manager.cc:1398] T 164e04f7c9454c189f6907b319023d99 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 01:54:10.856279 24141 tablet_bootstrap.cc:492] T f6cac9478db94a21a9799b80d24c8dc9 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:54:10.858304 24147 tablet_bootstrap.cc:492] T f6cac9478db94a21a9799b80d24c8dc9 P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:54:10.863619 24141 tablet_bootstrap.cc:654] T f6cac9478db94a21a9799b80d24c8dc9 P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:10.865656 24153 tablet_bootstrap.cc:492] T 59ce542c545f4ba590e41690e8df5fc6 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:54:10.867015 24153 tablet_bootstrap.cc:654] T 59ce542c545f4ba590e41690e8df5fc6 P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:10.868477 24150 tablet_bootstrap.cc:492] T 8c129f9289024ec9898fbbc06b2fb2cd P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:54:10.872495 21667 raft_consensus.cc:2932] T 2152c07e448b4582abff22b034363c70 P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:10.874621 24150 tablet_bootstrap.cc:654] T 8c129f9289024ec9898fbbc06b2fb2cd P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:10.877674 24141 tablet_bootstrap.cc:492] T f6cac9478db94a21a9799b80d24c8dc9 P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:54:10.877873 24141 ts_tablet_manager.cc:1367] T f6cac9478db94a21a9799b80d24c8dc9 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.003s
I20240502 01:54:10.879287 20362 raft_consensus.cc:1261] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:54:10.879438 24141 raft_consensus.cc:348] T f6cac9478db94a21a9799b80d24c8dc9 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:10.879630 24141 raft_consensus.cc:374] T f6cac9478db94a21a9799b80d24c8dc9 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:10.879699 24141 raft_consensus.cc:729] T f6cac9478db94a21a9799b80d24c8dc9 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:54:10.879931 24141 consensus_queue.cc:260] T f6cac9478db94a21a9799b80d24c8dc9 P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:10.880235 23961 consensus_queue.cc:1035] T a212c17ba64842c99b3d6a78a3b7cc8a P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:10.880507 24141 ts_tablet_manager.cc:1398] T f6cac9478db94a21a9799b80d24c8dc9 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:54:10.888924 24147 tablet_bootstrap.cc:654] T f6cac9478db94a21a9799b80d24c8dc9 P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:10.893684 20366 raft_consensus.cc:2932] T 2152c07e448b4582abff22b034363c70 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:10.896389 24153 tablet_bootstrap.cc:492] T 59ce542c545f4ba590e41690e8df5fc6 P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:54:10.897625 17474 raft_consensus.cc:2932] T 2152c07e448b4582abff22b034363c70 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:10.902544 24150 tablet_bootstrap.cc:492] T 8c129f9289024ec9898fbbc06b2fb2cd P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:54:10.906394 20625 raft_consensus.cc:2932] T 2152c07e448b4582abff22b034363c70 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:10.911937 24146 raft_consensus.cc:482] T 164e04f7c9454c189f6907b319023d99 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:10.912103 24146 raft_consensus.cc:504] T 164e04f7c9454c189f6907b319023d99 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:10.912947 24146 leader_election.cc:288] T 164e04f7c9454c189f6907b319023d99 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:54:10.898910 24153 ts_tablet_manager.cc:1367] T 59ce542c545f4ba590e41690e8df5fc6 P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.001s
I20240502 01:54:10.914870 24153 raft_consensus.cc:348] T 59ce542c545f4ba590e41690e8df5fc6 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:10.915518 20233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "164e04f7c9454c189f6907b319023d99" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:54:10.916124 24176 ts_tablet_manager.cc:902] T 4bb42883b1a0475b863d008bd01fa70a P 2f306549407b4bb281ccf2994035e4f1: Initiating tablet copy from peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
W20240502 01:54:10.916983 17543 leader_election.cc:341] T 164e04f7c9454c189f6907b319023d99 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:10.917510 24153 raft_consensus.cc:374] T 59ce542c545f4ba590e41690e8df5fc6 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:10.917663 24153 raft_consensus.cc:729] T 59ce542c545f4ba590e41690e8df5fc6 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:54:10.916519 17865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "164e04f7c9454c189f6907b319023d99" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:54:10.918212 24150 ts_tablet_manager.cc:1367] T 8c129f9289024ec9898fbbc06b2fb2cd P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.050s user 0.003s sys 0.000s
W20240502 01:54:10.919157 17543 leader_election.cc:341] T 164e04f7c9454c189f6907b319023d99 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:10.919294 17543 leader_election.cc:302] T 164e04f7c9454c189f6907b319023d99 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29; no voters: bb31bceba76d466db24b9341c04900c3, fdc2ac372492454c85387589a91f63af
I20240502 01:54:10.917964 24153 consensus_queue.cc:260] T 59ce542c545f4ba590e41690e8df5fc6 P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:10.920018 24146 raft_consensus.cc:2726] T 164e04f7c9454c189f6907b319023d99 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:54:10.919224 24150 raft_consensus.cc:348] T 8c129f9289024ec9898fbbc06b2fb2cd P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:10.920459 17476 raft_consensus.cc:2932] T ed1522ec150a46f28fd553f9b563651f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } } }
I20240502 01:54:10.920877 24150 raft_consensus.cc:374] T 8c129f9289024ec9898fbbc06b2fb2cd P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:10.920953 24150 raft_consensus.cc:729] T 8c129f9289024ec9898fbbc06b2fb2cd P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:54:10.921778 24153 ts_tablet_manager.cc:1398] T 59ce542c545f4ba590e41690e8df5fc6 P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20240502 01:54:10.921998 24150 consensus_queue.cc:260] T 8c129f9289024ec9898fbbc06b2fb2cd P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:10.924619 24150 ts_tablet_manager.cc:1398] T 8c129f9289024ec9898fbbc06b2fb2cd P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:54:10.925405 24150 tablet_bootstrap.cc:492] T 164e04f7c9454c189f6907b319023d99 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:54:10.925331 17326 catalog_manager.cc:5484] T 2152c07e448b4582abff22b034363c70 P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:10.932369 24150 tablet_bootstrap.cc:654] T 164e04f7c9454c189f6907b319023d99 P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:10.947784 24147 tablet_bootstrap.cc:492] T f6cac9478db94a21a9799b80d24c8dc9 P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:54:10.948179 24150 tablet_bootstrap.cc:492] T 164e04f7c9454c189f6907b319023d99 P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:54:10.948428 24150 ts_tablet_manager.cc:1367] T 164e04f7c9454c189f6907b319023d99 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.003s
I20240502 01:54:10.949594 24150 raft_consensus.cc:348] T 164e04f7c9454c189f6907b319023d99 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:10.949896 24150 raft_consensus.cc:374] T 164e04f7c9454c189f6907b319023d99 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:10.950013 24150 raft_consensus.cc:729] T 164e04f7c9454c189f6907b319023d99 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:54:10.950338 24150 consensus_queue.cc:260] T 164e04f7c9454c189f6907b319023d99 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:10.951059 24150 ts_tablet_manager.cc:1398] T 164e04f7c9454c189f6907b319023d99 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:10.951406 24150 tablet_bootstrap.cc:492] T 59ce542c545f4ba590e41690e8df5fc6 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:54:10.962541 24150 tablet_bootstrap.cc:654] T 59ce542c545f4ba590e41690e8df5fc6 P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:10.958582 24147 ts_tablet_manager.cc:1367] T f6cac9478db94a21a9799b80d24c8dc9 P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.100s user 0.005s sys 0.000s
I20240502 01:54:10.965886 24144 tablet_bootstrap.cc:492] T 164e04f7c9454c189f6907b319023d99 P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:54:10.966181 24144 ts_tablet_manager.cc:1367] T 164e04f7c9454c189f6907b319023d99 P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.125s user 0.009s sys 0.019s
I20240502 01:54:10.967101 24144 raft_consensus.cc:348] T 164e04f7c9454c189f6907b319023d99 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:10.967878 24147 raft_consensus.cc:348] T f6cac9478db94a21a9799b80d24c8dc9 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:10.968672 17862 consensus_queue.cc:237] T 2152c07e448b4582abff22b034363c70 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } }
I20240502 01:54:10.979097 24144 raft_consensus.cc:374] T 164e04f7c9454c189f6907b319023d99 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:10.979413 24144 raft_consensus.cc:729] T 164e04f7c9454c189f6907b319023d99 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:54:10.979976 24144 consensus_queue.cc:260] T 164e04f7c9454c189f6907b319023d99 P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:10.982602 24144 ts_tablet_manager.cc:1398] T 164e04f7c9454c189f6907b319023d99 P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20240502 01:54:10.987737 24144 tablet_bootstrap.cc:492] T f6cac9478db94a21a9799b80d24c8dc9 P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:54:10.989182 17476 raft_consensus.cc:1261] T 2152c07e448b4582abff22b034363c70 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:10.989758 21667 consensus_queue.cc:1035] T 2152c07e448b4582abff22b034363c70 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:10.991673 24147 raft_consensus.cc:374] T f6cac9478db94a21a9799b80d24c8dc9 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:10.992364 24144 tablet_bootstrap.cc:654] T f6cac9478db94a21a9799b80d24c8dc9 P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:10.974736 20628 raft_consensus.cc:1261] T 2152c07e448b4582abff22b034363c70 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:10.973526 20625 raft_consensus.cc:1261] T a212c17ba64842c99b3d6a78a3b7cc8a P f19cc6cb1e76460aa6711e49a5775091 [term 2 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:54:10.993309 21667 consensus_queue.cc:1035] T 2152c07e448b4582abff22b034363c70 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:10.993500 23961 consensus_queue.cc:1035] T a212c17ba64842c99b3d6a78a3b7cc8a P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:10.994215 24147 raft_consensus.cc:729] T f6cac9478db94a21a9799b80d24c8dc9 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:54:10.994493 24147 consensus_queue.cc:260] T f6cac9478db94a21a9799b80d24c8dc9 P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:10.995091 24147 ts_tablet_manager.cc:1398] T f6cac9478db94a21a9799b80d24c8dc9 P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.032s user 0.001s sys 0.000s
I20240502 01:54:10.996727 24166 raft_consensus.cc:2932] T 2152c07e448b4582abff22b034363c70 P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:10.997645 20625 raft_consensus.cc:2932] T 2152c07e448b4582abff22b034363c70 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:11.001091 24181 ts_tablet_manager.cc:902] T cf0237a4c65b495ba892871dd850bed9 P bb31bceba76d466db24b9341c04900c3: Initiating tablet copy from peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:54:11.002650 17322 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2152c07e448b4582abff22b034363c70 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:11.007313 17326 catalog_manager.cc:5484] T 2152c07e448b4582abff22b034363c70 P bb31bceba76d466db24b9341c04900c3 reported cstate change: config changed from index 3 to 4, VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) evicted. New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:11.009800 24144 tablet_bootstrap.cc:492] T f6cac9478db94a21a9799b80d24c8dc9 P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:54:11.010201 24144 ts_tablet_manager.cc:1367] T f6cac9478db94a21a9799b80d24c8dc9 P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.003s
I20240502 01:54:11.012862 24144 raft_consensus.cc:348] T f6cac9478db94a21a9799b80d24c8dc9 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:11.015074 24144 raft_consensus.cc:374] T f6cac9478db94a21a9799b80d24c8dc9 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:11.015362 24144 raft_consensus.cc:729] T f6cac9478db94a21a9799b80d24c8dc9 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:54:11.015936 24144 consensus_queue.cc:260] T f6cac9478db94a21a9799b80d24c8dc9 P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:11.025735 20347 tablet_service.cc:1514] Processing DeleteTablet for tablet 2152c07e448b4582abff22b034363c70 with delete_type TABLET_DATA_TOMBSTONED (TS 6a36c952c51542c0bf7ea3ec2857a9f9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40196
I20240502 01:54:11.026155 24135 tablet_replica.cc:330] T 2152c07e448b4582abff22b034363c70 P 6a36c952c51542c0bf7ea3ec2857a9f9: stopping tablet replica
I20240502 01:54:11.026387 24135 raft_consensus.cc:2229] T 2152c07e448b4582abff22b034363c70 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:11.026595 24135 raft_consensus.cc:2258] T 2152c07e448b4582abff22b034363c70 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:11.033313 24135 ts_tablet_manager.cc:1875] T 2152c07e448b4582abff22b034363c70 P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:11.037649 24176 tablet_copy_client.cc:308] T 4bb42883b1a0475b863d008bd01fa70a P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.68:44083
I20240502 01:54:11.043628 17476 raft_consensus.cc:1261] T a212c17ba64842c99b3d6a78a3b7cc8a P 4da690c0fa13407a85c0689ce8de9ade [term 2 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:54:11.044227 23967 consensus_queue.cc:1035] T a212c17ba64842c99b3d6a78a3b7cc8a P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:11.053699 17887 tablet_copy_service.cc:140] P bb31bceba76d466db24b9341c04900c3: Received BeginTabletCopySession request for tablet 4bb42883b1a0475b863d008bd01fa70a from peer 2f306549407b4bb281ccf2994035e4f1 ({username='slave'} at 127.11.42.71:48285)
I20240502 01:54:11.053923 17887 tablet_copy_service.cc:161] P bb31bceba76d466db24b9341c04900c3: Beginning new tablet copy session on tablet 4bb42883b1a0475b863d008bd01fa70a from peer 2f306549407b4bb281ccf2994035e4f1 at {username='slave'} at 127.11.42.71:48285: session id = 2f306549407b4bb281ccf2994035e4f1-4bb42883b1a0475b863d008bd01fa70a
I20240502 01:54:11.053508 24135 ts_tablet_manager.cc:1888] T 2152c07e448b4582abff22b034363c70 P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:54:11.054420 24135 log.cc:1196] T 2152c07e448b4582abff22b034363c70 P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal/wals/2152c07e448b4582abff22b034363c70
I20240502 01:54:11.055358 17321 catalog_manager.cc:4830] TS 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): tablet 2152c07e448b4582abff22b034363c70 (table rebalancer_extra_table_22 [id=2947767c2c5e4b8ea491bf140246cfa4]) successfully deleted
I20240502 01:54:11.058436 24178 raft_consensus.cc:482] T 164e04f7c9454c189f6907b319023d99 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:11.058460 24181 tablet_copy_client.cc:308] T cf0237a4c65b495ba892871dd850bed9 P bb31bceba76d466db24b9341c04900c3: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.72:42411
I20240502 01:54:11.058606 24178 raft_consensus.cc:504] T 164e04f7c9454c189f6907b319023d99 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:11.059350 24178 leader_election.cc:288] T 164e04f7c9454c189f6907b319023d99 P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:54:11.059962 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "164e04f7c9454c189f6907b319023d99" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:54:11.060201 17604 raft_consensus.cc:2454] T 164e04f7c9454c189f6907b319023d99 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 0.
I20240502 01:54:11.060695 20165 leader_election.cc:302] T 164e04f7c9454c189f6907b319023d99 P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:54:11.061221 24178 raft_consensus.cc:2781] T 164e04f7c9454c189f6907b319023d99 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:11.061336 24178 raft_consensus.cc:482] T 164e04f7c9454c189f6907b319023d99 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:11.061415 24178 raft_consensus.cc:3037] T 164e04f7c9454c189f6907b319023d99 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:11.063782 24172 raft_consensus.cc:482] T 8c129f9289024ec9898fbbc06b2fb2cd P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:11.063926 24172 raft_consensus.cc:504] T 8c129f9289024ec9898fbbc06b2fb2cd P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:11.064752 24172 leader_election.cc:288] T 8c129f9289024ec9898fbbc06b2fb2cd P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:54:11.065296 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c129f9289024ec9898fbbc06b2fb2cd" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:54:11.065508 17735 raft_consensus.cc:2454] T 8c129f9289024ec9898fbbc06b2fb2cd P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 0.
I20240502 01:54:11.066103 20167 leader_election.cc:302] T 8c129f9289024ec9898fbbc06b2fb2cd P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dfa1152987644bd781b46c7e42a7a685, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:54:11.066383 24172 raft_consensus.cc:2781] T 8c129f9289024ec9898fbbc06b2fb2cd P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:11.066478 24172 raft_consensus.cc:482] T 8c129f9289024ec9898fbbc06b2fb2cd P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:11.066552 24172 raft_consensus.cc:3037] T 8c129f9289024ec9898fbbc06b2fb2cd P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:11.064581 24178 raft_consensus.cc:504] T 164e04f7c9454c189f6907b319023d99 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:11.071166 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "164e04f7c9454c189f6907b319023d99" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:54:11.071400 17604 raft_consensus.cc:3037] T 164e04f7c9454c189f6907b319023d99 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:11.074270 17604 raft_consensus.cc:2454] T 164e04f7c9454c189f6907b319023d99 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 1.
I20240502 01:54:11.074744 20165 leader_election.cc:302] T 164e04f7c9454c189f6907b319023d99 P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:54:11.075057 23637 raft_consensus.cc:2781] T 164e04f7c9454c189f6907b319023d99 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:11.077761 24172 raft_consensus.cc:504] T 8c129f9289024ec9898fbbc06b2fb2cd P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:11.078392 24172 leader_election.cc:288] T 8c129f9289024ec9898fbbc06b2fb2cd P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 election: Requested vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:54:11.079105 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c129f9289024ec9898fbbc06b2fb2cd" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:54:11.079303 17735 raft_consensus.cc:3037] T 8c129f9289024ec9898fbbc06b2fb2cd P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:11.081977 17735 raft_consensus.cc:2454] T 8c129f9289024ec9898fbbc06b2fb2cd P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 1.
I20240502 01:54:11.083259 20167 leader_election.cc:302] T 8c129f9289024ec9898fbbc06b2fb2cd P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dfa1152987644bd781b46c7e42a7a685, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:54:11.083675 24172 raft_consensus.cc:2781] T 8c129f9289024ec9898fbbc06b2fb2cd P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:11.083822 24172 raft_consensus.cc:686] T 8c129f9289024ec9898fbbc06b2fb2cd P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Becoming Leader. State: Replica: fdc2ac372492454c85387589a91f63af, State: Running, Role: LEADER
I20240502 01:54:11.084574 24172 consensus_queue.cc:237] T 8c129f9289024ec9898fbbc06b2fb2cd P fdc2ac372492454c85387589a91f63af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:11.087764 24178 leader_election.cc:288] T 164e04f7c9454c189f6907b319023d99 P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 election: Requested vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:54:11.090600 17864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c129f9289024ec9898fbbc06b2fb2cd" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:54:11.090700 17863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c129f9289024ec9898fbbc06b2fb2cd" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:54:11.090808 17864 raft_consensus.cc:3037] T 8c129f9289024ec9898fbbc06b2fb2cd P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:11.091122 17867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "164e04f7c9454c189f6907b319023d99" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:54:11.091289 17867 raft_consensus.cc:3037] T 164e04f7c9454c189f6907b319023d99 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:11.094236 17865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "164e04f7c9454c189f6907b319023d99" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:54:11.095292 20650 tablet_copy_service.cc:140] P f19cc6cb1e76460aa6711e49a5775091: Received BeginTabletCopySession request for tablet cf0237a4c65b495ba892871dd850bed9 from peer bb31bceba76d466db24b9341c04900c3 ({username='slave'} at 127.11.42.68:36611)
I20240502 01:54:11.095458 20650 tablet_copy_service.cc:161] P f19cc6cb1e76460aa6711e49a5775091: Beginning new tablet copy session on tablet cf0237a4c65b495ba892871dd850bed9 from peer bb31bceba76d466db24b9341c04900c3 at {username='slave'} at 127.11.42.68:36611: session id = bb31bceba76d466db24b9341c04900c3-cf0237a4c65b495ba892871dd850bed9
I20240502 01:54:11.099956 24150 tablet_bootstrap.cc:492] T 59ce542c545f4ba590e41690e8df5fc6 P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:54:11.100126 24150 ts_tablet_manager.cc:1367] T 59ce542c545f4ba590e41690e8df5fc6 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.149s user 0.060s sys 0.042s
I20240502 01:54:11.100191 17867 raft_consensus.cc:2454] T 164e04f7c9454c189f6907b319023d99 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 1.
I20240502 01:54:11.099954 17476 raft_consensus.cc:2932] T 2152c07e448b4582abff22b034363c70 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:11.103588 24150 raft_consensus.cc:348] T 59ce542c545f4ba590e41690e8df5fc6 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:11.103788 24150 raft_consensus.cc:374] T 59ce542c545f4ba590e41690e8df5fc6 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:11.103874 24150 raft_consensus.cc:729] T 59ce542c545f4ba590e41690e8df5fc6 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:54:11.104089 24150 consensus_queue.cc:260] T 59ce542c545f4ba590e41690e8df5fc6 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:11.104614 17887 tablet_copy_source_session.cc:205] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:11.104661 20650 tablet_copy_source_session.cc:205] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:11.104800 24150 ts_tablet_manager.cc:1398] T 59ce542c545f4ba590e41690e8df5fc6 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20240502 01:54:11.107796 24176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bb42883b1a0475b863d008bd01fa70a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.109879 24181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf0237a4c65b495ba892871dd850bed9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.099251 17864 raft_consensus.cc:2454] T 8c129f9289024ec9898fbbc06b2fb2cd P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 1.
I20240502 01:54:11.112798 24176 tablet_copy_client.cc:783] T 4bb42883b1a0475b863d008bd01fa70a P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:11.113755 24199 ts_tablet_manager.cc:902] T 472d7827b3074236a81a3ede098f6c0a P 3fceccefba274b1dbbd8216af8b0bc29: Initiating tablet copy from peer 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:54:11.113751 24176 tablet_copy_client.cc:647] T 4bb42883b1a0475b863d008bd01fa70a P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:11.128077 24176 tablet_copy_client.cc:523] T 4bb42883b1a0475b863d008bd01fa70a P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:11.128356 24181 tablet_copy_client.cc:783] T cf0237a4c65b495ba892871dd850bed9 P bb31bceba76d466db24b9341c04900c3: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:11.128630 24181 tablet_copy_client.cc:647] T cf0237a4c65b495ba892871dd850bed9 P bb31bceba76d466db24b9341c04900c3: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:11.130530 24144 ts_tablet_manager.cc:1398] T f6cac9478db94a21a9799b80d24c8dc9 P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.120s user 0.002s sys 0.000s
I20240502 01:54:11.139689 24181 tablet_copy_client.cc:523] T cf0237a4c65b495ba892871dd850bed9 P bb31bceba76d466db24b9341c04900c3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:11.142048 17735 consensus_queue.cc:237] T a212c17ba64842c99b3d6a78a3b7cc8a P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:11.143848 24176 tablet_bootstrap.cc:492] T 4bb42883b1a0475b863d008bd01fa70a P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:54:11.144050 20628 raft_consensus.cc:1261] T a212c17ba64842c99b3d6a78a3b7cc8a P f19cc6cb1e76460aa6711e49a5775091 [term 2 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 01:54:11.148211 23967 consensus_queue.cc:1035] T a212c17ba64842c99b3d6a78a3b7cc8a P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:11.150190 17476 raft_consensus.cc:1261] T a212c17ba64842c99b3d6a78a3b7cc8a P 4da690c0fa13407a85c0689ce8de9ade [term 2 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 01:54:11.150944 23967 consensus_queue.cc:1035] T a212c17ba64842c99b3d6a78a3b7cc8a P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:11.153714 23961 raft_consensus.cc:2932] T a212c17ba64842c99b3d6a78a3b7cc8a P dfa1152987644bd781b46c7e42a7a685 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:11.160873 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a212c17ba64842c99b3d6a78a3b7cc8a with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:11.171486 17328 catalog_manager.cc:5484] T a212c17ba64842c99b3d6a78a3b7cc8a P dfa1152987644bd781b46c7e42a7a685 reported cstate change: config changed from index 6 to 8, VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) evicted. New cstate: current_term: 2 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:11.176849 20347 tablet_service.cc:1514] Processing DeleteTablet for tablet a212c17ba64842c99b3d6a78a3b7cc8a with delete_type TABLET_DATA_TOMBSTONED (TS 6a36c952c51542c0bf7ea3ec2857a9f9 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:40196
I20240502 01:54:11.178820 17476 raft_consensus.cc:2932] T a212c17ba64842c99b3d6a78a3b7cc8a P 4da690c0fa13407a85c0689ce8de9ade [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:11.180155 24181 tablet_bootstrap.cc:492] T cf0237a4c65b495ba892871dd850bed9 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:54:11.132872 23637 raft_consensus.cc:686] T 164e04f7c9454c189f6907b319023d99 P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Becoming Leader. State: Replica: fdc2ac372492454c85387589a91f63af, State: Running, Role: LEADER
I20240502 01:54:11.184463 23637 consensus_queue.cc:237] T 164e04f7c9454c189f6907b319023d99 P fdc2ac372492454c85387589a91f63af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:11.184473 20628 raft_consensus.cc:2932] T a212c17ba64842c99b3d6a78a3b7cc8a P f19cc6cb1e76460aa6711e49a5775091 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:11.197384 24135 tablet_replica.cc:330] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9: stopping tablet replica
I20240502 01:54:11.197824 24135 raft_consensus.cc:2229] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:11.198017 24135 raft_consensus.cc:2258] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:11.199600 24135 ts_tablet_manager.cc:1875] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:11.206651 24135 ts_tablet_manager.cc:1888] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240502 01:54:11.207051 24135 log.cc:1196] T a212c17ba64842c99b3d6a78a3b7cc8a P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal/wals/a212c17ba64842c99b3d6a78a3b7cc8a
I20240502 01:54:11.207856 17321 catalog_manager.cc:4830] TS 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): tablet a212c17ba64842c99b3d6a78a3b7cc8a (table rebalancer_extra_table_4 [id=dca50b50b02e4a8c8b90e0de6f305bd8]) successfully deleted
I20240502 01:54:11.219925 20628 consensus_queue.cc:237] T fbd94103cae04473ba81709da39d7ed8 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:54:11.225772 24144 tablet_bootstrap.cc:492] T 59ce542c545f4ba590e41690e8df5fc6 P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:54:11.226186 17334 catalog_manager.cc:5484] T 164e04f7c9454c189f6907b319023d99 P fdc2ac372492454c85387589a91f63af reported cstate change: term changed from 0 to 1, leader changed from <none> to fdc2ac372492454c85387589a91f63af (127.11.42.69). New cstate: current_term: 1 leader_uuid: "fdc2ac372492454c85387589a91f63af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:11.227864 24144 tablet_bootstrap.cc:654] T 59ce542c545f4ba590e41690e8df5fc6 P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:11.236833 17334 catalog_manager.cc:5484] T 8c129f9289024ec9898fbbc06b2fb2cd P fdc2ac372492454c85387589a91f63af reported cstate change: term changed from 0 to 1, leader changed from <none> to fdc2ac372492454c85387589a91f63af (127.11.42.69). New cstate: current_term: 1 leader_uuid: "fdc2ac372492454c85387589a91f63af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:11.251303 24144 tablet_bootstrap.cc:492] T 59ce542c545f4ba590e41690e8df5fc6 P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
W20240502 01:54:11.252420 20564 consensus_peers.cc:485] T fbd94103cae04473ba81709da39d7ed8 P f19cc6cb1e76460aa6711e49a5775091 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fbd94103cae04473ba81709da39d7ed8. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:11.254130 20494 raft_consensus.cc:1261] T fbd94103cae04473ba81709da39d7ed8 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:11.255081 24125 consensus_queue.cc:1035] T fbd94103cae04473ba81709da39d7ed8 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:11.254671 20233 raft_consensus.cc:1261] T fbd94103cae04473ba81709da39d7ed8 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:11.256379 24125 consensus_queue.cc:1035] T fbd94103cae04473ba81709da39d7ed8 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:11.257789 23852 raft_consensus.cc:482] T 59ce542c545f4ba590e41690e8df5fc6 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:11.257951 23852 raft_consensus.cc:504] T 59ce542c545f4ba590e41690e8df5fc6 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:11.258899 23852 leader_election.cc:288] T 59ce542c545f4ba590e41690e8df5fc6 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:54:11.261664 17864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59ce542c545f4ba590e41690e8df5fc6" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:54:11.261904 17864 raft_consensus.cc:2454] T 59ce542c545f4ba590e41690e8df5fc6 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 0.
I20240502 01:54:11.262769 17412 leader_election.cc:302] T 59ce542c545f4ba590e41690e8df5fc6 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:54:11.263373 20231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59ce542c545f4ba590e41690e8df5fc6" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
W20240502 01:54:11.263983 17412 leader_election.cc:341] T 59ce542c545f4ba590e41690e8df5fc6 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:11.265187 23852 raft_consensus.cc:2781] T 59ce542c545f4ba590e41690e8df5fc6 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:11.265276 23852 raft_consensus.cc:482] T 59ce542c545f4ba590e41690e8df5fc6 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:11.265336 23852 raft_consensus.cc:3037] T 59ce542c545f4ba590e41690e8df5fc6 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:11.268266 23852 raft_consensus.cc:504] T 59ce542c545f4ba590e41690e8df5fc6 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:11.269866 17864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59ce542c545f4ba590e41690e8df5fc6" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:54:11.270058 17864 raft_consensus.cc:3037] T 59ce542c545f4ba590e41690e8df5fc6 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:11.268956 23852 leader_election.cc:288] T 59ce542c545f4ba590e41690e8df5fc6 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Requested vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:54:11.272771 17864 raft_consensus.cc:2454] T 59ce542c545f4ba590e41690e8df5fc6 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
I20240502 01:54:11.273296 24144 ts_tablet_manager.cc:1367] T 59ce542c545f4ba590e41690e8df5fc6 P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.065s user 0.003s sys 0.002s
I20240502 01:54:11.273727 17412 leader_election.cc:302] T 59ce542c545f4ba590e41690e8df5fc6 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:54:11.273931 20231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59ce542c545f4ba590e41690e8df5fc6" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:54:11.274091 23855 raft_consensus.cc:2781] T 59ce542c545f4ba590e41690e8df5fc6 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election won for term 1
W20240502 01:54:11.274480 17412 leader_election.cc:341] T 59ce542c545f4ba590e41690e8df5fc6 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:11.276348 24144 raft_consensus.cc:348] T 59ce542c545f4ba590e41690e8df5fc6 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:11.277592 24144 raft_consensus.cc:374] T 59ce542c545f4ba590e41690e8df5fc6 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:11.277832 24144 raft_consensus.cc:729] T 59ce542c545f4ba590e41690e8df5fc6 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:54:11.278332 24144 consensus_queue.cc:260] T 59ce542c545f4ba590e41690e8df5fc6 P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:11.279305 24144 ts_tablet_manager.cc:1398] T 59ce542c545f4ba590e41690e8df5fc6 P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 01:54:11.289637 24181 tablet_bootstrap.cc:492] T cf0237a4c65b495ba892871dd850bed9 P bb31bceba76d466db24b9341c04900c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:11.291692 24010 raft_consensus.cc:2932] T fbd94103cae04473ba81709da39d7ed8 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:11.296029 20231 raft_consensus.cc:2932] T fbd94103cae04473ba81709da39d7ed8 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:11.302476 24215 ts_tablet_manager.cc:902] T ed1522ec150a46f28fd553f9b563651f P 6a36c952c51542c0bf7ea3ec2857a9f9: Initiating tablet copy from peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:54:11.303474 23637 raft_consensus.cc:482] T f6cac9478db94a21a9799b80d24c8dc9 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:11.303622 23637 raft_consensus.cc:504] T f6cac9478db94a21a9799b80d24c8dc9 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:11.304236 23637 leader_election.cc:288] T f6cac9478db94a21a9799b80d24c8dc9 P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:54:11.305111 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6cac9478db94a21a9799b80d24c8dc9" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:54:11.305337 17604 raft_consensus.cc:2454] T f6cac9478db94a21a9799b80d24c8dc9 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 0.
I20240502 01:54:11.306093 20165 leader_election.cc:302] T f6cac9478db94a21a9799b80d24c8dc9 P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:54:11.306937 23637 raft_consensus.cc:2781] T f6cac9478db94a21a9799b80d24c8dc9 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:11.307032 23637 raft_consensus.cc:482] T f6cac9478db94a21a9799b80d24c8dc9 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:11.307133 23637 raft_consensus.cc:3037] T f6cac9478db94a21a9799b80d24c8dc9 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:11.307276 20628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6cac9478db94a21a9799b80d24c8dc9" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091" is_pre_election: true
I20240502 01:54:11.307500 20628 raft_consensus.cc:2454] T f6cac9478db94a21a9799b80d24c8dc9 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 0.
I20240502 01:54:11.309662 23637 raft_consensus.cc:504] T f6cac9478db94a21a9799b80d24c8dc9 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:11.310231 23637 leader_election.cc:288] T f6cac9478db94a21a9799b80d24c8dc9 P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 election: Requested vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:54:11.311865 17330 catalog_manager.cc:5484] T fbd94103cae04473ba81709da39d7ed8 P fdc2ac372492454c85387589a91f63af reported cstate change: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:11.312217 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6cac9478db94a21a9799b80d24c8dc9" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:54:11.312424 17604 raft_consensus.cc:3037] T f6cac9478db94a21a9799b80d24c8dc9 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:11.315213 17604 raft_consensus.cc:2454] T f6cac9478db94a21a9799b80d24c8dc9 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 1.
I20240502 01:54:11.316009 20165 leader_election.cc:302] T f6cac9478db94a21a9799b80d24c8dc9 P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:54:11.316581 20628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6cac9478db94a21a9799b80d24c8dc9" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091"
I20240502 01:54:11.316816 20628 raft_consensus.cc:3037] T f6cac9478db94a21a9799b80d24c8dc9 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:11.319382 20628 raft_consensus.cc:2454] T f6cac9478db94a21a9799b80d24c8dc9 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 1.
I20240502 01:54:11.327102 23637 raft_consensus.cc:2781] T f6cac9478db94a21a9799b80d24c8dc9 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:11.327610 23637 raft_consensus.cc:686] T f6cac9478db94a21a9799b80d24c8dc9 P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Becoming Leader. State: Replica: fdc2ac372492454c85387589a91f63af, State: Running, Role: LEADER
I20240502 01:54:11.327963 23637 consensus_queue.cc:237] T f6cac9478db94a21a9799b80d24c8dc9 P fdc2ac372492454c85387589a91f63af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:11.332376 24181 tablet_bootstrap.cc:492] T cf0237a4c65b495ba892871dd850bed9 P bb31bceba76d466db24b9341c04900c3: Bootstrap complete.
I20240502 01:54:11.332707 24181 ts_tablet_manager.cc:1367] T cf0237a4c65b495ba892871dd850bed9 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.153s user 0.008s sys 0.003s
I20240502 01:54:11.333545 24181 raft_consensus.cc:348] T cf0237a4c65b495ba892871dd850bed9 P bb31bceba76d466db24b9341c04900c3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } }
I20240502 01:54:11.335090 23855 raft_consensus.cc:686] T 59ce542c545f4ba590e41690e8df5fc6 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Becoming Leader. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Running, Role: LEADER
I20240502 01:54:11.336038 24181 raft_consensus.cc:729] T cf0237a4c65b495ba892871dd850bed9 P bb31bceba76d466db24b9341c04900c3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: LEARNER
I20240502 01:54:11.336269 24181 consensus_queue.cc:260] T cf0237a4c65b495ba892871dd850bed9 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } }
I20240502 01:54:11.335701 23855 consensus_queue.cc:237] T 59ce542c545f4ba590e41690e8df5fc6 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:11.341792 17334 catalog_manager.cc:5484] T f6cac9478db94a21a9799b80d24c8dc9 P fdc2ac372492454c85387589a91f63af reported cstate change: term changed from 0 to 1, leader changed from <none> to fdc2ac372492454c85387589a91f63af (127.11.42.69). New cstate: current_term: 1 leader_uuid: "fdc2ac372492454c85387589a91f63af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:11.352881 24176 tablet_bootstrap.cc:492] T 4bb42883b1a0475b863d008bd01fa70a P 2f306549407b4bb281ccf2994035e4f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:11.354969 24181 ts_tablet_manager.cc:1398] T cf0237a4c65b495ba892871dd850bed9 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.022s user 0.001s sys 0.000s
I20240502 01:54:11.357487 17864 raft_consensus.cc:1203] T cf0237a4c65b495ba892871dd850bed9 P bb31bceba76d466db24b9341c04900c3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 01:54:11.360190 17330 catalog_manager.cc:5484] T 59ce542c545f4ba590e41690e8df5fc6 P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: term changed from 0 to 1, leader changed from <none> to 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65). New cstate: current_term: 1 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:11.362192 20650 tablet_copy_service.cc:342] P f19cc6cb1e76460aa6711e49a5775091: Request end of tablet copy session bb31bceba76d466db24b9341c04900c3-cf0237a4c65b495ba892871dd850bed9 received from {username='slave'} at 127.11.42.68:36611
I20240502 01:54:11.362362 20650 tablet_copy_service.cc:433] P f19cc6cb1e76460aa6711e49a5775091: ending tablet copy session bb31bceba76d466db24b9341c04900c3-cf0237a4c65b495ba892871dd850bed9 on tablet cf0237a4c65b495ba892871dd850bed9 with peer bb31bceba76d466db24b9341c04900c3
I20240502 01:54:11.367444 24215 tablet_copy_client.cc:308] T ed1522ec150a46f28fd553f9b563651f P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.68:44083
I20240502 01:54:11.368198 17887 tablet_copy_service.cc:140] P bb31bceba76d466db24b9341c04900c3: Received BeginTabletCopySession request for tablet ed1522ec150a46f28fd553f9b563651f from peer 6a36c952c51542c0bf7ea3ec2857a9f9 ({username='slave'} at 127.11.42.70:41203)
I20240502 01:54:11.368342 17887 tablet_copy_service.cc:161] P bb31bceba76d466db24b9341c04900c3: Beginning new tablet copy session on tablet ed1522ec150a46f28fd553f9b563651f from peer 6a36c952c51542c0bf7ea3ec2857a9f9 at {username='slave'} at 127.11.42.70:41203: session id = 6a36c952c51542c0bf7ea3ec2857a9f9-ed1522ec150a46f28fd553f9b563651f
I20240502 01:54:11.371652 17887 tablet_copy_source_session.cc:205] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:11.374622 24215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed1522ec150a46f28fd553f9b563651f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.376518 24176 tablet_bootstrap.cc:492] T 4bb42883b1a0475b863d008bd01fa70a P 2f306549407b4bb281ccf2994035e4f1: Bootstrap complete.
I20240502 01:54:11.376863 24176 ts_tablet_manager.cc:1367] T 4bb42883b1a0475b863d008bd01fa70a P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.233s user 0.047s sys 0.008s
I20240502 01:54:11.377171 24199 tablet_copy_client.cc:308] T 472d7827b3074236a81a3ede098f6c0a P 3fceccefba274b1dbbd8216af8b0bc29: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.70:39155
I20240502 01:54:11.377723 24176 raft_consensus.cc:348] T 4bb42883b1a0475b863d008bd01fa70a P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
I20240502 01:54:11.377491 20494 raft_consensus.cc:2932] T fbd94103cae04473ba81709da39d7ed8 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:11.378008 20387 tablet_copy_service.cc:140] P 6a36c952c51542c0bf7ea3ec2857a9f9: Received BeginTabletCopySession request for tablet 472d7827b3074236a81a3ede098f6c0a from peer 3fceccefba274b1dbbd8216af8b0bc29 ({username='slave'} at 127.11.42.66:36601)
I20240502 01:54:11.378135 20387 tablet_copy_service.cc:161] P 6a36c952c51542c0bf7ea3ec2857a9f9: Beginning new tablet copy session on tablet 472d7827b3074236a81a3ede098f6c0a from peer 3fceccefba274b1dbbd8216af8b0bc29 at {username='slave'} at 127.11.42.66:36601: session id = 3fceccefba274b1dbbd8216af8b0bc29-472d7827b3074236a81a3ede098f6c0a
I20240502 01:54:11.380966 20387 tablet_copy_source_session.cc:205] T 472d7827b3074236a81a3ede098f6c0a P 6a36c952c51542c0bf7ea3ec2857a9f9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:11.388549 24215 tablet_copy_client.cc:783] T ed1522ec150a46f28fd553f9b563651f P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:11.388830 24215 tablet_copy_client.cc:647] T ed1522ec150a46f28fd553f9b563651f P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:11.388912 24199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 472d7827b3074236a81a3ede098f6c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.390491 24215 tablet_copy_client.cc:523] T ed1522ec150a46f28fd553f9b563651f P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:11.392000 24176 raft_consensus.cc:729] T 4bb42883b1a0475b863d008bd01fa70a P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: LEARNER
I20240502 01:54:11.392241 24176 consensus_queue.cc:260] T 4bb42883b1a0475b863d008bd01fa70a P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
I20240502 01:54:11.394228 24215 tablet_bootstrap.cc:492] T ed1522ec150a46f28fd553f9b563651f P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:54:11.396349 24176 ts_tablet_manager.cc:1398] T 4bb42883b1a0475b863d008bd01fa70a P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.019s user 0.000s sys 0.002s
I20240502 01:54:11.397150 24199 tablet_copy_client.cc:783] T 472d7827b3074236a81a3ede098f6c0a P 3fceccefba274b1dbbd8216af8b0bc29: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:11.397405 24199 tablet_copy_client.cc:647] T 472d7827b3074236a81a3ede098f6c0a P 3fceccefba274b1dbbd8216af8b0bc29: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:11.399163 24199 tablet_copy_client.cc:523] T 472d7827b3074236a81a3ede098f6c0a P 3fceccefba274b1dbbd8216af8b0bc29: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:11.400597 17887 tablet_copy_service.cc:342] P bb31bceba76d466db24b9341c04900c3: Request end of tablet copy session 2f306549407b4bb281ccf2994035e4f1-4bb42883b1a0475b863d008bd01fa70a received from {username='slave'} at 127.11.42.71:48285
I20240502 01:54:11.400713 17887 tablet_copy_service.cc:433] P bb31bceba76d466db24b9341c04900c3: ending tablet copy session 2f306549407b4bb281ccf2994035e4f1-4bb42883b1a0475b863d008bd01fa70a on tablet 4bb42883b1a0475b863d008bd01fa70a with peer 2f306549407b4bb281ccf2994035e4f1
I20240502 01:54:11.409911 24199 tablet_bootstrap.cc:492] T 472d7827b3074236a81a3ede098f6c0a P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:54:11.435523 24199 tablet_bootstrap.cc:492] T 472d7827b3074236a81a3ede098f6c0a P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:11.435935 24199 tablet_bootstrap.cc:492] T 472d7827b3074236a81a3ede098f6c0a P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap complete.
I20240502 01:54:11.436201 24199 ts_tablet_manager.cc:1367] T 472d7827b3074236a81a3ede098f6c0a P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.026s user 0.014s sys 0.006s
I20240502 01:54:11.436818 24199 raft_consensus.cc:348] T 472d7827b3074236a81a3ede098f6c0a P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } }
I20240502 01:54:11.437091 24199 raft_consensus.cc:729] T 472d7827b3074236a81a3ede098f6c0a P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: LEARNER
I20240502 01:54:11.437322 24199 consensus_queue.cc:260] T 472d7827b3074236a81a3ede098f6c0a P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } }
I20240502 01:54:11.440165 24199 ts_tablet_manager.cc:1398] T 472d7827b3074236a81a3ede098f6c0a P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:54:11.440538 20625 consensus_queue.cc:237] T 68bc23294ac64c9486bffebbdb16b508 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } }
I20240502 01:54:11.441947 20387 tablet_copy_service.cc:342] P 6a36c952c51542c0bf7ea3ec2857a9f9: Request end of tablet copy session 3fceccefba274b1dbbd8216af8b0bc29-472d7827b3074236a81a3ede098f6c0a received from {username='slave'} at 127.11.42.66:36601
I20240502 01:54:11.442092 20387 tablet_copy_service.cc:433] P 6a36c952c51542c0bf7ea3ec2857a9f9: ending tablet copy session 3fceccefba274b1dbbd8216af8b0bc29-472d7827b3074236a81a3ede098f6c0a on tablet 472d7827b3074236a81a3ede098f6c0a with peer 3fceccefba274b1dbbd8216af8b0bc29
I20240502 01:54:11.446203 20233 raft_consensus.cc:1261] T 68bc23294ac64c9486bffebbdb16b508 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:54:11.446512 20562 consensus_peers.cc:485] T 68bc23294ac64c9486bffebbdb16b508 P f19cc6cb1e76460aa6711e49a5775091 -> Peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): Couldn't send request to peer 3fceccefba274b1dbbd8216af8b0bc29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 68bc23294ac64c9486bffebbdb16b508. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:11.446846 20494 raft_consensus.cc:1261] T 68bc23294ac64c9486bffebbdb16b508 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:11.447389 24125 consensus_queue.cc:1035] T 68bc23294ac64c9486bffebbdb16b508 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:11.448052 24010 consensus_queue.cc:1035] T 68bc23294ac64c9486bffebbdb16b508 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:11.460245 24125 raft_consensus.cc:2932] T 68bc23294ac64c9486bffebbdb16b508 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } } }
I20240502 01:54:11.466713 17334 catalog_manager.cc:5484] T 68bc23294ac64c9486bffebbdb16b508 P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: config changed from index -1 to 2, NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) added. New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:11.468786 20498 raft_consensus.cc:2932] T 68bc23294ac64c9486bffebbdb16b508 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } } }
I20240502 01:54:11.471019 20233 raft_consensus.cc:2932] T 68bc23294ac64c9486bffebbdb16b508 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } } }
I20240502 01:54:11.478765 24215 tablet_bootstrap.cc:492] T ed1522ec150a46f28fd553f9b563651f P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:11.479271 24215 tablet_bootstrap.cc:492] T ed1522ec150a46f28fd553f9b563651f P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap complete.
I20240502 01:54:11.479569 24215 ts_tablet_manager.cc:1367] T ed1522ec150a46f28fd553f9b563651f P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.085s user 0.050s sys 0.004s
I20240502 01:54:11.480505 24215 raft_consensus.cc:348] T ed1522ec150a46f28fd553f9b563651f P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } }
I20240502 01:54:11.480798 24215 raft_consensus.cc:729] T ed1522ec150a46f28fd553f9b563651f P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: LEARNER
I20240502 01:54:11.481040 24215 consensus_queue.cc:260] T ed1522ec150a46f28fd553f9b563651f P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } }
I20240502 01:54:11.483006 24215 ts_tablet_manager.cc:1398] T ed1522ec150a46f28fd553f9b563651f P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240502 01:54:11.486177 17887 tablet_copy_service.cc:342] P bb31bceba76d466db24b9341c04900c3: Request end of tablet copy session 6a36c952c51542c0bf7ea3ec2857a9f9-ed1522ec150a46f28fd553f9b563651f received from {username='slave'} at 127.11.42.70:41203
I20240502 01:54:11.486320 17887 tablet_copy_service.cc:433] P bb31bceba76d466db24b9341c04900c3: ending tablet copy session 6a36c952c51542c0bf7ea3ec2857a9f9-ed1522ec150a46f28fd553f9b563651f on tablet ed1522ec150a46f28fd553f9b563651f with peer 6a36c952c51542c0bf7ea3ec2857a9f9
I20240502 01:54:11.493252 20498 raft_consensus.cc:1203] T 4bb42883b1a0475b863d008bd01fa70a P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 01:54:11.499387 20625 consensus_queue.cc:237] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:54:11.501714 20233 raft_consensus.cc:1261] T 1b14ac3413004127b72904b631ec1b94 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:11.501755 17604 raft_consensus.cc:1261] T 1b14ac3413004127b72904b631ec1b94 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:11.502215 24127 consensus_queue.cc:1035] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:54:11.502513 20564 consensus_peers.cc:485] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1b14ac3413004127b72904b631ec1b94. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:11.502550 24125 consensus_queue.cc:1035] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:11.504981 24010 raft_consensus.cc:2932] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:11.505599 17604 raft_consensus.cc:2932] T 1b14ac3413004127b72904b631ec1b94 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:11.505564 20233 raft_consensus.cc:2932] T 1b14ac3413004127b72904b631ec1b94 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:11.511003 17326 catalog_manager.cc:5484] T 1b14ac3413004127b72904b631ec1b94 P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:11.526595 20625 consensus_queue.cc:237] T 75bb4e19cf57405fabcb532b55a29ced P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } }
I20240502 01:54:11.530047 17864 raft_consensus.cc:1261] T 75bb4e19cf57405fabcb532b55a29ced P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:54:11.530443 20564 consensus_peers.cc:485] T 75bb4e19cf57405fabcb532b55a29ced P f19cc6cb1e76460aa6711e49a5775091 -> Peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Couldn't send request to peer 4da690c0fa13407a85c0689ce8de9ade. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 75bb4e19cf57405fabcb532b55a29ced. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:11.530550 24125 consensus_queue.cc:1035] T 75bb4e19cf57405fabcb532b55a29ced P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:11.530690 20366 raft_consensus.cc:1261] T 75bb4e19cf57405fabcb532b55a29ced P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:11.531179 24125 consensus_queue.cc:1035] T 75bb4e19cf57405fabcb532b55a29ced P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:11.533118 24010 raft_consensus.cc:2932] T 75bb4e19cf57405fabcb532b55a29ced P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } } }
I20240502 01:54:11.533744 17864 raft_consensus.cc:2932] T 75bb4e19cf57405fabcb532b55a29ced P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } } }
I20240502 01:54:11.534186 20366 raft_consensus.cc:2932] T 75bb4e19cf57405fabcb532b55a29ced P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } } }
I20240502 01:54:11.539227 17326 catalog_manager.cc:5484] T 75bb4e19cf57405fabcb532b55a29ced P bb31bceba76d466db24b9341c04900c3 reported cstate change: config changed from index -1 to 2, NON_VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) added. New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } } }
I20240502 01:54:11.563098 17735 consensus_queue.cc:237] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } }
I20240502 01:54:11.564579 23637 consensus_queue.cc:1035] T 8c129f9289024ec9898fbbc06b2fb2cd P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:54:11.575287 17672 consensus_peers.cc:485] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 -> Peer 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): Couldn't send request to peer 6a36c952c51542c0bf7ea3ec2857a9f9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8e7f88b6c509477fb0900024e7de8f00. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:11.576342 20231 raft_consensus.cc:1261] T 8e7f88b6c509477fb0900024e7de8f00 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:11.576896 17864 raft_consensus.cc:1261] T 8e7f88b6c509477fb0900024e7de8f00 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:11.577533 24203 consensus_queue.cc:1035] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:11.578063 23961 consensus_queue.cc:1035] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:11.587108 24202 raft_consensus.cc:2932] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } } }
I20240502 01:54:11.588805 17864 raft_consensus.cc:2932] T 8e7f88b6c509477fb0900024e7de8f00 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } } }
I20240502 01:54:11.595602 17334 catalog_manager.cc:5484] T 8e7f88b6c509477fb0900024e7de8f00 P bb31bceba76d466db24b9341c04900c3 reported cstate change: config changed from index 6 to 7, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } } }
I20240502 01:54:11.598659 17328 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:54:11.600524 24197 consensus_queue.cc:1035] T 8c129f9289024ec9898fbbc06b2fb2cd P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:11.627017 17864 raft_consensus.cc:1261] T 59ce542c545f4ba590e41690e8df5fc6 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:11.627156 20233 raft_consensus.cc:2932] T 8e7f88b6c509477fb0900024e7de8f00 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } } }
I20240502 01:54:11.631072 20232 raft_consensus.cc:3037] T 59ce542c545f4ba590e41690e8df5fc6 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:11.633487 24212 consensus_queue.cc:1035] T 59ce542c545f4ba590e41690e8df5fc6 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:11.637063 20232 raft_consensus.cc:1261] T 59ce542c545f4ba590e41690e8df5fc6 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:11.637658 24212 consensus_queue.cc:1035] T 59ce542c545f4ba590e41690e8df5fc6 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:11.644724 17328 catalog_manager.cc:5576] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339) does not have the latest schema for tablet 59ce542c545f4ba590e41690e8df5fc6 (table rebalancer_dynamic_table [id=eb791abbdca04194aa499cb7e38e1dc6]). Expected version 1 got 0
I20240502 01:54:11.651268 24271 tablet.cc:1722] T 59ce542c545f4ba590e41690e8df5fc6 P 4da690c0fa13407a85c0689ce8de9ade: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:11.651716 17334 catalog_manager.cc:5576] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339) does not have the latest schema for tablet 59ce542c545f4ba590e41690e8df5fc6 (table rebalancer_dynamic_table [id=eb791abbdca04194aa499cb7e38e1dc6]). Expected version 1 got 0
I20240502 01:54:11.652824 24271 tablet.cc:1620] T 59ce542c545f4ba590e41690e8df5fc6 P 4da690c0fa13407a85c0689ce8de9ade: MemRowSet was empty: no flush needed.
I20240502 01:54:11.654183 17334 catalog_manager.cc:5576] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321) does not have the latest schema for tablet 59ce542c545f4ba590e41690e8df5fc6 (table rebalancer_dynamic_table [id=eb791abbdca04194aa499cb7e38e1dc6]). Expected version 1 got 0
I20240502 01:54:11.654487 24272 tablet.cc:1722] T 59ce542c545f4ba590e41690e8df5fc6 P fdc2ac372492454c85387589a91f63af: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:11.658921 24272 tablet.cc:1620] T 59ce542c545f4ba590e41690e8df5fc6 P fdc2ac372492454c85387589a91f63af: MemRowSet was empty: no flush needed.
I20240502 01:54:11.661849 17334 catalog_manager.cc:5576] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083) does not have the latest schema for tablet 59ce542c545f4ba590e41690e8df5fc6 (table rebalancer_dynamic_table [id=eb791abbdca04194aa499cb7e38e1dc6]). Expected version 1 got 0
I20240502 01:54:11.668066 24273 tablet.cc:1722] T 59ce542c545f4ba590e41690e8df5fc6 P bb31bceba76d466db24b9341c04900c3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:11.668494 24273 tablet.cc:1620] T 59ce542c545f4ba590e41690e8df5fc6 P bb31bceba76d466db24b9341c04900c3: MemRowSet was empty: no flush needed.
I20240502 01:54:11.669456 17328 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=eb791abbdca04194aa499cb7e38e1dc6] alter complete (version 1)
I20240502 01:54:11.670151 17334 catalog_manager.cc:5576] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083) does not have the latest schema for tablet 59ce542c545f4ba590e41690e8df5fc6 (table rebalancer_dynamic_table [id=eb791abbdca04194aa499cb7e38e1dc6]). Expected version 1 got 0
I20240502 01:54:11.686882 17606 raft_consensus.cc:1203] T 472d7827b3074236a81a3ede098f6c0a P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:11.698091 24274 ts_tablet_manager.cc:902] T fbd94103cae04473ba81709da39d7ed8 P dfa1152987644bd781b46c7e42a7a685: Initiating tablet copy from peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:54:11.699641 24274 tablet_copy_client.cc:308] T fbd94103cae04473ba81709da39d7ed8 P dfa1152987644bd781b46c7e42a7a685: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.72:42411
I20240502 01:54:11.700398 20650 tablet_copy_service.cc:140] P f19cc6cb1e76460aa6711e49a5775091: Received BeginTabletCopySession request for tablet fbd94103cae04473ba81709da39d7ed8 from peer dfa1152987644bd781b46c7e42a7a685 ({username='slave'} at 127.11.42.67:36277)
I20240502 01:54:11.700559 20650 tablet_copy_service.cc:161] P f19cc6cb1e76460aa6711e49a5775091: Beginning new tablet copy session on tablet fbd94103cae04473ba81709da39d7ed8 from peer dfa1152987644bd781b46c7e42a7a685 at {username='slave'} at 127.11.42.67:36277: session id = dfa1152987644bd781b46c7e42a7a685-fbd94103cae04473ba81709da39d7ed8
I20240502 01:54:11.700670 20366 consensus_queue.cc:237] T 2714919062f5465c8cde1a3c4720978a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } }
I20240502 01:54:11.703213 20650 tablet_copy_source_session.cc:205] T fbd94103cae04473ba81709da39d7ed8 P f19cc6cb1e76460aa6711e49a5775091: Tablet Copy: opened 0 blocks and 1 log segments
W20240502 01:54:11.712586 20300 consensus_peers.cc:485] T 2714919062f5465c8cde1a3c4720978a P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): Couldn't send request to peer 3fceccefba274b1dbbd8216af8b0bc29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2714919062f5465c8cde1a3c4720978a. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:11.713120 20498 raft_consensus.cc:1261] T 2714919062f5465c8cde1a3c4720978a P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:11.714393 24172 consensus_queue.cc:1035] T 164e04f7c9454c189f6907b319023d99 P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:11.715131 20232 raft_consensus.cc:1261] T 2714919062f5465c8cde1a3c4720978a P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:11.715459 23908 consensus_queue.cc:1035] T 2714919062f5465c8cde1a3c4720978a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:11.715818 24274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbd94103cae04473ba81709da39d7ed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.717958 23908 consensus_queue.cc:1035] T 2714919062f5465c8cde1a3c4720978a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:11.734355 23908 raft_consensus.cc:2932] T 2714919062f5465c8cde1a3c4720978a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } } }
I20240502 01:54:11.736652 24274 tablet_copy_client.cc:783] T fbd94103cae04473ba81709da39d7ed8 P dfa1152987644bd781b46c7e42a7a685: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:11.736883 24274 tablet_copy_client.cc:647] T fbd94103cae04473ba81709da39d7ed8 P dfa1152987644bd781b46c7e42a7a685: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:11.737807 20498 raft_consensus.cc:2932] T 2714919062f5465c8cde1a3c4720978a P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } } }
I20240502 01:54:11.739485 20232 raft_consensus.cc:2932] T 2714919062f5465c8cde1a3c4720978a P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } } }
I20240502 01:54:11.741940 17328 catalog_manager.cc:5484] T 2714919062f5465c8cde1a3c4720978a P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: config changed from index -1 to 2, NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) added. New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:11.747993 24172 consensus_queue.cc:1035] T f6cac9478db94a21a9799b80d24c8dc9 P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:11.750129 24274 tablet_copy_client.cc:523] T fbd94103cae04473ba81709da39d7ed8 P dfa1152987644bd781b46c7e42a7a685: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:11.757407 24274 tablet_bootstrap.cc:492] T fbd94103cae04473ba81709da39d7ed8 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:54:11.761310 24172 consensus_queue.cc:1035] T 164e04f7c9454c189f6907b319023d99 P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:11.786763 17326 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:54:11.797057 24274 tablet_bootstrap.cc:492] T fbd94103cae04473ba81709da39d7ed8 P dfa1152987644bd781b46c7e42a7a685: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:11.797405 24274 tablet_bootstrap.cc:492] T fbd94103cae04473ba81709da39d7ed8 P dfa1152987644bd781b46c7e42a7a685: Bootstrap complete.
I20240502 01:54:11.797696 24274 ts_tablet_manager.cc:1367] T fbd94103cae04473ba81709da39d7ed8 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.040s user 0.006s sys 0.000s
I20240502 01:54:11.800949 24274 raft_consensus.cc:348] T fbd94103cae04473ba81709da39d7ed8 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:54:11.801208 24274 raft_consensus.cc:729] T fbd94103cae04473ba81709da39d7ed8 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: LEARNER
I20240502 01:54:11.801378 24274 consensus_queue.cc:260] T fbd94103cae04473ba81709da39d7ed8 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:54:11.801909 23637 consensus_queue.cc:1035] T f6cac9478db94a21a9799b80d24c8dc9 P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:11.803282 24274 ts_tablet_manager.cc:1398] T fbd94103cae04473ba81709da39d7ed8 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:54:11.804343 20650 tablet_copy_service.cc:342] P f19cc6cb1e76460aa6711e49a5775091: Request end of tablet copy session dfa1152987644bd781b46c7e42a7a685-fbd94103cae04473ba81709da39d7ed8 received from {username='slave'} at 127.11.42.67:36277
I20240502 01:54:11.804481 20650 tablet_copy_service.cc:433] P f19cc6cb1e76460aa6711e49a5775091: ending tablet copy session dfa1152987644bd781b46c7e42a7a685-fbd94103cae04473ba81709da39d7ed8 on tablet fbd94103cae04473ba81709da39d7ed8 with peer dfa1152987644bd781b46c7e42a7a685
I20240502 01:54:11.805644 24271 tablet.cc:1722] T 59ce542c545f4ba590e41690e8df5fc6 P 4da690c0fa13407a85c0689ce8de9ade: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:11.806150 24271 tablet.cc:1620] T 59ce542c545f4ba590e41690e8df5fc6 P 4da690c0fa13407a85c0689ce8de9ade: MemRowSet was empty: no flush needed.
I20240502 01:54:11.807227 24273 tablet.cc:1722] T 59ce542c545f4ba590e41690e8df5fc6 P bb31bceba76d466db24b9341c04900c3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:11.807730 24273 tablet.cc:1620] T 59ce542c545f4ba590e41690e8df5fc6 P bb31bceba76d466db24b9341c04900c3: MemRowSet was empty: no flush needed.
I20240502 01:54:11.819834 17330 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=eb791abbdca04194aa499cb7e38e1dc6] alter complete (version 2)
I20240502 01:54:11.824756 24127 raft_consensus.cc:1050] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091: attempting to promote NON_VOTER bb31bceba76d466db24b9341c04900c3 to VOTER
I20240502 01:54:11.824908 24272 tablet.cc:1722] T 59ce542c545f4ba590e41690e8df5fc6 P fdc2ac372492454c85387589a91f63af: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:11.825838 24272 tablet.cc:1620] T 59ce542c545f4ba590e41690e8df5fc6 P fdc2ac372492454c85387589a91f63af: MemRowSet was empty: no flush needed.
I20240502 01:54:11.825655 24127 consensus_queue.cc:237] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } }
I20240502 01:54:11.828377 17867 raft_consensus.cc:1261] T cf0237a4c65b495ba892871dd850bed9 P bb31bceba76d466db24b9341c04900c3 [term 1 LEARNER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:11.829097 24125 consensus_queue.cc:1035] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:11.830991 17737 raft_consensus.cc:1261] T cf0237a4c65b495ba892871dd850bed9 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:11.831123 20498 raft_consensus.cc:1261] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:11.831584 24010 consensus_queue.cc:1035] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:11.833176 24010 consensus_queue.cc:1035] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:11.846077 24125 raft_consensus.cc:2932] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:11.846951 20498 raft_consensus.cc:2932] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:11.850417 17867 raft_consensus.cc:2932] T cf0237a4c65b495ba892871dd850bed9 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:11.853446 17328 catalog_manager.cc:5484] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: config changed from index 5 to 6, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:11.859061 17735 raft_consensus.cc:2932] T cf0237a4c65b495ba892871dd850bed9 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:11.876251 20366 raft_consensus.cc:1203] T ed1522ec150a46f28fd553f9b563651f P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 01:54:11.927742 17328 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:54:11.931501 17334 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_extra_table_46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:11.938791 24271 tablet.cc:1722] T 59ce542c545f4ba590e41690e8df5fc6 P 4da690c0fa13407a85c0689ce8de9ade: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:11.939117 24272 tablet.cc:1722] T 59ce542c545f4ba590e41690e8df5fc6 P fdc2ac372492454c85387589a91f63af: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:11.943068 24274 ts_tablet_manager.cc:902] T 1b14ac3413004127b72904b631ec1b94 P dfa1152987644bd781b46c7e42a7a685: Initiating tablet copy from peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:54:11.947808 24273 tablet.cc:1722] T 59ce542c545f4ba590e41690e8df5fc6 P bb31bceba76d466db24b9341c04900c3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:11.951988 17333 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table_new_name [id=eb791abbdca04194aa499cb7e38e1dc6] alter complete (version 3)
I20240502 01:54:11.972105 17585 tablet_service.cc:1467] Processing CreateTablet for tablet fa4d38dc9b674a9b90d4793dad2687ec (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=26c9a097a1a4464a9d467639ce3fe653]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:11.972364 20345 tablet_service.cc:1467] Processing CreateTablet for tablet fa4d38dc9b674a9b90d4793dad2687ec (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=26c9a097a1a4464a9d467639ce3fe653]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:11.972577 17585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa4d38dc9b674a9b90d4793dad2687ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.972780 20345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa4d38dc9b674a9b90d4793dad2687ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.976855 20610 tablet_service.cc:1467] Processing CreateTablet for tablet 59be89ca2c0b4275844f1e73cd20595b (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=26c9a097a1a4464a9d467639ce3fe653]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:11.977325 20610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59be89ca2c0b4275844f1e73cd20595b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.977334 20347 tablet_service.cc:1467] Processing CreateTablet for tablet 59be89ca2c0b4275844f1e73cd20595b (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=26c9a097a1a4464a9d467639ce3fe653]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:11.977720 20347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59be89ca2c0b4275844f1e73cd20595b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.982084 20609 tablet_service.cc:1467] Processing CreateTablet for tablet c491a39ca87847d9ace688346c0baf57 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=26c9a097a1a4464a9d467639ce3fe653]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:11.982478 20609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c491a39ca87847d9ace688346c0baf57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.976940 20478 tablet_service.cc:1467] Processing CreateTablet for tablet 59be89ca2c0b4275844f1e73cd20595b (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=26c9a097a1a4464a9d467639ce3fe653]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:11.983104 20478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59be89ca2c0b4275844f1e73cd20595b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.985594 20346 tablet_service.cc:1467] Processing CreateTablet for tablet c491a39ca87847d9ace688346c0baf57 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=26c9a097a1a4464a9d467639ce3fe653]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:11.986030 20346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c491a39ca87847d9ace688346c0baf57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.989761 17456 tablet_service.cc:1467] Processing CreateTablet for tablet fa4d38dc9b674a9b90d4793dad2687ec (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=26c9a097a1a4464a9d467639ce3fe653]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:11.990207 17456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa4d38dc9b674a9b90d4793dad2687ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.989761 17454 tablet_service.cc:1467] Processing CreateTablet for tablet c491a39ca87847d9ace688346c0baf57 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=26c9a097a1a4464a9d467639ce3fe653]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:11.993779 17454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c491a39ca87847d9ace688346c0baf57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.995147 24323 tablet_bootstrap.cc:492] T 59be89ca2c0b4275844f1e73cd20595b P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:54:11.996680 24323 tablet_bootstrap.cc:654] T 59be89ca2c0b4275844f1e73cd20595b P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:12.003736 24323 tablet_bootstrap.cc:492] T 59be89ca2c0b4275844f1e73cd20595b P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:54:12.004181 24323 ts_tablet_manager.cc:1367] T 59be89ca2c0b4275844f1e73cd20595b P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240502 01:54:12.005852 24323 raft_consensus.cc:348] T 59be89ca2c0b4275844f1e73cd20595b P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:12.006691 24323 raft_consensus.cc:374] T 59be89ca2c0b4275844f1e73cd20595b P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:12.006959 24323 raft_consensus.cc:729] T 59be89ca2c0b4275844f1e73cd20595b P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:54:12.007400 24323 consensus_queue.cc:260] T 59be89ca2c0b4275844f1e73cd20595b P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:12.008468 24323 ts_tablet_manager.cc:1398] T 59be89ca2c0b4275844f1e73cd20595b P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:54:12.014565 24274 tablet_copy_client.cc:308] T 1b14ac3413004127b72904b631ec1b94 P dfa1152987644bd781b46c7e42a7a685: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.72:42411
I20240502 01:54:12.015237 20650 tablet_copy_service.cc:140] P f19cc6cb1e76460aa6711e49a5775091: Received BeginTabletCopySession request for tablet 1b14ac3413004127b72904b631ec1b94 from peer dfa1152987644bd781b46c7e42a7a685 ({username='slave'} at 127.11.42.67:36277)
I20240502 01:54:12.015404 20650 tablet_copy_service.cc:161] P f19cc6cb1e76460aa6711e49a5775091: Beginning new tablet copy session on tablet 1b14ac3413004127b72904b631ec1b94 from peer dfa1152987644bd781b46c7e42a7a685 at {username='slave'} at 127.11.42.67:36277: session id = dfa1152987644bd781b46c7e42a7a685-1b14ac3413004127b72904b631ec1b94
I20240502 01:54:12.018399 20650 tablet_copy_source_session.cc:205] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:12.019968 24274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b14ac3413004127b72904b631ec1b94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:12.035917 24329 ts_tablet_manager.cc:902] T 8e7f88b6c509477fb0900024e7de8f00 P 6a36c952c51542c0bf7ea3ec2857a9f9: Initiating tablet copy from peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:54:12.037246 24274 tablet_copy_client.cc:783] T 1b14ac3413004127b72904b631ec1b94 P dfa1152987644bd781b46c7e42a7a685: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:12.037459 24274 tablet_copy_client.cc:647] T 1b14ac3413004127b72904b631ec1b94 P dfa1152987644bd781b46c7e42a7a685: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:12.043964 24274 tablet_copy_client.cc:523] T 1b14ac3413004127b72904b631ec1b94 P dfa1152987644bd781b46c7e42a7a685: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:12.046880 24332 tablet_bootstrap.cc:492] T fa4d38dc9b674a9b90d4793dad2687ec P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:54:12.049890 24323 tablet_bootstrap.cc:492] T c491a39ca87847d9ace688346c0baf57 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:54:12.051309 24332 tablet_bootstrap.cc:654] T fa4d38dc9b674a9b90d4793dad2687ec P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:12.051542 24323 tablet_bootstrap.cc:654] T c491a39ca87847d9ace688346c0baf57 P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:12.053488 24166 raft_consensus.cc:1050] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3: attempting to promote NON_VOTER 2f306549407b4bb281ccf2994035e4f1 to VOTER
I20240502 01:54:12.053886 24274 tablet_bootstrap.cc:492] T 1b14ac3413004127b72904b631ec1b94 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:54:12.054373 24166 consensus_queue.cc:237] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } }
I20240502 01:54:12.056986 17334 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:12.057336 17334 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:12.070876 24331 tablet_bootstrap.cc:492] T c491a39ca87847d9ace688346c0baf57 P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:54:12.074441 24321 ts_tablet_manager.cc:902] T 68bc23294ac64c9486bffebbdb16b508 P 3fceccefba274b1dbbd8216af8b0bc29: Initiating tablet copy from peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:54:12.075690 24324 tablet_bootstrap.cc:492] T 59be89ca2c0b4275844f1e73cd20595b P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:54:12.077055 20498 raft_consensus.cc:1261] T 4bb42883b1a0475b863d008bd01fa70a P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:12.077669 24173 consensus_queue.cc:1035] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:12.077939 24323 tablet_bootstrap.cc:492] T c491a39ca87847d9ace688346c0baf57 P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:54:12.078104 24323 ts_tablet_manager.cc:1367] T c491a39ca87847d9ace688346c0baf57 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.003s
I20240502 01:54:12.078902 24323 raft_consensus.cc:348] T c491a39ca87847d9ace688346c0baf57 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:12.079216 24323 raft_consensus.cc:374] T c491a39ca87847d9ace688346c0baf57 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:12.079349 24323 raft_consensus.cc:729] T c491a39ca87847d9ace688346c0baf57 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:54:12.079674 24323 consensus_queue.cc:260] T c491a39ca87847d9ace688346c0baf57 P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:12.081315 17334 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet 59ce542c545f4ba590e41690e8df5fc6
I20240502 01:54:12.082396 20211 tablet_service.cc:1514] Processing DeleteTablet for tablet 59ce542c545f4ba590e41690e8df5fc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:12 UTC) from {username='slave'} at 127.0.0.1:44624
I20240502 01:54:12.083279 17847 tablet_service.cc:1514] Processing DeleteTablet for tablet 59ce542c545f4ba590e41690e8df5fc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:12 UTC) from {username='slave'} at 127.0.0.1:47226
I20240502 01:54:12.090437 17456 tablet_service.cc:1514] Processing DeleteTablet for tablet 59ce542c545f4ba590e41690e8df5fc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:12 UTC) from {username='slave'} at 127.0.0.1:54520
I20240502 01:54:12.093096 24344 tablet_replica.cc:330] T 59ce542c545f4ba590e41690e8df5fc6 P 4da690c0fa13407a85c0689ce8de9ade: stopping tablet replica
I20240502 01:54:12.093263 24324 tablet_bootstrap.cc:654] T 59be89ca2c0b4275844f1e73cd20595b P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:12.094512 24323 ts_tablet_manager.cc:1398] T c491a39ca87847d9ace688346c0baf57 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240502 01:54:12.094513 24344 raft_consensus.cc:2229] T 59ce542c545f4ba590e41690e8df5fc6 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:12.094923 24344 raft_consensus.cc:2258] T 59ce542c545f4ba590e41690e8df5fc6 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:12.095098 24339 ts_tablet_manager.cc:902] T 75bb4e19cf57405fabcb532b55a29ced P 4da690c0fa13407a85c0689ce8de9ade: Initiating tablet copy from peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:54:12.098223 24340 tablet_replica.cc:330] T 59ce542c545f4ba590e41690e8df5fc6 P fdc2ac372492454c85387589a91f63af: stopping tablet replica
I20240502 01:54:12.098678 24340 raft_consensus.cc:2229] T 59ce542c545f4ba590e41690e8df5fc6 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:12.100452 24340 raft_consensus.cc:2258] T 59ce542c545f4ba590e41690e8df5fc6 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:12.102222 24323 tablet_bootstrap.cc:492] T fa4d38dc9b674a9b90d4793dad2687ec P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:54:12.107980 24323 tablet_bootstrap.cc:654] T fa4d38dc9b674a9b90d4793dad2687ec P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:12.108062 24340 ts_tablet_manager.cc:1875] T 59ce542c545f4ba590e41690e8df5fc6 P fdc2ac372492454c85387589a91f63af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:12.112844 17606 raft_consensus.cc:1261] T 4bb42883b1a0475b863d008bd01fa70a P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:12.113428 24173 consensus_queue.cc:1035] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:12.117413 24274 tablet_bootstrap.cc:492] T 1b14ac3413004127b72904b631ec1b94 P dfa1152987644bd781b46c7e42a7a685: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:12.117799 24274 tablet_bootstrap.cc:492] T 1b14ac3413004127b72904b631ec1b94 P dfa1152987644bd781b46c7e42a7a685: Bootstrap complete.
I20240502 01:54:12.118034 24274 ts_tablet_manager.cc:1367] T 1b14ac3413004127b72904b631ec1b94 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.064s user 0.007s sys 0.000s
I20240502 01:54:12.118686 24274 raft_consensus.cc:348] T 1b14ac3413004127b72904b631ec1b94 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:54:12.118928 24274 raft_consensus.cc:729] T 1b14ac3413004127b72904b631ec1b94 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: LEARNER
I20240502 01:54:12.119102 24274 consensus_queue.cc:260] T 1b14ac3413004127b72904b631ec1b94 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:54:12.098956 24344 ts_tablet_manager.cc:1875] T 59ce542c545f4ba590e41690e8df5fc6 P 4da690c0fa13407a85c0689ce8de9ade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:12.125757 24329 tablet_copy_client.cc:308] T 8e7f88b6c509477fb0900024e7de8f00 P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.67:36115
I20240502 01:54:12.126471 17757 tablet_copy_service.cc:140] P dfa1152987644bd781b46c7e42a7a685: Received BeginTabletCopySession request for tablet 8e7f88b6c509477fb0900024e7de8f00 from peer 6a36c952c51542c0bf7ea3ec2857a9f9 ({username='slave'} at 127.11.42.70:55001)
I20240502 01:54:12.126521 24324 tablet_bootstrap.cc:492] T 59be89ca2c0b4275844f1e73cd20595b P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:54:12.126600 17757 tablet_copy_service.cc:161] P dfa1152987644bd781b46c7e42a7a685: Beginning new tablet copy session on tablet 8e7f88b6c509477fb0900024e7de8f00 from peer 6a36c952c51542c0bf7ea3ec2857a9f9 at {username='slave'} at 127.11.42.70:55001: session id = 6a36c952c51542c0bf7ea3ec2857a9f9-8e7f88b6c509477fb0900024e7de8f00
I20240502 01:54:12.126657 24324 ts_tablet_manager.cc:1367] T 59be89ca2c0b4275844f1e73cd20595b P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.051s user 0.005s sys 0.000s
I20240502 01:54:12.127617 24324 raft_consensus.cc:348] T 59be89ca2c0b4275844f1e73cd20595b P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:12.127802 24324 raft_consensus.cc:374] T 59be89ca2c0b4275844f1e73cd20595b P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:12.127895 24324 raft_consensus.cc:729] T 59be89ca2c0b4275844f1e73cd20595b P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:54:12.128114 24324 consensus_queue.cc:260] T 59be89ca2c0b4275844f1e73cd20595b P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:12.128852 24324 ts_tablet_manager.cc:1398] T 59be89ca2c0b4275844f1e73cd20595b P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:12.129632 17757 tablet_copy_source_session.cc:205] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:12.130342 24341 tablet_replica.cc:330] T 59ce542c545f4ba590e41690e8df5fc6 P bb31bceba76d466db24b9341c04900c3: stopping tablet replica
I20240502 01:54:12.130581 24341 raft_consensus.cc:2229] T 59ce542c545f4ba590e41690e8df5fc6 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:12.130828 24341 raft_consensus.cc:2258] T 59ce542c545f4ba590e41690e8df5fc6 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:12.131165 24329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e7f88b6c509477fb0900024e7de8f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:12.133263 24322 tablet_bootstrap.cc:492] T fa4d38dc9b674a9b90d4793dad2687ec P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:54:12.135172 24322 tablet_bootstrap.cc:654] T fa4d38dc9b674a9b90d4793dad2687ec P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:12.138092 23908 raft_consensus.cc:482] T 59be89ca2c0b4275844f1e73cd20595b P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:12.138235 23908 raft_consensus.cc:504] T 59be89ca2c0b4275844f1e73cd20595b P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:12.138892 23908 leader_election.cc:288] T 59be89ca2c0b4275844f1e73cd20595b P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:54:12.139851 20494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59be89ca2c0b4275844f1e73cd20595b" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
I20240502 01:54:12.140048 24274 ts_tablet_manager.cc:1398] T 1b14ac3413004127b72904b631ec1b94 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20240502 01:54:12.140045 20494 raft_consensus.cc:2454] T 59be89ca2c0b4275844f1e73cd20595b P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 0.
I20240502 01:54:12.140822 20303 leader_election.cc:302] T 59be89ca2c0b4275844f1e73cd20595b P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 6a36c952c51542c0bf7ea3ec2857a9f9; no voters:
I20240502 01:54:12.141397 23611 raft_consensus.cc:2781] T 59be89ca2c0b4275844f1e73cd20595b P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:12.141513 23611 raft_consensus.cc:482] T 59be89ca2c0b4275844f1e73cd20595b P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:12.141608 23611 raft_consensus.cc:3037] T 59be89ca2c0b4275844f1e73cd20595b P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:12.144162 23611 raft_consensus.cc:504] T 59be89ca2c0b4275844f1e73cd20595b P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:12.144771 23611 leader_election.cc:288] T 59be89ca2c0b4275844f1e73cd20595b P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Requested vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:54:12.146116 20496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59be89ca2c0b4275844f1e73cd20595b" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
I20240502 01:54:12.146306 20496 raft_consensus.cc:3037] T 59be89ca2c0b4275844f1e73cd20595b P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:12.148825 20496 raft_consensus.cc:2454] T 59be89ca2c0b4275844f1e73cd20595b P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 1.
I20240502 01:54:12.149639 24329 tablet_copy_client.cc:783] T 8e7f88b6c509477fb0900024e7de8f00 P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:12.150946 20303 leader_election.cc:302] T 59be89ca2c0b4275844f1e73cd20595b P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 6a36c952c51542c0bf7ea3ec2857a9f9; no voters:
I20240502 01:54:12.151422 23908 raft_consensus.cc:2781] T 59be89ca2c0b4275844f1e73cd20595b P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:12.151582 23908 raft_consensus.cc:686] T 59be89ca2c0b4275844f1e73cd20595b P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Becoming Leader. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Running, Role: LEADER
I20240502 01:54:12.151871 23908 consensus_queue.cc:237] T 59be89ca2c0b4275844f1e73cd20595b P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:12.149926 24329 tablet_copy_client.cc:647] T 8e7f88b6c509477fb0900024e7de8f00 P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:12.162802 24329 tablet_copy_client.cc:523] T 8e7f88b6c509477fb0900024e7de8f00 P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:12.164544 24344 ts_tablet_manager.cc:1888] T 59ce542c545f4ba590e41690e8df5fc6 P 4da690c0fa13407a85c0689ce8de9ade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:12.164732 24344 log.cc:1196] T 59ce542c545f4ba590e41690e8df5fc6 P 4da690c0fa13407a85c0689ce8de9ade: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/59ce542c545f4ba590e41690e8df5fc6
I20240502 01:54:12.164777 24321 tablet_copy_client.cc:308] T 68bc23294ac64c9486bffebbdb16b508 P 3fceccefba274b1dbbd8216af8b0bc29: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.72:42411
I20240502 01:54:12.167541 24329 tablet_bootstrap.cc:492] T 8e7f88b6c509477fb0900024e7de8f00 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:54:12.168040 24344 ts_tablet_manager.cc:1909] T 59ce542c545f4ba590e41690e8df5fc6 P 4da690c0fa13407a85c0689ce8de9ade: Deleting consensus metadata
I20240502 01:54:12.170195 17321 catalog_manager.cc:4830] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): tablet 59ce542c545f4ba590e41690e8df5fc6 (table rebalancer_dynamic_table_new_name [id=eb791abbdca04194aa499cb7e38e1dc6]) successfully deleted
I20240502 01:54:12.170706 24332 tablet_bootstrap.cc:492] T fa4d38dc9b674a9b90d4793dad2687ec P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:54:12.170986 24322 tablet_bootstrap.cc:492] T fa4d38dc9b674a9b90d4793dad2687ec P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:54:12.171200 24322 ts_tablet_manager.cc:1367] T fa4d38dc9b674a9b90d4793dad2687ec P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.000s
I20240502 01:54:12.171517 24332 ts_tablet_manager.cc:1367] T fa4d38dc9b674a9b90d4793dad2687ec P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.125s user 0.004s sys 0.000s
I20240502 01:54:12.172366 20650 tablet_copy_service.cc:342] P f19cc6cb1e76460aa6711e49a5775091: Request end of tablet copy session dfa1152987644bd781b46c7e42a7a685-1b14ac3413004127b72904b631ec1b94 received from {username='slave'} at 127.11.42.67:36277
I20240502 01:54:12.172744 20650 tablet_copy_service.cc:433] P f19cc6cb1e76460aa6711e49a5775091: ending tablet copy session dfa1152987644bd781b46c7e42a7a685-1b14ac3413004127b72904b631ec1b94 on tablet 1b14ac3413004127b72904b631ec1b94 with peer dfa1152987644bd781b46c7e42a7a685
I20240502 01:54:12.173246 20649 tablet_copy_service.cc:140] P f19cc6cb1e76460aa6711e49a5775091: Received BeginTabletCopySession request for tablet 68bc23294ac64c9486bffebbdb16b508 from peer 3fceccefba274b1dbbd8216af8b0bc29 ({username='slave'} at 127.11.42.66:54577)
I20240502 01:54:12.173449 20649 tablet_copy_service.cc:161] P f19cc6cb1e76460aa6711e49a5775091: Beginning new tablet copy session on tablet 68bc23294ac64c9486bffebbdb16b508 from peer 3fceccefba274b1dbbd8216af8b0bc29 at {username='slave'} at 127.11.42.66:54577: session id = 3fceccefba274b1dbbd8216af8b0bc29-68bc23294ac64c9486bffebbdb16b508
I20240502 01:54:12.179895 17326 catalog_manager.cc:5484] T 59be89ca2c0b4275844f1e73cd20595b P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70). New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:12.184758 17334 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:54:12.185909 20649 tablet_copy_source_session.cc:205] T 68bc23294ac64c9486bffebbdb16b508 P f19cc6cb1e76460aa6711e49a5775091: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:12.189067 24173 raft_consensus.cc:2932] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:12.186993 20625 raft_consensus.cc:1261] T 4bb42883b1a0475b863d008bd01fa70a P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:12.190959 24167 consensus_queue.cc:1035] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:12.194499 20494 raft_consensus.cc:2932] T 4bb42883b1a0475b863d008bd01fa70a P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:12.196249 17604 raft_consensus.cc:2932] T 4bb42883b1a0475b863d008bd01fa70a P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:12.199474 17328 catalog_manager.cc:5484] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 reported cstate change: config changed from index 7 to 8, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:12.206683 20628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59be89ca2c0b4275844f1e73cd20595b" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091" is_pre_election: true
I20240502 01:54:12.207136 20627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59be89ca2c0b4275844f1e73cd20595b" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091"
W20240502 01:54:12.207505 20302 leader_election.cc:341] T 59be89ca2c0b4275844f1e73cd20595b P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Illegal state: must be running to vote when last-logged opid is not known
W20240502 01:54:12.208766 20302 leader_election.cc:341] T 59be89ca2c0b4275844f1e73cd20595b P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:12.211809 17456 tablet_service.cc:1467] Processing CreateTablet for tablet 57e3c62666b1428ba01ef75abd15e3ef (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c11e10ff0c07463f897b3b61d9e5d870]), partition=
I20240502 01:54:12.212203 17456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57e3c62666b1428ba01ef75abd15e3ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:12.216166 17737 raft_consensus.cc:1203] T fbd94103cae04473ba81709da39d7ed8 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:12.218703 20609 tablet_service.cc:1467] Processing CreateTablet for tablet 57e3c62666b1428ba01ef75abd15e3ef (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c11e10ff0c07463f897b3b61d9e5d870]), partition=
I20240502 01:54:12.219084 20609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57e3c62666b1428ba01ef75abd15e3ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:12.229991 24323 tablet_bootstrap.cc:492] T fa4d38dc9b674a9b90d4793dad2687ec P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:54:12.230806 24323 ts_tablet_manager.cc:1367] T fa4d38dc9b674a9b90d4793dad2687ec P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.128s user 0.026s sys 0.030s
I20240502 01:54:12.232683 24323 raft_consensus.cc:348] T fa4d38dc9b674a9b90d4793dad2687ec P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:12.233834 24323 raft_consensus.cc:374] T fa4d38dc9b674a9b90d4793dad2687ec P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:12.233916 24323 raft_consensus.cc:729] T fa4d38dc9b674a9b90d4793dad2687ec P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:54:12.234779 24323 consensus_queue.cc:260] T fa4d38dc9b674a9b90d4793dad2687ec P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:12.236979 24323 ts_tablet_manager.cc:1398] T fa4d38dc9b674a9b90d4793dad2687ec P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 01:54:12.239120 24339 tablet_copy_client.cc:308] T 75bb4e19cf57405fabcb532b55a29ced P 4da690c0fa13407a85c0689ce8de9ade: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.72:42411
I20240502 01:54:12.239867 20649 tablet_copy_service.cc:140] P f19cc6cb1e76460aa6711e49a5775091: Received BeginTabletCopySession request for tablet 75bb4e19cf57405fabcb532b55a29ced from peer 4da690c0fa13407a85c0689ce8de9ade ({username='slave'} at 127.11.42.65:34833)
I20240502 01:54:12.240044 20649 tablet_copy_service.cc:161] P f19cc6cb1e76460aa6711e49a5775091: Beginning new tablet copy session on tablet 75bb4e19cf57405fabcb532b55a29ced from peer 4da690c0fa13407a85c0689ce8de9ade at {username='slave'} at 127.11.42.65:34833: session id = 4da690c0fa13407a85c0689ce8de9ade-75bb4e19cf57405fabcb532b55a29ced
I20240502 01:54:12.242856 20649 tablet_copy_source_session.cc:205] T 75bb4e19cf57405fabcb532b55a29ced P f19cc6cb1e76460aa6711e49a5775091: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:12.247509 23611 raft_consensus.cc:1050] T 472d7827b3074236a81a3ede098f6c0a P 6a36c952c51542c0bf7ea3ec2857a9f9: attempting to promote NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 to VOTER
I20240502 01:54:12.248376 23611 consensus_queue.cc:237] T 472d7827b3074236a81a3ede098f6c0a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } }
I20240502 01:54:12.254949 24339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75bb4e19cf57405fabcb532b55a29ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:12.171989 24322 raft_consensus.cc:348] T fa4d38dc9b674a9b90d4793dad2687ec P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:12.257809 24322 raft_consensus.cc:374] T fa4d38dc9b674a9b90d4793dad2687ec P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:12.257980 24322 raft_consensus.cc:729] T fa4d38dc9b674a9b90d4793dad2687ec P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:54:12.258406 24322 consensus_queue.cc:260] T fa4d38dc9b674a9b90d4793dad2687ec P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:12.260607 24322 ts_tablet_manager.cc:1398] T fa4d38dc9b674a9b90d4793dad2687ec P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.089s user 0.002s sys 0.000s
I20240502 01:54:12.262058 17847 tablet_service.cc:1467] Processing CreateTablet for tablet 57e3c62666b1428ba01ef75abd15e3ef (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c11e10ff0c07463f897b3b61d9e5d870]), partition=
I20240502 01:54:12.262856 17847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57e3c62666b1428ba01ef75abd15e3ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:12.269062 24340 ts_tablet_manager.cc:1888] T 59ce542c545f4ba590e41690e8df5fc6 P fdc2ac372492454c85387589a91f63af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:12.269326 24340 log.cc:1196] T 59ce542c545f4ba590e41690e8df5fc6 P fdc2ac372492454c85387589a91f63af: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal/wals/59ce542c545f4ba590e41690e8df5fc6
I20240502 01:54:12.269878 24340 ts_tablet_manager.cc:1909] T 59ce542c545f4ba590e41690e8df5fc6 P fdc2ac372492454c85387589a91f63af: Deleting consensus metadata
I20240502 01:54:12.277616 17476 raft_consensus.cc:1261] T 472d7827b3074236a81a3ede098f6c0a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:12.278887 20233 raft_consensus.cc:1261] T 472d7827b3074236a81a3ede098f6c0a P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:12.278884 23908 consensus_queue.cc:1035] T 472d7827b3074236a81a3ede098f6c0a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:12.279424 22876 consensus_queue.cc:1035] T 472d7827b3074236a81a3ede098f6c0a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:12.278649 17604 raft_consensus.cc:1261] T 472d7827b3074236a81a3ede098f6c0a P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEARNER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:12.295264 23611 consensus_queue.cc:1035] T 472d7827b3074236a81a3ede098f6c0a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 01:54:12.305352 17800 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.42.66:38531, user_credentials={real_user=slave}} blocked reactor thread for 102480us
I20240502 01:54:12.171351 24341 ts_tablet_manager.cc:1875] T 59ce542c545f4ba590e41690e8df5fc6 P bb31bceba76d466db24b9341c04900c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:12.309907 24331 tablet_bootstrap.cc:654] T c491a39ca87847d9ace688346c0baf57 P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
W20240502 01:54:12.304956 20433 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.42.65:38321, user_credentials={real_user=slave}} blocked reactor thread for 86392.7us
W20240502 01:54:12.312682 20562 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.42.66:38531, user_credentials={real_user=slave}} blocked reactor thread for 133018us
I20240502 01:54:12.315241 24332 raft_consensus.cc:348] T fa4d38dc9b674a9b90d4793dad2687ec P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:12.315601 24332 raft_consensus.cc:374] T fa4d38dc9b674a9b90d4793dad2687ec P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:12.315651 24321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68bc23294ac64c9486bffebbdb16b508. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:12.315841 20625 raft_consensus.cc:2932] T 4bb42883b1a0475b863d008bd01fa70a P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:12.327024 24166 raft_consensus.cc:1050] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3: attempting to promote NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 to VOTER
I20240502 01:54:12.327751 24332 raft_consensus.cc:729] T fa4d38dc9b674a9b90d4793dad2687ec P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:54:12.327953 24166 consensus_queue.cc:237] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } }
I20240502 01:54:12.328140 24332 consensus_queue.cc:260] T fa4d38dc9b674a9b90d4793dad2687ec P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:12.329106 17322 catalog_manager.cc:4830] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): tablet 59ce542c545f4ba590e41690e8df5fc6 (table rebalancer_dynamic_table_new_name [id=eb791abbdca04194aa499cb7e38e1dc6]) successfully deleted
I20240502 01:54:12.331669 23908 raft_consensus.cc:2932] T 472d7827b3074236a81a3ede098f6c0a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:12.337589 20232 raft_consensus.cc:2932] T 472d7827b3074236a81a3ede098f6c0a P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:12.340723 17326 catalog_manager.cc:5484] T 472d7827b3074236a81a3ede098f6c0a P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:12.340576 17474 raft_consensus.cc:2932] T 472d7827b3074236a81a3ede098f6c0a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:12.348553 24331 tablet_bootstrap.cc:492] T c491a39ca87847d9ace688346c0baf57 P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:54:12.350116 24331 ts_tablet_manager.cc:1367] T c491a39ca87847d9ace688346c0baf57 P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.280s user 0.004s sys 0.000s
I20240502 01:54:12.351140 24331 raft_consensus.cc:348] T c491a39ca87847d9ace688346c0baf57 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:12.355060 24331 raft_consensus.cc:374] T c491a39ca87847d9ace688346c0baf57 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:12.355798 24331 raft_consensus.cc:729] T c491a39ca87847d9ace688346c0baf57 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:54:12.356468 24331 consensus_queue.cc:260] T c491a39ca87847d9ace688346c0baf57 P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:12.356886 24332 ts_tablet_manager.cc:1398] T fa4d38dc9b674a9b90d4793dad2687ec P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.185s user 0.002s sys 0.000s
I20240502 01:54:12.357210 24332 tablet_bootstrap.cc:492] T c491a39ca87847d9ace688346c0baf57 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:54:12.358994 24331 ts_tablet_manager.cc:1398] T c491a39ca87847d9ace688346c0baf57 P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240502 01:54:12.359423 24331 tablet_bootstrap.cc:492] T 59be89ca2c0b4275844f1e73cd20595b P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:54:12.360325 20233 raft_consensus.cc:1261] T ed1522ec150a46f28fd553f9b563651f P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:12.360858 24362 consensus_queue.cc:1035] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:12.369498 24331 tablet_bootstrap.cc:654] T 59be89ca2c0b4275844f1e73cd20595b P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:12.378569 17474 raft_consensus.cc:1261] T ed1522ec150a46f28fd553f9b563651f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:12.386693 24362 consensus_queue.cc:1035] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:12.389114 20366 raft_consensus.cc:1261] T ed1522ec150a46f28fd553f9b563651f P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:12.394114 24332 tablet_bootstrap.cc:654] T c491a39ca87847d9ace688346c0baf57 P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:12.394114 24331 tablet_bootstrap.cc:492] T 59be89ca2c0b4275844f1e73cd20595b P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:54:12.394295 24331 ts_tablet_manager.cc:1367] T 59be89ca2c0b4275844f1e73cd20595b P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.035s user 0.008s sys 0.002s
I20240502 01:54:12.394202 24363 consensus_queue.cc:1035] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:12.395171 24331 raft_consensus.cc:348] T 59be89ca2c0b4275844f1e73cd20595b P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:12.395447 24331 raft_consensus.cc:374] T 59be89ca2c0b4275844f1e73cd20595b P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:12.395550 24331 raft_consensus.cc:729] T 59be89ca2c0b4275844f1e73cd20595b P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:54:12.395992 24331 consensus_queue.cc:260] T 59be89ca2c0b4275844f1e73cd20595b P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:12.397080 24329 tablet_bootstrap.cc:492] T 8e7f88b6c509477fb0900024e7de8f00 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:12.397648 24329 tablet_bootstrap.cc:492] T 8e7f88b6c509477fb0900024e7de8f00 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap complete.
I20240502 01:54:12.397951 24329 ts_tablet_manager.cc:1367] T 8e7f88b6c509477fb0900024e7de8f00 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.230s user 0.053s sys 0.000s
I20240502 01:54:12.398702 24329 raft_consensus.cc:348] T 8e7f88b6c509477fb0900024e7de8f00 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } }
I20240502 01:54:12.398955 24329 raft_consensus.cc:729] T 8e7f88b6c509477fb0900024e7de8f00 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: LEARNER
I20240502 01:54:12.399144 24329 consensus_queue.cc:260] T 8e7f88b6c509477fb0900024e7de8f00 P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } }
I20240502 01:54:12.399737 24329 ts_tablet_manager.cc:1398] T 8e7f88b6c509477fb0900024e7de8f00 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:54:12.400223 24331 ts_tablet_manager.cc:1398] T 59be89ca2c0b4275844f1e73cd20595b P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 01:54:12.400770 17757 tablet_copy_service.cc:342] P dfa1152987644bd781b46c7e42a7a685: Request end of tablet copy session 6a36c952c51542c0bf7ea3ec2857a9f9-8e7f88b6c509477fb0900024e7de8f00 received from {username='slave'} at 127.11.42.70:55001
I20240502 01:54:12.400887 17757 tablet_copy_service.cc:433] P dfa1152987644bd781b46c7e42a7a685: ending tablet copy session 6a36c952c51542c0bf7ea3ec2857a9f9-8e7f88b6c509477fb0900024e7de8f00 on tablet 8e7f88b6c509477fb0900024e7de8f00 with peer 6a36c952c51542c0bf7ea3ec2857a9f9
I20240502 01:54:12.404654 24331 tablet_bootstrap.cc:492] T 57e3c62666b1428ba01ef75abd15e3ef P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:54:12.409001 24339 tablet_copy_client.cc:783] T 75bb4e19cf57405fabcb532b55a29ced P 4da690c0fa13407a85c0689ce8de9ade: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:12.409265 24339 tablet_copy_client.cc:647] T 75bb4e19cf57405fabcb532b55a29ced P 4da690c0fa13407a85c0689ce8de9ade: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:12.409622 24362 raft_consensus.cc:2932] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:54:12.412715 24339 tablet_copy_client.cc:523] T 75bb4e19cf57405fabcb532b55a29ced P 4da690c0fa13407a85c0689ce8de9ade: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:12.415553 24341 ts_tablet_manager.cc:1888] T 59ce542c545f4ba590e41690e8df5fc6 P bb31bceba76d466db24b9341c04900c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:12.416604 24341 log.cc:1196] T 59ce542c545f4ba590e41690e8df5fc6 P bb31bceba76d466db24b9341c04900c3: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/wals/59ce542c545f4ba590e41690e8df5fc6
I20240502 01:54:12.417248 24331 tablet_bootstrap.cc:654] T 57e3c62666b1428ba01ef75abd15e3ef P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:12.417537 17474 raft_consensus.cc:2932] T ed1522ec150a46f28fd553f9b563651f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:54:12.418154 17328 catalog_manager.cc:5484] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 reported cstate change: config changed from index 7 to 8, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:12.425593 20366 raft_consensus.cc:2932] T ed1522ec150a46f28fd553f9b563651f P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:54:12.428555 24332 tablet_bootstrap.cc:492] T c491a39ca87847d9ace688346c0baf57 P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:54:12.418691 24390 tablet_bootstrap.cc:492] T 57e3c62666b1428ba01ef75abd15e3ef P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:54:12.428993 24341 ts_tablet_manager.cc:1909] T 59ce542c545f4ba590e41690e8df5fc6 P bb31bceba76d466db24b9341c04900c3: Deleting consensus metadata
I20240502 01:54:12.430076 24390 tablet_bootstrap.cc:654] T 57e3c62666b1428ba01ef75abd15e3ef P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:12.430053 17322 catalog_manager.cc:4830] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): tablet 59ce542c545f4ba590e41690e8df5fc6 (table rebalancer_dynamic_table_new_name [id=eb791abbdca04194aa499cb7e38e1dc6]) successfully deleted
I20240502 01:54:12.430640 24332 ts_tablet_manager.cc:1367] T c491a39ca87847d9ace688346c0baf57 P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.073s user 0.005s sys 0.000s
I20240502 01:54:12.436233 24339 tablet_bootstrap.cc:492] T 75bb4e19cf57405fabcb532b55a29ced P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:54:12.444276 24381 ts_tablet_manager.cc:902] T 2714919062f5465c8cde1a3c4720978a P 3fceccefba274b1dbbd8216af8b0bc29: Initiating tablet copy from peer 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:54:12.446913 24332 raft_consensus.cc:348] T c491a39ca87847d9ace688346c0baf57 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:12.447197 24332 raft_consensus.cc:374] T c491a39ca87847d9ace688346c0baf57 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:12.447345 24332 raft_consensus.cc:729] T c491a39ca87847d9ace688346c0baf57 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:54:12.447695 24332 consensus_queue.cc:260] T c491a39ca87847d9ace688346c0baf57 P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:12.449522 24332 ts_tablet_manager.cc:1398] T c491a39ca87847d9ace688346c0baf57 P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20240502 01:54:12.449836 24332 tablet_bootstrap.cc:492] T 57e3c62666b1428ba01ef75abd15e3ef P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:54:12.453071 24332 tablet_bootstrap.cc:654] T 57e3c62666b1428ba01ef75abd15e3ef P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:12.460975 24390 tablet_bootstrap.cc:492] T 57e3c62666b1428ba01ef75abd15e3ef P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:54:12.462066 24390 ts_tablet_manager.cc:1367] T 57e3c62666b1428ba01ef75abd15e3ef P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.043s user 0.007s sys 0.000s
I20240502 01:54:12.466749 24331 tablet_bootstrap.cc:492] T 57e3c62666b1428ba01ef75abd15e3ef P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:54:12.463107 24390 raft_consensus.cc:348] T 57e3c62666b1428ba01ef75abd15e3ef P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:12.466948 24390 raft_consensus.cc:374] T 57e3c62666b1428ba01ef75abd15e3ef P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:12.467056 24390 raft_consensus.cc:729] T 57e3c62666b1428ba01ef75abd15e3ef P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:54:12.467327 24390 consensus_queue.cc:260] T 57e3c62666b1428ba01ef75abd15e3ef P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:12.467839 24331 ts_tablet_manager.cc:1367] T 57e3c62666b1428ba01ef75abd15e3ef P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.063s user 0.009s sys 0.000s
I20240502 01:54:12.468837 24331 raft_consensus.cc:348] T 57e3c62666b1428ba01ef75abd15e3ef P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:12.469429 24390 ts_tablet_manager.cc:1398] T 57e3c62666b1428ba01ef75abd15e3ef P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240502 01:54:12.469673 24331 raft_consensus.cc:374] T 57e3c62666b1428ba01ef75abd15e3ef P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:12.469818 24331 raft_consensus.cc:729] T 57e3c62666b1428ba01ef75abd15e3ef P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:54:12.470079 24331 consensus_queue.cc:260] T 57e3c62666b1428ba01ef75abd15e3ef P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:12.472070 24331 ts_tablet_manager.cc:1398] T 57e3c62666b1428ba01ef75abd15e3ef P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:54:12.479655 24332 tablet_bootstrap.cc:492] T 57e3c62666b1428ba01ef75abd15e3ef P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:54:12.479804 24332 ts_tablet_manager.cc:1367] T 57e3c62666b1428ba01ef75abd15e3ef P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.000s
I20240502 01:54:12.480466 24332 raft_consensus.cc:348] T 57e3c62666b1428ba01ef75abd15e3ef P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:12.476971 20366 consensus_queue.cc:237] T 472d7827b3074236a81a3ede098f6c0a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } }
I20240502 01:54:12.483107 17604 raft_consensus.cc:1261] T 472d7827b3074236a81a3ede098f6c0a P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:12.484220 23908 consensus_queue.cc:1035] T 472d7827b3074236a81a3ede098f6c0a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:12.485457 17604 raft_consensus.cc:2932] T 472d7827b3074236a81a3ede098f6c0a P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:12.490696 17867 consensus_queue.cc:237] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } }
I20240502 01:54:12.492987 20231 raft_consensus.cc:2932] T ed1522ec150a46f28fd553f9b563651f P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:54:12.498622 24332 raft_consensus.cc:374] T 57e3c62666b1428ba01ef75abd15e3ef P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:12.499009 24339 tablet_bootstrap.cc:492] T 75bb4e19cf57405fabcb532b55a29ced P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:12.499614 24339 tablet_bootstrap.cc:492] T 75bb4e19cf57405fabcb532b55a29ced P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap complete.
I20240502 01:54:12.493274 17474 raft_consensus.cc:1261] T 472d7827b3074236a81a3ede098f6c0a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:12.499080 24332 raft_consensus.cc:729] T 57e3c62666b1428ba01ef75abd15e3ef P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:54:12.499976 24332 consensus_queue.cc:260] T 57e3c62666b1428ba01ef75abd15e3ef P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:12.500233 23908 consensus_queue.cc:1035] T 472d7827b3074236a81a3ede098f6c0a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:12.502528 24332 ts_tablet_manager.cc:1398] T 57e3c62666b1428ba01ef75abd15e3ef P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20240502 01:54:12.502846 24339 ts_tablet_manager.cc:1367] T 75bb4e19cf57405fabcb532b55a29ced P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.067s user 0.008s sys 0.000s
I20240502 01:54:12.503039 23908 raft_consensus.cc:2932] T 472d7827b3074236a81a3ede098f6c0a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:12.505066 23611 raft_consensus.cc:482] T c491a39ca87847d9ace688346c0baf57 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:12.505225 23611 raft_consensus.cc:504] T c491a39ca87847d9ace688346c0baf57 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:12.505823 23611 leader_election.cc:288] T c491a39ca87847d9ace688346c0baf57 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:54:12.505777 24339 raft_consensus.cc:348] T 75bb4e19cf57405fabcb532b55a29ced P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } }
I20240502 01:54:12.506140 24339 raft_consensus.cc:729] T 75bb4e19cf57405fabcb532b55a29ced P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: LEARNER
I20240502 01:54:12.506381 24339 consensus_queue.cc:260] T 75bb4e19cf57405fabcb532b55a29ced P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } }
I20240502 01:54:12.506903 20625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c491a39ca87847d9ace688346c0baf57" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091" is_pre_election: true
I20240502 01:54:12.507144 20625 raft_consensus.cc:2454] T c491a39ca87847d9ace688346c0baf57 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 0.
I20240502 01:54:12.507601 24339 ts_tablet_manager.cc:1398] T 75bb4e19cf57405fabcb532b55a29ced P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 01:54:12.508590 17476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c491a39ca87847d9ace688346c0baf57" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:54:12.508795 17476 raft_consensus.cc:2454] T c491a39ca87847d9ace688346c0baf57 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 0.
I20240502 01:54:12.509331 20649 tablet_copy_service.cc:342] P f19cc6cb1e76460aa6711e49a5775091: Request end of tablet copy session 4da690c0fa13407a85c0689ce8de9ade-75bb4e19cf57405fabcb532b55a29ced received from {username='slave'} at 127.11.42.65:34833
I20240502 01:54:12.509443 20649 tablet_copy_service.cc:433] P f19cc6cb1e76460aa6711e49a5775091: ending tablet copy session 4da690c0fa13407a85c0689ce8de9ade-75bb4e19cf57405fabcb532b55a29ced on tablet 75bb4e19cf57405fabcb532b55a29ced with peer 4da690c0fa13407a85c0689ce8de9ade
I20240502 01:54:12.510116 20302 leader_election.cc:302] T c491a39ca87847d9ace688346c0baf57 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:54:12.510411 23611 raft_consensus.cc:2781] T c491a39ca87847d9ace688346c0baf57 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:12.510502 23611 raft_consensus.cc:482] T c491a39ca87847d9ace688346c0baf57 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:12.510604 23611 raft_consensus.cc:3037] T c491a39ca87847d9ace688346c0baf57 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:12.513957 17474 raft_consensus.cc:2932] T 472d7827b3074236a81a3ede098f6c0a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:12.516258 20232 raft_consensus.cc:1261] T ed1522ec150a46f28fd553f9b563651f P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:12.516819 24166 consensus_queue.cc:1035] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:12.517266 24146 raft_consensus.cc:482] T fa4d38dc9b674a9b90d4793dad2687ec P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:12.517426 24146 raft_consensus.cc:504] T fa4d38dc9b674a9b90d4793dad2687ec P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:12.518185 24146 leader_election.cc:288] T fa4d38dc9b674a9b90d4793dad2687ec P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:54:12.519948 24167 raft_consensus.cc:2932] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:54:12.524130 17322 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ed1522ec150a46f28fd553f9b563651f with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:12.524187 17476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa4d38dc9b674a9b90d4793dad2687ec" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:54:12.524381 17476 raft_consensus.cc:2454] T fa4d38dc9b674a9b90d4793dad2687ec P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:54:12.524930 20362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa4d38dc9b674a9b90d4793dad2687ec" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:54:12.525154 20362 raft_consensus.cc:2454] T fa4d38dc9b674a9b90d4793dad2687ec P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:54:12.527062 17542 leader_election.cc:302] T fa4d38dc9b674a9b90d4793dad2687ec P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, 4da690c0fa13407a85c0689ce8de9ade; no voters:
I20240502 01:54:12.528228 23611 raft_consensus.cc:504] T c491a39ca87847d9ace688346c0baf57 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:12.528579 17334 catalog_manager.cc:5484] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 reported cstate change: config changed from index 8 to 9, VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) evicted. New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:12.529166 23611 leader_election.cc:288] T c491a39ca87847d9ace688346c0baf57 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Requested vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:54:12.530051 20625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c491a39ca87847d9ace688346c0baf57" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091"
I20240502 01:54:12.530282 20625 raft_consensus.cc:3037] T c491a39ca87847d9ace688346c0baf57 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:12.530537 20231 raft_consensus.cc:2932] T ed1522ec150a46f28fd553f9b563651f P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:54:12.532260 23807 raft_consensus.cc:2781] T fa4d38dc9b674a9b90d4793dad2687ec P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:12.532390 23807 raft_consensus.cc:482] T fa4d38dc9b674a9b90d4793dad2687ec P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:12.532469 23807 raft_consensus.cc:3037] T fa4d38dc9b674a9b90d4793dad2687ec P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:12.535262 23807 raft_consensus.cc:504] T fa4d38dc9b674a9b90d4793dad2687ec P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:12.535936 23807 leader_election.cc:288] T fa4d38dc9b674a9b90d4793dad2687ec P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Requested vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:54:12.537600 20625 raft_consensus.cc:2454] T c491a39ca87847d9ace688346c0baf57 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 1.
I20240502 01:54:12.538992 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c491a39ca87847d9ace688346c0baf57" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:54:12.539177 17475 raft_consensus.cc:3037] T c491a39ca87847d9ace688346c0baf57 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:12.541868 17475 raft_consensus.cc:2454] T c491a39ca87847d9ace688346c0baf57 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 1.
I20240502 01:54:12.543290 24321 tablet_copy_client.cc:783] T 68bc23294ac64c9486bffebbdb16b508 P 3fceccefba274b1dbbd8216af8b0bc29: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:12.543542 24321 tablet_copy_client.cc:647] T 68bc23294ac64c9486bffebbdb16b508 P 3fceccefba274b1dbbd8216af8b0bc29: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:12.543524 17476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa4d38dc9b674a9b90d4793dad2687ec" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:54:12.543687 17476 raft_consensus.cc:3037] T fa4d38dc9b674a9b90d4793dad2687ec P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:12.551750 20302 leader_election.cc:302] T c491a39ca87847d9ace688346c0baf57 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:54:12.552503 22876 raft_consensus.cc:2781] T c491a39ca87847d9ace688346c0baf57 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:12.552624 22876 raft_consensus.cc:686] T c491a39ca87847d9ace688346c0baf57 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Becoming Leader. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Running, Role: LEADER
I20240502 01:54:12.552897 22876 consensus_queue.cc:237] T c491a39ca87847d9ace688346c0baf57 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:12.554510 20362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa4d38dc9b674a9b90d4793dad2687ec" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
I20240502 01:54:12.554687 20362 raft_consensus.cc:3037] T fa4d38dc9b674a9b90d4793dad2687ec P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:12.559589 17737 raft_consensus.cc:1203] T 1b14ac3413004127b72904b631ec1b94 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:12.563181 24359 raft_consensus.cc:482] T 59be89ca2c0b4275844f1e73cd20595b P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:12.563323 24359 raft_consensus.cc:504] T 59be89ca2c0b4275844f1e73cd20595b P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:12.564023 24359 leader_election.cc:288] T 59be89ca2c0b4275844f1e73cd20595b P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:54:12.564889 20364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59be89ca2c0b4275844f1e73cd20595b" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:54:12.565608 20364 raft_consensus.cc:1203] T 8e7f88b6c509477fb0900024e7de8f00 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 01:54:12.573755 17476 raft_consensus.cc:2454] T fa4d38dc9b674a9b90d4793dad2687ec P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:54:12.574373 17542 leader_election.cc:302] T fa4d38dc9b674a9b90d4793dad2687ec P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, 4da690c0fa13407a85c0689ce8de9ade; no voters:
I20240502 01:54:12.575768 17456 tablet_service.cc:1514] Processing DeleteTablet for tablet ed1522ec150a46f28fd553f9b563651f with delete_type TABLET_DATA_TOMBSTONED (TS 4da690c0fa13407a85c0689ce8de9ade not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:54520
I20240502 01:54:12.576133 24344 tablet_replica.cc:330] T ed1522ec150a46f28fd553f9b563651f P 4da690c0fa13407a85c0689ce8de9ade: stopping tablet replica
I20240502 01:54:12.576334 24344 raft_consensus.cc:2229] T ed1522ec150a46f28fd553f9b563651f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:12.576589 24344 raft_consensus.cc:2258] T ed1522ec150a46f28fd553f9b563651f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:12.577201 24363 raft_consensus.cc:482] T 57e3c62666b1428ba01ef75abd15e3ef P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:12.577355 24363 raft_consensus.cc:504] T 57e3c62666b1428ba01ef75abd15e3ef P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:12.577819 24344 ts_tablet_manager.cc:1875] T ed1522ec150a46f28fd553f9b563651f P 4da690c0fa13407a85c0689ce8de9ade: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:12.577992 24363 leader_election.cc:288] T 57e3c62666b1428ba01ef75abd15e3ef P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:54:12.579174 24321 tablet_copy_client.cc:523] T 68bc23294ac64c9486bffebbdb16b508 P 3fceccefba274b1dbbd8216af8b0bc29: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:12.584235 24146 raft_consensus.cc:2781] T fa4d38dc9b674a9b90d4793dad2687ec P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:12.584429 24146 raft_consensus.cc:686] T fa4d38dc9b674a9b90d4793dad2687ec P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Becoming Leader. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Running, Role: LEADER
I20240502 01:54:12.584458 24321 tablet_bootstrap.cc:492] T 68bc23294ac64c9486bffebbdb16b508 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:54:12.584690 24146 consensus_queue.cc:237] T fa4d38dc9b674a9b90d4793dad2687ec P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:12.594792 17475 raft_consensus.cc:1203] T 75bb4e19cf57405fabcb532b55a29ced P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:12.603142 20362 raft_consensus.cc:2454] T fa4d38dc9b674a9b90d4793dad2687ec P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:54:12.616590 24344 ts_tablet_manager.cc:1888] T ed1522ec150a46f28fd553f9b563651f P 4da690c0fa13407a85c0689ce8de9ade: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:54:12.619920 24125 raft_consensus.cc:482] T 57e3c62666b1428ba01ef75abd15e3ef P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:12.620085 24125 raft_consensus.cc:504] T 57e3c62666b1428ba01ef75abd15e3ef P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:12.620711 24125 leader_election.cc:288] T 57e3c62666b1428ba01ef75abd15e3ef P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:54:12.622360 17867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e3c62666b1428ba01ef75abd15e3ef" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:54:12.622568 17867 raft_consensus.cc:2454] T 57e3c62666b1428ba01ef75abd15e3ef P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 0.
I20240502 01:54:12.623113 20565 leader_election.cc:302] T 57e3c62666b1428ba01ef75abd15e3ef P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb31bceba76d466db24b9341c04900c3, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:54:12.623944 24125 raft_consensus.cc:2781] T 57e3c62666b1428ba01ef75abd15e3ef P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:12.624050 24125 raft_consensus.cc:482] T 57e3c62666b1428ba01ef75abd15e3ef P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:12.624146 24125 raft_consensus.cc:3037] T 57e3c62666b1428ba01ef75abd15e3ef P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:12.627707 24125 raft_consensus.cc:504] T 57e3c62666b1428ba01ef75abd15e3ef P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:12.630448 17867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e3c62666b1428ba01ef75abd15e3ef" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:54:12.630630 17867 raft_consensus.cc:3037] T 57e3c62666b1428ba01ef75abd15e3ef P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:12.635231 17867 raft_consensus.cc:2454] T 57e3c62666b1428ba01ef75abd15e3ef P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 1.
I20240502 01:54:12.635772 20565 leader_election.cc:302] T 57e3c62666b1428ba01ef75abd15e3ef P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb31bceba76d466db24b9341c04900c3, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:54:12.636103 24127 raft_consensus.cc:2781] T 57e3c62666b1428ba01ef75abd15e3ef P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:12.636224 24127 raft_consensus.cc:686] T 57e3c62666b1428ba01ef75abd15e3ef P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Becoming Leader. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Running, Role: LEADER
I20240502 01:54:12.636508 24127 consensus_queue.cc:237] T 57e3c62666b1428ba01ef75abd15e3ef P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:12.641170 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 472d7827b3074236a81a3ede098f6c0a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:12.644392 24321 tablet_bootstrap.cc:492] T 68bc23294ac64c9486bffebbdb16b508 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:12.644845 24321 tablet_bootstrap.cc:492] T 68bc23294ac64c9486bffebbdb16b508 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap complete.
I20240502 01:54:12.647722 20498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59be89ca2c0b4275844f1e73cd20595b" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
I20240502 01:54:12.647998 20498 raft_consensus.cc:2379] T 59be89ca2c0b4275844f1e73cd20595b P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f19cc6cb1e76460aa6711e49a5775091 in current term 1: Already voted for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in this term.
I20240502 01:54:12.650234 20565 leader_election.cc:302] T 59be89ca2c0b4275844f1e73cd20595b P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f19cc6cb1e76460aa6711e49a5775091; no voters: 2f306549407b4bb281ccf2994035e4f1, 6a36c952c51542c0bf7ea3ec2857a9f9
I20240502 01:54:12.650743 24127 raft_consensus.cc:3037] T 59be89ca2c0b4275844f1e73cd20595b P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:12.655449 24127 raft_consensus.cc:2726] T 59be89ca2c0b4275844f1e73cd20595b P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:54:12.656738 24321 ts_tablet_manager.cc:1367] T 68bc23294ac64c9486bffebbdb16b508 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.072s user 0.002s sys 0.008s
I20240502 01:54:12.659915 24321 raft_consensus.cc:348] T 68bc23294ac64c9486bffebbdb16b508 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } }
I20240502 01:54:12.660156 24321 raft_consensus.cc:729] T 68bc23294ac64c9486bffebbdb16b508 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: LEARNER
I20240502 01:54:12.660382 24321 consensus_queue.cc:260] T 68bc23294ac64c9486bffebbdb16b508 P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } }
I20240502 01:54:12.663893 17334 catalog_manager.cc:5484] T 472d7827b3074236a81a3ede098f6c0a P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: config changed from index 3 to 4, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:12.664717 24321 ts_tablet_manager.cc:1398] T 68bc23294ac64c9486bffebbdb16b508 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240502 01:54:12.666992 20362 raft_consensus.cc:1261] T ed1522ec150a46f28fd553f9b563651f P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:12.668648 20625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e3c62666b1428ba01ef75abd15e3ef" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091" is_pre_election: true
I20240502 01:54:12.669615 20211 tablet_service.cc:1514] Processing DeleteTablet for tablet 472d7827b3074236a81a3ede098f6c0a with delete_type TABLET_DATA_TOMBSTONED (TS fdc2ac372492454c85387589a91f63af not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44624
I20240502 01:54:12.670007 24340 tablet_replica.cc:330] T 472d7827b3074236a81a3ede098f6c0a P fdc2ac372492454c85387589a91f63af: stopping tablet replica
I20240502 01:54:12.670220 24340 raft_consensus.cc:2229] T 472d7827b3074236a81a3ede098f6c0a P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:12.670436 24340 raft_consensus.cc:2258] T 472d7827b3074236a81a3ede098f6c0a P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:12.671564 24340 ts_tablet_manager.cc:1875] T 472d7827b3074236a81a3ede098f6c0a P fdc2ac372492454c85387589a91f63af: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:12.677445 24167 consensus_queue.cc:1035] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:12.678980 20362 raft_consensus.cc:2932] T ed1522ec150a46f28fd553f9b563651f P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:54:12.681377 24340 ts_tablet_manager.cc:1888] T 472d7827b3074236a81a3ede098f6c0a P fdc2ac372492454c85387589a91f63af: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:54:12.681524 24340 log.cc:1196] T 472d7827b3074236a81a3ede098f6c0a P fdc2ac372492454c85387589a91f63af: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal/wals/472d7827b3074236a81a3ede098f6c0a
I20240502 01:54:12.684489 24125 leader_election.cc:288] T 57e3c62666b1428ba01ef75abd15e3ef P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 election: Requested vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:54:12.621781 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e3c62666b1428ba01ef75abd15e3ef" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:54:12.630192 17473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e3c62666b1428ba01ef75abd15e3ef" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:54:12.688586 17475 raft_consensus.cc:2454] T 57e3c62666b1428ba01ef75abd15e3ef P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 0.
I20240502 01:54:12.688688 17322 catalog_manager.cc:4830] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): tablet 472d7827b3074236a81a3ede098f6c0a (table rebalancer_extra_table_12 [id=ecb981650f8c425798b67a5194ad793c]) successfully deleted
I20240502 01:54:12.668133 17476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e3c62666b1428ba01ef75abd15e3ef" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:54:12.688899 17476 raft_consensus.cc:2454] T 57e3c62666b1428ba01ef75abd15e3ef P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 0.
I20240502 01:54:12.689929 20649 tablet_copy_service.cc:342] P f19cc6cb1e76460aa6711e49a5775091: Request end of tablet copy session 3fceccefba274b1dbbd8216af8b0bc29-68bc23294ac64c9486bffebbdb16b508 received from {username='slave'} at 127.11.42.66:54577
I20240502 01:54:12.690066 20649 tablet_copy_service.cc:433] P f19cc6cb1e76460aa6711e49a5775091: ending tablet copy session 3fceccefba274b1dbbd8216af8b0bc29-68bc23294ac64c9486bffebbdb16b508 on tablet 68bc23294ac64c9486bffebbdb16b508 with peer 3fceccefba274b1dbbd8216af8b0bc29
I20240502 01:54:12.691138 17802 leader_election.cc:302] T 57e3c62666b1428ba01ef75abd15e3ef P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, bb31bceba76d466db24b9341c04900c3; no voters: f19cc6cb1e76460aa6711e49a5775091
I20240502 01:54:12.691466 24167 raft_consensus.cc:2741] T 57e3c62666b1428ba01ef75abd15e3ef P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240502 01:54:12.688401 24344 log.cc:1196] T ed1522ec150a46f28fd553f9b563651f P 4da690c0fa13407a85c0689ce8de9ade: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/ed1522ec150a46f28fd553f9b563651f
I20240502 01:54:12.693514 17321 catalog_manager.cc:4830] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): tablet ed1522ec150a46f28fd553f9b563651f (table rebalance_test_table_11 [id=5e2315325f6c4e7f8834471d07555cf5]) successfully deleted
I20240502 01:54:12.697856 17328 catalog_manager.cc:5484] T 57e3c62666b1428ba01ef75abd15e3ef P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: term changed from 0 to 1, leader changed from <none> to f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72). New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:12.705090 17606 raft_consensus.cc:2932] T 472d7827b3074236a81a3ede098f6c0a P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:12.722011 24359 raft_consensus.cc:1050] T fbd94103cae04473ba81709da39d7ed8 P f19cc6cb1e76460aa6711e49a5775091: attempting to promote NON_VOTER dfa1152987644bd781b46c7e42a7a685 to VOTER
I20240502 01:54:12.722887 24359 consensus_queue.cc:237] T fbd94103cae04473ba81709da39d7ed8 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
W20240502 01:54:12.726486 17543 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.42.69:34339, user_credentials={real_user=slave}} blocked reactor thread for 151632us
I20240502 01:54:12.728753 17334 catalog_manager.cc:5484] T fa4d38dc9b674a9b90d4793dad2687ec P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66). New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:12.740701 17737 raft_consensus.cc:1261] T fbd94103cae04473ba81709da39d7ed8 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:12.743655 20232 raft_consensus.cc:1261] T fbd94103cae04473ba81709da39d7ed8 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:12.743753 20494 raft_consensus.cc:1261] T fbd94103cae04473ba81709da39d7ed8 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:12.743201 24127 consensus_queue.cc:1035] T fbd94103cae04473ba81709da39d7ed8 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 01:54:12.744779 24125 consensus_queue.cc:1035] T fbd94103cae04473ba81709da39d7ed8 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:12.755729 24125 consensus_queue.cc:1035] T fbd94103cae04473ba81709da39d7ed8 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 01:54:12.813299 20303 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.42.69:34339, user_credentials={real_user=slave}} blocked reactor thread for 74098.4us
I20240502 01:54:12.822572 24381 tablet_copy_client.cc:308] T 2714919062f5465c8cde1a3c4720978a P 3fceccefba274b1dbbd8216af8b0bc29: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.70:39155
I20240502 01:54:12.823199 20387 tablet_copy_service.cc:140] P 6a36c952c51542c0bf7ea3ec2857a9f9: Received BeginTabletCopySession request for tablet 2714919062f5465c8cde1a3c4720978a from peer 3fceccefba274b1dbbd8216af8b0bc29 ({username='slave'} at 127.11.42.66:36601)
I20240502 01:54:12.823381 20387 tablet_copy_service.cc:161] P 6a36c952c51542c0bf7ea3ec2857a9f9: Beginning new tablet copy session on tablet 2714919062f5465c8cde1a3c4720978a from peer 3fceccefba274b1dbbd8216af8b0bc29 at {username='slave'} at 127.11.42.66:36601: session id = 3fceccefba274b1dbbd8216af8b0bc29-2714919062f5465c8cde1a3c4720978a
I20240502 01:54:12.823145 17334 catalog_manager.cc:5484] T c491a39ca87847d9ace688346c0baf57 P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70). New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:12.826004 20387 tablet_copy_source_session.cc:205] T 2714919062f5465c8cde1a3c4720978a P 6a36c952c51542c0bf7ea3ec2857a9f9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:12.827724 24381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2714919062f5465c8cde1a3c4720978a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:12.835407 24381 tablet_copy_client.cc:783] T 2714919062f5465c8cde1a3c4720978a P 3fceccefba274b1dbbd8216af8b0bc29: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:12.835656 24381 tablet_copy_client.cc:647] T 2714919062f5465c8cde1a3c4720978a P 3fceccefba274b1dbbd8216af8b0bc29: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:12.836102 24127 raft_consensus.cc:2932] T fbd94103cae04473ba81709da39d7ed8 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:12.838310 22876 consensus_queue.cc:1035] T 59be89ca2c0b4275844f1e73cd20595b P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:54:12.839154 20564 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.42.67:36115, user_credentials={real_user=slave}} blocked reactor thread for 97913.5us
I20240502 01:54:12.842821 24381 tablet_copy_client.cc:523] T 2714919062f5465c8cde1a3c4720978a P 3fceccefba274b1dbbd8216af8b0bc29: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:12.848835 20494 raft_consensus.cc:2932] T fbd94103cae04473ba81709da39d7ed8 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:12.850327 24381 tablet_bootstrap.cc:492] T 2714919062f5465c8cde1a3c4720978a P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:54:12.850090 20232 raft_consensus.cc:2932] T fbd94103cae04473ba81709da39d7ed8 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:12.851444 17334 catalog_manager.cc:5484] T fbd94103cae04473ba81709da39d7ed8 P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:12.870680 24381 tablet_bootstrap.cc:492] T 2714919062f5465c8cde1a3c4720978a P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:12.871956 24381 tablet_bootstrap.cc:492] T 2714919062f5465c8cde1a3c4720978a P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap complete.
I20240502 01:54:12.872236 24381 ts_tablet_manager.cc:1367] T 2714919062f5465c8cde1a3c4720978a P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.001s
I20240502 01:54:12.872439 23908 consensus_queue.cc:1035] T 59be89ca2c0b4275844f1e73cd20595b P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:12.873744 24381 raft_consensus.cc:348] T 2714919062f5465c8cde1a3c4720978a P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } }
I20240502 01:54:12.873993 24381 raft_consensus.cc:729] T 2714919062f5465c8cde1a3c4720978a P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: LEARNER
I20240502 01:54:12.874217 24381 consensus_queue.cc:260] T 2714919062f5465c8cde1a3c4720978a P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } }
I20240502 01:54:12.877339 24381 ts_tablet_manager.cc:1398] T 2714919062f5465c8cde1a3c4720978a P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 01:54:12.878407 20387 tablet_copy_service.cc:342] P 6a36c952c51542c0bf7ea3ec2857a9f9: Request end of tablet copy session 3fceccefba274b1dbbd8216af8b0bc29-2714919062f5465c8cde1a3c4720978a received from {username='slave'} at 127.11.42.66:36601
I20240502 01:54:12.878526 20387 tablet_copy_service.cc:433] P 6a36c952c51542c0bf7ea3ec2857a9f9: ending tablet copy session 3fceccefba274b1dbbd8216af8b0bc29-2714919062f5465c8cde1a3c4720978a on tablet 2714919062f5465c8cde1a3c4720978a with peer 3fceccefba274b1dbbd8216af8b0bc29
I20240502 01:54:12.893576 17737 raft_consensus.cc:2932] T fbd94103cae04473ba81709da39d7ed8 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:12.907400 20625 consensus_queue.cc:237] T fbd94103cae04473ba81709da39d7ed8 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:12.909963 20232 raft_consensus.cc:1261] T fbd94103cae04473ba81709da39d7ed8 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:12.910807 24415 consensus_queue.cc:1035] T fbd94103cae04473ba81709da39d7ed8 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:12.912577 17737 raft_consensus.cc:1261] T fbd94103cae04473ba81709da39d7ed8 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:12.913511 24419 raft_consensus.cc:2932] T fbd94103cae04473ba81709da39d7ed8 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:12.915333 17328 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:54:12.919517 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fbd94103cae04473ba81709da39d7ed8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:12.920171 20231 raft_consensus.cc:2932] T fbd94103cae04473ba81709da39d7ed8 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:12.921059 24415 consensus_queue.cc:1035] T fbd94103cae04473ba81709da39d7ed8 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:12.923468 17476 raft_consensus.cc:3037] T 57e3c62666b1428ba01ef75abd15e3ef P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:12.923907 17867 raft_consensus.cc:1261] T 57e3c62666b1428ba01ef75abd15e3ef P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:12.924603 24127 consensus_queue.cc:1035] T 57e3c62666b1428ba01ef75abd15e3ef P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:12.923445 17334 catalog_manager.cc:5484] T fbd94103cae04473ba81709da39d7ed8 P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: config changed from index 3 to 4, VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) evicted. New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:12.926337 17476 raft_consensus.cc:1261] T 57e3c62666b1428ba01ef75abd15e3ef P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:12.928681 24127 consensus_queue.cc:1035] T 57e3c62666b1428ba01ef75abd15e3ef P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:12.933120 17737 raft_consensus.cc:2932] T fbd94103cae04473ba81709da39d7ed8 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:12.935186 17864 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4bb42883b1a0475b863d008bd01fa70a"
dest_uuid: "bb31bceba76d466db24b9341c04900c3"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59090
I20240502 01:54:12.935391 17864 raft_consensus.cc:595] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:54:12.937870 20478 tablet_service.cc:1514] Processing DeleteTablet for tablet fbd94103cae04473ba81709da39d7ed8 with delete_type TABLET_DATA_TOMBSTONED (TS 2f306549407b4bb281ccf2994035e4f1 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35018
I20240502 01:54:12.941773 24435 tablet_replica.cc:330] T fbd94103cae04473ba81709da39d7ed8 P 2f306549407b4bb281ccf2994035e4f1: stopping tablet replica
I20240502 01:54:12.942364 24435 raft_consensus.cc:2229] T fbd94103cae04473ba81709da39d7ed8 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:12.951393 24435 raft_consensus.cc:2258] T fbd94103cae04473ba81709da39d7ed8 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:12.960863 17334 catalog_manager.cc:5576] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083) does not have the latest schema for tablet 57e3c62666b1428ba01ef75abd15e3ef (table rebalancer_dynamic_table [id=c11e10ff0c07463f897b3b61d9e5d870]). Expected version 1 got 0
I20240502 01:54:12.967208 17334 catalog_manager.cc:5576] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321) does not have the latest schema for tablet 57e3c62666b1428ba01ef75abd15e3ef (table rebalancer_dynamic_table [id=c11e10ff0c07463f897b3b61d9e5d870]). Expected version 1 got 0
I20240502 01:54:12.968603 24435 ts_tablet_manager.cc:1875] T fbd94103cae04473ba81709da39d7ed8 P 2f306549407b4bb281ccf2994035e4f1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:12.975390 24442 tablet.cc:1722] T 57e3c62666b1428ba01ef75abd15e3ef P 4da690c0fa13407a85c0689ce8de9ade: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:12.975457 24441 tablet.cc:1722] T 57e3c62666b1428ba01ef75abd15e3ef P f19cc6cb1e76460aa6711e49a5775091: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:12.977694 23908 consensus_queue.cc:1035] T c491a39ca87847d9ace688346c0baf57 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:12.978058 24442 tablet.cc:1620] T 57e3c62666b1428ba01ef75abd15e3ef P 4da690c0fa13407a85c0689ce8de9ade: MemRowSet was empty: no flush needed.
I20240502 01:54:12.983263 17334 catalog_manager.cc:5576] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321) does not have the latest schema for tablet 57e3c62666b1428ba01ef75abd15e3ef (table rebalancer_dynamic_table [id=c11e10ff0c07463f897b3b61d9e5d870]). Expected version 1 got 0
I20240502 01:54:12.987382 24441 tablet.cc:1620] T 57e3c62666b1428ba01ef75abd15e3ef P f19cc6cb1e76460aa6711e49a5775091: MemRowSet was empty: no flush needed.
I20240502 01:54:12.987600 24203 raft_consensus.cc:1050] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685: attempting to promote NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 to VOTER
I20240502 01:54:12.988521 24203 consensus_queue.cc:237] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } }
I20240502 01:54:12.994951 17334 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=c11e10ff0c07463f897b3b61d9e5d870] alter complete (version 1)
I20240502 01:54:12.998767 17604 raft_consensus.cc:1203] T 2714919062f5465c8cde1a3c4720978a P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:13.007416 24435 ts_tablet_manager.cc:1888] T fbd94103cae04473ba81709da39d7ed8 P 2f306549407b4bb281ccf2994035e4f1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:54:13.010637 20232 raft_consensus.cc:1261] T 8e7f88b6c509477fb0900024e7de8f00 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:13.011584 20362 raft_consensus.cc:1261] T 8e7f88b6c509477fb0900024e7de8f00 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:13.011837 24435 log.cc:1196] T fbd94103cae04473ba81709da39d7ed8 P 2f306549407b4bb281ccf2994035e4f1: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal/wals/fbd94103cae04473ba81709da39d7ed8
I20240502 01:54:13.012396 24364 consensus_queue.cc:1035] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:13.013056 24444 tablet.cc:1722] T 57e3c62666b1428ba01ef75abd15e3ef P bb31bceba76d466db24b9341c04900c3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:13.012960 17322 catalog_manager.cc:4830] TS 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): tablet fbd94103cae04473ba81709da39d7ed8 (table rebalancer_extra_table_16 [id=61898cfacac64133a52ca6484799789a]) successfully deleted
I20240502 01:54:13.013576 24444 tablet.cc:1620] T 57e3c62666b1428ba01ef75abd15e3ef P bb31bceba76d466db24b9341c04900c3: MemRowSet was empty: no flush needed.
I20240502 01:54:13.014240 24364 consensus_queue.cc:1035] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:13.020438 17334 catalog_manager.cc:5576] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083) does not have the latest schema for tablet 57e3c62666b1428ba01ef75abd15e3ef (table rebalancer_dynamic_table [id=c11e10ff0c07463f897b3b61d9e5d870]). Expected version 1 got 0
I20240502 01:54:13.023947 24425 consensus_queue.cc:1035] T c491a39ca87847d9ace688346c0baf57 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:13.040360 17867 raft_consensus.cc:1261] T 8e7f88b6c509477fb0900024e7de8f00 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:13.040899 24364 consensus_queue.cc:1035] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:13.075238 24364 raft_consensus.cc:2932] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:54:13.081882 17867 raft_consensus.cc:2932] T 8e7f88b6c509477fb0900024e7de8f00 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:54:13.083189 20362 raft_consensus.cc:2932] T 8e7f88b6c509477fb0900024e7de8f00 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:54:13.087872 17333 catalog_manager.cc:5484] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 reported cstate change: config changed from index 7 to 8, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:13.090242 24419 raft_consensus.cc:1050] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091: attempting to promote NON_VOTER dfa1152987644bd781b46c7e42a7a685 to VOTER
I20240502 01:54:13.091070 24419 consensus_queue.cc:237] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:13.094902 17737 raft_consensus.cc:1261] T 1b14ac3413004127b72904b631ec1b94 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:13.095496 24415 consensus_queue.cc:1035] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:13.104827 24403 consensus_queue.cc:1035] T fa4d38dc9b674a9b90d4793dad2687ec P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:13.107860 17604 raft_consensus.cc:1203] T 68bc23294ac64c9486bffebbdb16b508 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:13.112493 17333 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:54:13.120005 17604 raft_consensus.cc:1261] T 1b14ac3413004127b72904b631ec1b94 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:13.120527 24415 consensus_queue.cc:1035] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:13.128430 24441 tablet.cc:1722] T 57e3c62666b1428ba01ef75abd15e3ef P f19cc6cb1e76460aa6711e49a5775091: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:13.128930 24441 tablet.cc:1620] T 57e3c62666b1428ba01ef75abd15e3ef P f19cc6cb1e76460aa6711e49a5775091: MemRowSet was empty: no flush needed.
I20240502 01:54:13.140224 24403 consensus_queue.cc:1035] T fa4d38dc9b674a9b90d4793dad2687ec P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:13.153861 24444 tablet.cc:1722] T 57e3c62666b1428ba01ef75abd15e3ef P bb31bceba76d466db24b9341c04900c3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:13.154341 24444 tablet.cc:1620] T 57e3c62666b1428ba01ef75abd15e3ef P bb31bceba76d466db24b9341c04900c3: MemRowSet was empty: no flush needed.
W20240502 01:54:13.166038 20168 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.42.68:44083, user_credentials={real_user=slave}} blocked reactor thread for 61753.7us
I20240502 01:54:13.166083 17328 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=c11e10ff0c07463f897b3b61d9e5d870] alter complete (version 2)
I20240502 01:54:13.167402 20232 raft_consensus.cc:1261] T 1b14ac3413004127b72904b631ec1b94 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:13.169620 24415 consensus_queue.cc:1035] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:13.174394 24167 raft_consensus.cc:979] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3: : Instructing follower 2f306549407b4bb281ccf2994035e4f1 to start an election
I20240502 01:54:13.174513 24167 raft_consensus.cc:1067] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Signalling peer 2f306549407b4bb281ccf2994035e4f1 to start an election
I20240502 01:54:13.176589 20498 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4bb42883b1a0475b863d008bd01fa70a"
dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
from {username='slave'} at 127.11.42.68:49559
I20240502 01:54:13.176915 20498 raft_consensus.cc:482] T 4bb42883b1a0475b863d008bd01fa70a P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:13.177065 20498 raft_consensus.cc:3037] T 4bb42883b1a0475b863d008bd01fa70a P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:13.178293 17737 consensus_queue.cc:237] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } }
I20240502 01:54:13.180408 20362 raft_consensus.cc:1261] T 8e7f88b6c509477fb0900024e7de8f00 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:13.181059 24203 consensus_queue.cc:1035] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:13.180186 20498 raft_consensus.cc:504] T 4bb42883b1a0475b863d008bd01fa70a P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } }
I20240502 01:54:13.184480 17605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bb42883b1a0475b863d008bd01fa70a" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:54:13.184523 17867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bb42883b1a0475b863d008bd01fa70a" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:54:13.184688 17867 raft_consensus.cc:3032] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:13.184756 17605 raft_consensus.cc:3037] T 4bb42883b1a0475b863d008bd01fa70a P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:13.184739 17867 raft_consensus.cc:729] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Running, Role: LEADER
I20240502 01:54:13.186607 17867 consensus_queue.cc:260] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } }
I20240502 01:54:13.187043 17867 raft_consensus.cc:3037] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:13.187809 17605 raft_consensus.cc:2454] T 4bb42883b1a0475b863d008bd01fa70a P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 2.
I20240502 01:54:13.190132 17867 raft_consensus.cc:2454] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 2.
I20240502 01:54:13.190538 20434 leader_election.cc:302] T 4bb42883b1a0475b863d008bd01fa70a P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 3fceccefba274b1dbbd8216af8b0bc29, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:54:13.190832 23694 raft_consensus.cc:2781] T 4bb42883b1a0475b863d008bd01fa70a P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:13.190985 23694 raft_consensus.cc:686] T 4bb42883b1a0475b863d008bd01fa70a P 2f306549407b4bb281ccf2994035e4f1 [term 2 LEADER]: Becoming Leader. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Running, Role: LEADER
I20240502 01:54:13.191259 23694 consensus_queue.cc:237] T 4bb42883b1a0475b863d008bd01fa70a P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } }
I20240502 01:54:13.191393 23967 raft_consensus.cc:2932] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:54:13.192328 20362 raft_consensus.cc:2932] T 8e7f88b6c509477fb0900024e7de8f00 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:54:13.198733 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8e7f88b6c509477fb0900024e7de8f00 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:13.200664 17333 catalog_manager.cc:5484] T 4bb42883b1a0475b863d008bd01fa70a P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: term changed from 1 to 2, leader changed from bb31bceba76d466db24b9341c04900c3 (127.11.42.68) to 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71). New cstate: current_term: 2 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:13.202780 17326 catalog_manager.cc:5484] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 reported cstate change: config changed from index 8 to 9, VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) evicted. New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:13.205462 20232 raft_consensus.cc:2932] T 8e7f88b6c509477fb0900024e7de8f00 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:54:13.211670 17847 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e7f88b6c509477fb0900024e7de8f00 with delete_type TABLET_DATA_TOMBSTONED (TS bb31bceba76d466db24b9341c04900c3 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:47226
I20240502 01:54:13.223449 20498 leader_election.cc:288] T 4bb42883b1a0475b863d008bd01fa70a P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 election: Requested vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:54:13.223920 20628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bb42883b1a0475b863d008bd01fa70a" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "f19cc6cb1e76460aa6711e49a5775091"
I20240502 01:54:13.224181 20628 raft_consensus.cc:3037] T 4bb42883b1a0475b863d008bd01fa70a P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:13.225682 20232 raft_consensus.cc:1261] T 8e7f88b6c509477fb0900024e7de8f00 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:13.227152 20628 raft_consensus.cc:2454] T 4bb42883b1a0475b863d008bd01fa70a P f19cc6cb1e76460aa6711e49a5775091 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 2.
I20240502 01:54:13.228361 23967 consensus_queue.cc:1035] T 8e7f88b6c509477fb0900024e7de8f00 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:13.233167 20232 raft_consensus.cc:2932] T 8e7f88b6c509477fb0900024e7de8f00 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:54:13.250381 24464 tablet_replica.cc:330] T 8e7f88b6c509477fb0900024e7de8f00 P bb31bceba76d466db24b9341c04900c3: stopping tablet replica
I20240502 01:54:13.251284 24415 raft_consensus.cc:2932] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:13.257475 20232 raft_consensus.cc:2932] T 1b14ac3413004127b72904b631ec1b94 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:13.259371 17605 raft_consensus.cc:2932] T 1b14ac3413004127b72904b631ec1b94 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:13.259963 17333 catalog_manager.cc:5484] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:13.272235 24464 raft_consensus.cc:2229] T 8e7f88b6c509477fb0900024e7de8f00 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:13.273943 17328 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:54:13.276060 24464 raft_consensus.cc:2258] T 8e7f88b6c509477fb0900024e7de8f00 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:13.281721 24464 ts_tablet_manager.cc:1875] T 8e7f88b6c509477fb0900024e7de8f00 P bb31bceba76d466db24b9341c04900c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:13.284018 24441 tablet.cc:1722] T 57e3c62666b1428ba01ef75abd15e3ef P f19cc6cb1e76460aa6711e49a5775091: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:13.285255 24444 tablet.cc:1722] T 57e3c62666b1428ba01ef75abd15e3ef P bb31bceba76d466db24b9341c04900c3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:13.293779 17328 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table_new_name [id=c11e10ff0c07463f897b3b61d9e5d870] alter complete (version 3)
I20240502 01:54:13.295607 24415 raft_consensus.cc:1050] T 75bb4e19cf57405fabcb532b55a29ced P f19cc6cb1e76460aa6711e49a5775091: attempting to promote NON_VOTER 4da690c0fa13407a85c0689ce8de9ade to VOTER
I20240502 01:54:13.296178 24442 tablet.cc:1722] T 57e3c62666b1428ba01ef75abd15e3ef P 4da690c0fa13407a85c0689ce8de9ade: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:13.296360 24415 consensus_queue.cc:237] T 75bb4e19cf57405fabcb532b55a29ced P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } }
I20240502 01:54:13.296689 24442 tablet.cc:1620] T 57e3c62666b1428ba01ef75abd15e3ef P 4da690c0fa13407a85c0689ce8de9ade: MemRowSet was empty: no flush needed.
I20240502 01:54:13.298213 17737 raft_consensus.cc:2932] T 1b14ac3413004127b72904b631ec1b94 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:13.305409 24442 tablet.cc:1722] T 57e3c62666b1428ba01ef75abd15e3ef P 4da690c0fa13407a85c0689ce8de9ade: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:13.306495 24464 ts_tablet_manager.cc:1888] T 8e7f88b6c509477fb0900024e7de8f00 P bb31bceba76d466db24b9341c04900c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:54:13.306612 24464 log.cc:1196] T 8e7f88b6c509477fb0900024e7de8f00 P bb31bceba76d466db24b9341c04900c3: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/wals/8e7f88b6c509477fb0900024e7de8f00
I20240502 01:54:13.307740 17322 catalog_manager.cc:4830] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): tablet 8e7f88b6c509477fb0900024e7de8f00 (table rebalance_test_table_1 [id=e431affbb02a4744b67a522521a3ee04]) successfully deleted
I20240502 01:54:13.318439 17476 raft_consensus.cc:1261] T 75bb4e19cf57405fabcb532b55a29ced P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEARNER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:13.319069 24125 consensus_queue.cc:1035] T 75bb4e19cf57405fabcb532b55a29ced P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:13.320909 17333 catalog_manager.cc:5576] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321) does not have the latest schema for tablet 57e3c62666b1428ba01ef75abd15e3ef (table rebalancer_dynamic_table_new_name [id=c11e10ff0c07463f897b3b61d9e5d870]). Expected version 3 got 2
I20240502 01:54:13.323711 17867 raft_consensus.cc:1261] T 75bb4e19cf57405fabcb532b55a29ced P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:13.324679 24125 consensus_queue.cc:1035] T 75bb4e19cf57405fabcb532b55a29ced P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:13.325232 20366 raft_consensus.cc:1261] T 75bb4e19cf57405fabcb532b55a29ced P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:13.326082 24125 consensus_queue.cc:1035] T 75bb4e19cf57405fabcb532b55a29ced P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:13.331049 24127 raft_consensus.cc:2932] T 75bb4e19cf57405fabcb532b55a29ced P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:13.335341 20366 raft_consensus.cc:2932] T 75bb4e19cf57405fabcb532b55a29ced P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:13.337852 17333 catalog_manager.cc:5484] T 75bb4e19cf57405fabcb532b55a29ced P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:13.348372 17867 raft_consensus.cc:2932] T 75bb4e19cf57405fabcb532b55a29ced P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:13.363155 17476 raft_consensus.cc:2932] T 75bb4e19cf57405fabcb532b55a29ced P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:13.367410 20628 consensus_queue.cc:237] T 75bb4e19cf57405fabcb532b55a29ced P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } }
I20240502 01:54:13.369688 17867 raft_consensus.cc:1261] T 75bb4e19cf57405fabcb532b55a29ced P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:13.372313 24359 consensus_queue.cc:1035] T 75bb4e19cf57405fabcb532b55a29ced P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:13.373936 17476 raft_consensus.cc:1261] T 75bb4e19cf57405fabcb532b55a29ced P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:13.374914 24415 consensus_queue.cc:1035] T 75bb4e19cf57405fabcb532b55a29ced P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:13.374938 20628 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cf0237a4c65b495ba892871dd850bed9"
dest_uuid: "f19cc6cb1e76460aa6711e49a5775091"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45728
I20240502 01:54:13.375509 20628 raft_consensus.cc:595] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:54:13.375665 24359 raft_consensus.cc:2932] T 75bb4e19cf57405fabcb532b55a29ced P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:13.376626 17867 raft_consensus.cc:2932] T 75bb4e19cf57405fabcb532b55a29ced P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:13.377707 17476 raft_consensus.cc:2932] T 75bb4e19cf57405fabcb532b55a29ced P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:13.378767 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 75bb4e19cf57405fabcb532b55a29ced with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:13.384176 17330 catalog_manager.cc:5484] T 75bb4e19cf57405fabcb532b55a29ced P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: config changed from index 3 to 4, VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) evicted. New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:13.389223 20345 tablet_service.cc:1514] Processing DeleteTablet for tablet 75bb4e19cf57405fabcb532b55a29ced with delete_type TABLET_DATA_TOMBSTONED (TS 6a36c952c51542c0bf7ea3ec2857a9f9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40196
I20240502 01:54:13.389914 24477 tablet_replica.cc:330] T 75bb4e19cf57405fabcb532b55a29ced P 6a36c952c51542c0bf7ea3ec2857a9f9: stopping tablet replica
I20240502 01:54:13.390123 24127 raft_consensus.cc:979] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091: : Instructing follower dfa1152987644bd781b46c7e42a7a685 to start an election
I20240502 01:54:13.390240 24477 raft_consensus.cc:2229] T 75bb4e19cf57405fabcb532b55a29ced P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:13.390262 24127 raft_consensus.cc:1067] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Signalling peer dfa1152987644bd781b46c7e42a7a685 to start an election
I20240502 01:54:13.390625 24477 raft_consensus.cc:2258] T 75bb4e19cf57405fabcb532b55a29ced P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:13.391170 17737 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cf0237a4c65b495ba892871dd850bed9"
dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
from {username='slave'} at 127.11.42.72:53665
I20240502 01:54:13.391347 17737 raft_consensus.cc:482] T cf0237a4c65b495ba892871dd850bed9 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:13.391427 17737 raft_consensus.cc:3037] T cf0237a4c65b495ba892871dd850bed9 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:13.397004 17333 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:13.397230 17333 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:13.397090 17737 raft_consensus.cc:504] T cf0237a4c65b495ba892871dd850bed9 P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } }
I20240502 01:54:13.398609 20628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf0237a4c65b495ba892871dd850bed9" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "f19cc6cb1e76460aa6711e49a5775091"
I20240502 01:54:13.398715 17737 leader_election.cc:288] T cf0237a4c65b495ba892871dd850bed9 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 2 election: Requested vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:54:13.398892 20628 raft_consensus.cc:3032] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:13.398991 20628 raft_consensus.cc:729] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Running, Role: LEADER
I20240502 01:54:13.399358 20628 consensus_queue.cc:260] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } }
I20240502 01:54:13.399623 20498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf0237a4c65b495ba892871dd850bed9" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
I20240502 01:54:13.399820 20628 raft_consensus.cc:3037] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:13.399840 20498 raft_consensus.cc:3037] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:13.399932 24477 ts_tablet_manager.cc:1875] T 75bb4e19cf57405fabcb532b55a29ced P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:13.402544 20628 raft_consensus.cc:2454] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 2.
I20240502 01:54:13.402544 20498 raft_consensus.cc:2454] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 2.
I20240502 01:54:13.402930 17867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf0237a4c65b495ba892871dd850bed9" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:54:13.403118 17867 raft_consensus.cc:3037] T cf0237a4c65b495ba892871dd850bed9 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:13.403116 17673 leader_election.cc:302] T cf0237a4c65b495ba892871dd850bed9 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, dfa1152987644bd781b46c7e42a7a685, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:54:13.403381 23967 raft_consensus.cc:2781] T cf0237a4c65b495ba892871dd850bed9 P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:13.403515 23967 raft_consensus.cc:686] T cf0237a4c65b495ba892871dd850bed9 P dfa1152987644bd781b46c7e42a7a685 [term 2 LEADER]: Becoming Leader. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Running, Role: LEADER
I20240502 01:54:13.403753 23967 consensus_queue.cc:237] T cf0237a4c65b495ba892871dd850bed9 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } }
I20240502 01:54:13.405169 17867 raft_consensus.cc:2454] T cf0237a4c65b495ba892871dd850bed9 P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 2.
I20240502 01:54:13.405227 24477 ts_tablet_manager.cc:1888] T 75bb4e19cf57405fabcb532b55a29ced P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:54:13.405413 24477 log.cc:1196] T 75bb4e19cf57405fabcb532b55a29ced P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal/wals/75bb4e19cf57405fabcb532b55a29ced
I20240502 01:54:13.406306 17321 catalog_manager.cc:4830] TS 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): tablet 75bb4e19cf57405fabcb532b55a29ced (table rebalancer_extra_table_20 [id=0e1c9ce53d7c47f7a42d19e60342c557]) successfully deleted
I20240502 01:54:13.408361 17333 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet 57e3c62666b1428ba01ef75abd15e3ef
I20240502 01:54:13.408681 17326 catalog_manager.cc:5484] T cf0237a4c65b495ba892871dd850bed9 P dfa1152987644bd781b46c7e42a7a685 reported cstate change: term changed from 1 to 2, leader changed from f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) to dfa1152987644bd781b46c7e42a7a685 (127.11.42.67). New cstate: current_term: 2 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:13.409838 17847 tablet_service.cc:1514] Processing DeleteTablet for tablet 57e3c62666b1428ba01ef75abd15e3ef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:13 UTC) from {username='slave'} at 127.0.0.1:47226
I20240502 01:54:13.410084 24464 tablet_replica.cc:330] T 57e3c62666b1428ba01ef75abd15e3ef P bb31bceba76d466db24b9341c04900c3: stopping tablet replica
I20240502 01:54:13.410264 24464 raft_consensus.cc:2229] T 57e3c62666b1428ba01ef75abd15e3ef P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:13.410490 24464 raft_consensus.cc:2258] T 57e3c62666b1428ba01ef75abd15e3ef P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:13.410753 20609 tablet_service.cc:1514] Processing DeleteTablet for tablet 57e3c62666b1428ba01ef75abd15e3ef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:13 UTC) from {username='slave'} at 127.0.0.1:33698
I20240502 01:54:13.411486 24464 ts_tablet_manager.cc:1875] T 57e3c62666b1428ba01ef75abd15e3ef P bb31bceba76d466db24b9341c04900c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:13.413205 17456 tablet_service.cc:1514] Processing DeleteTablet for tablet 57e3c62666b1428ba01ef75abd15e3ef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:13 UTC) from {username='slave'} at 127.0.0.1:54520
I20240502 01:54:13.413811 24486 tablet_replica.cc:330] T 57e3c62666b1428ba01ef75abd15e3ef P 4da690c0fa13407a85c0689ce8de9ade: stopping tablet replica
I20240502 01:54:13.414029 24486 raft_consensus.cc:2229] T 57e3c62666b1428ba01ef75abd15e3ef P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:13.414193 24485 tablet_replica.cc:330] T 57e3c62666b1428ba01ef75abd15e3ef P f19cc6cb1e76460aa6711e49a5775091: stopping tablet replica
I20240502 01:54:13.414263 24486 raft_consensus.cc:2258] T 57e3c62666b1428ba01ef75abd15e3ef P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:13.414407 24485 raft_consensus.cc:2229] T 57e3c62666b1428ba01ef75abd15e3ef P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:13.414757 24485 raft_consensus.cc:2258] T 57e3c62666b1428ba01ef75abd15e3ef P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:13.415477 24486 ts_tablet_manager.cc:1875] T 57e3c62666b1428ba01ef75abd15e3ef P 4da690c0fa13407a85c0689ce8de9ade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:13.416002 24485 ts_tablet_manager.cc:1875] T 57e3c62666b1428ba01ef75abd15e3ef P f19cc6cb1e76460aa6711e49a5775091: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:13.418037 24464 ts_tablet_manager.cc:1888] T 57e3c62666b1428ba01ef75abd15e3ef P bb31bceba76d466db24b9341c04900c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:13.418198 24464 log.cc:1196] T 57e3c62666b1428ba01ef75abd15e3ef P bb31bceba76d466db24b9341c04900c3: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/wals/57e3c62666b1428ba01ef75abd15e3ef
I20240502 01:54:13.418610 24464 ts_tablet_manager.cc:1909] T 57e3c62666b1428ba01ef75abd15e3ef P bb31bceba76d466db24b9341c04900c3: Deleting consensus metadata
I20240502 01:54:13.419564 17322 catalog_manager.cc:4830] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): tablet 57e3c62666b1428ba01ef75abd15e3ef (table rebalancer_dynamic_table_new_name [id=c11e10ff0c07463f897b3b61d9e5d870]) successfully deleted
I20240502 01:54:13.421965 24486 ts_tablet_manager.cc:1888] T 57e3c62666b1428ba01ef75abd15e3ef P 4da690c0fa13407a85c0689ce8de9ade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:13.422151 24486 log.cc:1196] T 57e3c62666b1428ba01ef75abd15e3ef P 4da690c0fa13407a85c0689ce8de9ade: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/57e3c62666b1428ba01ef75abd15e3ef
I20240502 01:54:13.422725 24486 ts_tablet_manager.cc:1909] T 57e3c62666b1428ba01ef75abd15e3ef P 4da690c0fa13407a85c0689ce8de9ade: Deleting consensus metadata
I20240502 01:54:13.423767 17321 catalog_manager.cc:4830] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): tablet 57e3c62666b1428ba01ef75abd15e3ef (table rebalancer_dynamic_table_new_name [id=c11e10ff0c07463f897b3b61d9e5d870]) successfully deleted
I20240502 01:54:13.425415 24485 ts_tablet_manager.cc:1888] T 57e3c62666b1428ba01ef75abd15e3ef P f19cc6cb1e76460aa6711e49a5775091: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:13.425535 24485 log.cc:1196] T 57e3c62666b1428ba01ef75abd15e3ef P f19cc6cb1e76460aa6711e49a5775091: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal/wals/57e3c62666b1428ba01ef75abd15e3ef
I20240502 01:54:13.425974 24485 ts_tablet_manager.cc:1909] T 57e3c62666b1428ba01ef75abd15e3ef P f19cc6cb1e76460aa6711e49a5775091: Deleting consensus metadata
I20240502 01:54:13.426853 17321 catalog_manager.cc:4830] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): tablet 57e3c62666b1428ba01ef75abd15e3ef (table rebalancer_dynamic_table_new_name [id=c11e10ff0c07463f897b3b61d9e5d870]) successfully deleted
I20240502 01:54:13.447594 17737 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5df83f83d2674155afba93c2d992af0a"
dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40406
I20240502 01:54:13.447806 17737 raft_consensus.cc:595] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:54:13.460907 17326 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_extra_table_48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:13.469710 20345 tablet_service.cc:1467] Processing CreateTablet for tablet acbd240a2783424e96bd589f0232daf6 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=516d492205794b72a52193ce4c29c429]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:13.470206 20345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acbd240a2783424e96bd589f0232daf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:13.473877 17716 tablet_service.cc:1467] Processing CreateTablet for tablet acbd240a2783424e96bd589f0232daf6 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=516d492205794b72a52193ce4c29c429]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:13.474332 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acbd240a2783424e96bd589f0232daf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:13.474320 17454 tablet_service.cc:1467] Processing CreateTablet for tablet 277c6ae5c8dc4f8eae261bf06890d793 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=516d492205794b72a52193ce4c29c429]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:13.474732 17454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 277c6ae5c8dc4f8eae261bf06890d793. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:13.477428 17456 tablet_service.cc:1467] Processing CreateTablet for tablet 5e3695e8ad0a44de9ba23ae3610aacec (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=516d492205794b72a52193ce4c29c429]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:13.477834 17456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e3695e8ad0a44de9ba23ae3610aacec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:13.479545 20477 tablet_service.cc:1467] Processing CreateTablet for tablet 5e3695e8ad0a44de9ba23ae3610aacec (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=516d492205794b72a52193ce4c29c429]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:13.479545 20478 tablet_service.cc:1467] Processing CreateTablet for tablet acbd240a2783424e96bd589f0232daf6 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=516d492205794b72a52193ce4c29c429]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:13.479977 20478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acbd240a2783424e96bd589f0232daf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:13.480222 17847 tablet_service.cc:1467] Processing CreateTablet for tablet 277c6ae5c8dc4f8eae261bf06890d793 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=516d492205794b72a52193ce4c29c429]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:13.480590 17847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 277c6ae5c8dc4f8eae261bf06890d793. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:13.484781 20477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e3695e8ad0a44de9ba23ae3610aacec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:13.484997 20211 tablet_service.cc:1467] Processing CreateTablet for tablet 5e3695e8ad0a44de9ba23ae3610aacec (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=516d492205794b72a52193ce4c29c429]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:13.485035 20212 tablet_service.cc:1467] Processing CreateTablet for tablet 277c6ae5c8dc4f8eae261bf06890d793 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=516d492205794b72a52193ce4c29c429]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:13.485678 20212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 277c6ae5c8dc4f8eae261bf06890d793. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:13.489035 20211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e3695e8ad0a44de9ba23ae3610aacec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:13.507099 23611 raft_consensus.cc:1050] T 2714919062f5465c8cde1a3c4720978a P 6a36c952c51542c0bf7ea3ec2857a9f9: attempting to promote NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 to VOTER
I20240502 01:54:13.508059 23611 consensus_queue.cc:237] T 2714919062f5465c8cde1a3c4720978a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } }
I20240502 01:54:13.512599 24514 tablet_bootstrap.cc:492] T acbd240a2783424e96bd589f0232daf6 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:54:13.515173 24514 tablet_bootstrap.cc:654] T acbd240a2783424e96bd589f0232daf6 P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:13.519186 23967 raft_consensus.cc:979] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685: : Instructing follower 4da690c0fa13407a85c0689ce8de9ade to start an election
I20240502 01:54:13.519326 23967 raft_consensus.cc:1067] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Signalling peer 4da690c0fa13407a85c0689ce8de9ade to start an election
I20240502 01:54:13.520116 17475 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5df83f83d2674155afba93c2d992af0a"
dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
from {username='slave'} at 127.11.42.67:54725
I20240502 01:54:13.520323 17475 raft_consensus.cc:482] T 5df83f83d2674155afba93c2d992af0a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:13.520465 17475 raft_consensus.cc:3037] T 5df83f83d2674155afba93c2d992af0a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:13.523190 17475 raft_consensus.cc:504] T 5df83f83d2674155afba93c2d992af0a P 4da690c0fa13407a85c0689ce8de9ade [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } }
I20240502 01:54:13.523583 24514 tablet_bootstrap.cc:492] T acbd240a2783424e96bd589f0232daf6 P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:54:13.524251 24514 ts_tablet_manager.cc:1367] T acbd240a2783424e96bd589f0232daf6 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.001s
I20240502 01:54:13.524725 17475 leader_election.cc:288] T 5df83f83d2674155afba93c2d992af0a P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 election: Requested vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:54:13.524804 20231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df83f83d2674155afba93c2d992af0a" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:54:13.524822 20628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df83f83d2674155afba93c2d992af0a" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "f19cc6cb1e76460aa6711e49a5775091"
I20240502 01:54:13.525084 20628 raft_consensus.cc:3037] T 5df83f83d2674155afba93c2d992af0a P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:13.525027 20231 raft_consensus.cc:3037] T 5df83f83d2674155afba93c2d992af0a P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:13.525388 17604 raft_consensus.cc:1261] T 2714919062f5465c8cde1a3c4720978a P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEARNER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:13.526564 24513 tablet_bootstrap.cc:492] T 5e3695e8ad0a44de9ba23ae3610aacec P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:54:13.526618 23908 consensus_queue.cc:1035] T 2714919062f5465c8cde1a3c4720978a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:13.527726 20628 raft_consensus.cc:2454] T 5df83f83d2674155afba93c2d992af0a P f19cc6cb1e76460aa6711e49a5775091 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 2.
I20240502 01:54:13.527997 20231 raft_consensus.cc:2454] T 5df83f83d2674155afba93c2d992af0a P fdc2ac372492454c85387589a91f63af [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 2.
I20240502 01:54:13.528478 17412 leader_election.cc:302] T 5df83f83d2674155afba93c2d992af0a P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, f19cc6cb1e76460aa6711e49a5775091, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:54:13.529152 24212 raft_consensus.cc:2781] T 5df83f83d2674155afba93c2d992af0a P 4da690c0fa13407a85c0689ce8de9ade [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:13.526131 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df83f83d2674155afba93c2d992af0a" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:54:13.529325 24212 raft_consensus.cc:686] T 5df83f83d2674155afba93c2d992af0a P 4da690c0fa13407a85c0689ce8de9ade [term 2 LEADER]: Becoming Leader. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Running, Role: LEADER
I20240502 01:54:13.529448 17737 raft_consensus.cc:3032] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:13.529556 17737 raft_consensus.cc:729] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Running, Role: LEADER
I20240502 01:54:13.529619 24212 consensus_queue.cc:237] T 5df83f83d2674155afba93c2d992af0a P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } }
I20240502 01:54:13.529843 17737 consensus_queue.cc:260] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } }
I20240502 01:54:13.530335 17737 raft_consensus.cc:3037] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:13.530277 24514 raft_consensus.cc:348] T acbd240a2783424e96bd589f0232daf6 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:13.530548 24514 raft_consensus.cc:374] T acbd240a2783424e96bd589f0232daf6 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:13.530656 24514 raft_consensus.cc:729] T acbd240a2783424e96bd589f0232daf6 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:54:13.530927 24514 consensus_queue.cc:260] T acbd240a2783424e96bd589f0232daf6 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:13.531550 24514 ts_tablet_manager.cc:1398] T acbd240a2783424e96bd589f0232daf6 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:54:13.533200 17737 raft_consensus.cc:2454] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 2.
I20240502 01:54:13.534569 20231 raft_consensus.cc:1261] T 2714919062f5465c8cde1a3c4720978a P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:13.535133 23908 consensus_queue.cc:1035] T 2714919062f5465c8cde1a3c4720978a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:13.536229 17326 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:54:13.536798 24127 raft_consensus.cc:1050] T 68bc23294ac64c9486bffebbdb16b508 P f19cc6cb1e76460aa6711e49a5775091: attempting to promote NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 to VOTER
I20240502 01:54:13.537569 24127 consensus_queue.cc:237] T 68bc23294ac64c9486bffebbdb16b508 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } }
I20240502 01:54:13.544277 17328 catalog_manager.cc:5484] T 5df83f83d2674155afba93c2d992af0a P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: term changed from 1 to 2, leader changed from dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) to 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65). New cstate: current_term: 2 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:13.544904 24513 tablet_bootstrap.cc:654] T 5e3695e8ad0a44de9ba23ae3610aacec P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:13.547206 20609 tablet_service.cc:1467] Processing CreateTablet for tablet dc5b634d198f4408b30d859addd79de3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6dc57802503d4ec5b7d530db90daec99]), partition=
I20240502 01:54:13.547644 20609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc5b634d198f4408b30d859addd79de3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:13.548847 17716 tablet_service.cc:1467] Processing CreateTablet for tablet dc5b634d198f4408b30d859addd79de3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6dc57802503d4ec5b7d530db90daec99]), partition=
I20240502 01:54:13.549163 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc5b634d198f4408b30d859addd79de3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:13.555550 24514 tablet_bootstrap.cc:492] T dc5b634d198f4408b30d859addd79de3 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:54:13.557060 24514 tablet_bootstrap.cc:654] T dc5b634d198f4408b30d859addd79de3 P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:13.559849 20498 raft_consensus.cc:1261] T 2714919062f5465c8cde1a3c4720978a P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:13.560912 24512 tablet_bootstrap.cc:492] T acbd240a2783424e96bd589f0232daf6 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:54:13.562534 17604 raft_consensus.cc:1261] T 68bc23294ac64c9486bffebbdb16b508 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEARNER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:13.563521 24512 tablet_bootstrap.cc:654] T acbd240a2783424e96bd589f0232daf6 P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:13.563588 20232 raft_consensus.cc:1261] T 68bc23294ac64c9486bffebbdb16b508 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:13.563728 20498 raft_consensus.cc:1261] T 68bc23294ac64c9486bffebbdb16b508 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:13.564229 24359 consensus_queue.cc:1035] T 68bc23294ac64c9486bffebbdb16b508 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:13.564558 24125 consensus_queue.cc:1035] T 68bc23294ac64c9486bffebbdb16b508 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:13.564851 24127 consensus_queue.cc:1035] T 68bc23294ac64c9486bffebbdb16b508 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:13.567521 24514 tablet_bootstrap.cc:492] T dc5b634d198f4408b30d859addd79de3 P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:54:13.567698 24514 ts_tablet_manager.cc:1367] T dc5b634d198f4408b30d859addd79de3 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.008s
I20240502 01:54:13.568394 24514 raft_consensus.cc:348] T dc5b634d198f4408b30d859addd79de3 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:13.568617 24514 raft_consensus.cc:374] T dc5b634d198f4408b30d859addd79de3 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:13.568572 23908 consensus_queue.cc:1035] T 2714919062f5465c8cde1a3c4720978a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:13.568722 24514 raft_consensus.cc:729] T dc5b634d198f4408b30d859addd79de3 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:54:13.568833 17867 raft_consensus.cc:1261] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:54:13.569087 24514 consensus_queue.cc:260] T dc5b634d198f4408b30d859addd79de3 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:13.569790 24514 ts_tablet_manager.cc:1398] T dc5b634d198f4408b30d859addd79de3 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:13.569772 23694 consensus_queue.cc:1035] T 4bb42883b1a0475b863d008bd01fa70a P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240502 01:54:13.575309 24359 raft_consensus.cc:2932] T 68bc23294ac64c9486bffebbdb16b508 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:13.576582 17604 raft_consensus.cc:2932] T 68bc23294ac64c9486bffebbdb16b508 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:13.580777 20232 raft_consensus.cc:2932] T 68bc23294ac64c9486bffebbdb16b508 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:13.581856 17328 catalog_manager.cc:5484] T 68bc23294ac64c9486bffebbdb16b508 P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:13.592130 20211 tablet_service.cc:1467] Processing CreateTablet for tablet dc5b634d198f4408b30d859addd79de3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6dc57802503d4ec5b7d530db90daec99]), partition=
I20240502 01:54:13.592482 20211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc5b634d198f4408b30d859addd79de3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:13.612782 24535 tablet_bootstrap.cc:492] T dc5b634d198f4408b30d859addd79de3 P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:54:13.627684 24535 tablet_bootstrap.cc:654] T dc5b634d198f4408b30d859addd79de3 P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:13.639958 24535 tablet_bootstrap.cc:492] T dc5b634d198f4408b30d859addd79de3 P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:54:13.640074 24203 raft_consensus.cc:482] T acbd240a2783424e96bd589f0232daf6 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:13.640214 24203 raft_consensus.cc:504] T acbd240a2783424e96bd589f0232daf6 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:13.641355 24535 ts_tablet_manager.cc:1367] T dc5b634d198f4408b30d859addd79de3 P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.006s
I20240502 01:54:13.644872 17604 raft_consensus.cc:1261] T 4bb42883b1a0475b863d008bd01fa70a P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:54:13.645640 23694 consensus_queue.cc:1035] T 4bb42883b1a0475b863d008bd01fa70a P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:13.645792 24535 raft_consensus.cc:348] T dc5b634d198f4408b30d859addd79de3 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:13.648262 24535 raft_consensus.cc:374] T dc5b634d198f4408b30d859addd79de3 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:13.648317 24203 leader_election.cc:288] T acbd240a2783424e96bd589f0232daf6 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:54:13.648600 24535 raft_consensus.cc:729] T dc5b634d198f4408b30d859addd79de3 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:54:13.649135 20362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acbd240a2783424e96bd589f0232daf6" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:54:13.649416 24535 consensus_queue.cc:260] T dc5b634d198f4408b30d859addd79de3 P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
W20240502 01:54:13.649711 17672 leader_election.cc:341] T acbd240a2783424e96bd589f0232daf6 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:13.650060 20496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acbd240a2783424e96bd589f0232daf6" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
W20240502 01:54:13.651904 17673 leader_election.cc:341] T acbd240a2783424e96bd589f0232daf6 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:13.652007 17673 leader_election.cc:302] T acbd240a2783424e96bd589f0232daf6 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dfa1152987644bd781b46c7e42a7a685; no voters: 2f306549407b4bb281ccf2994035e4f1, 6a36c952c51542c0bf7ea3ec2857a9f9
I20240502 01:54:13.652417 24203 raft_consensus.cc:2726] T acbd240a2783424e96bd589f0232daf6 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:54:13.665455 24517 tablet_bootstrap.cc:492] T acbd240a2783424e96bd589f0232daf6 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:54:13.667469 24517 tablet_bootstrap.cc:654] T acbd240a2783424e96bd589f0232daf6 P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:13.669899 24533 tablet_bootstrap.cc:492] T 277c6ae5c8dc4f8eae261bf06890d793 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:54:13.671662 24533 tablet_bootstrap.cc:654] T 277c6ae5c8dc4f8eae261bf06890d793 P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:13.675107 24535 ts_tablet_manager.cc:1398] T dc5b634d198f4408b30d859addd79de3 P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.032s user 0.000s sys 0.002s
I20240502 01:54:13.678277 20628 raft_consensus.cc:1261] T 4bb42883b1a0475b863d008bd01fa70a P f19cc6cb1e76460aa6711e49a5775091 [term 2 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:54:13.681380 23611 raft_consensus.cc:2932] T 2714919062f5465c8cde1a3c4720978a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:13.683283 24512 tablet_bootstrap.cc:492] T acbd240a2783424e96bd589f0232daf6 P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:54:13.683434 24512 ts_tablet_manager.cc:1367] T acbd240a2783424e96bd589f0232daf6 P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.123s user 0.011s sys 0.000s
I20240502 01:54:13.684216 24512 raft_consensus.cc:348] T acbd240a2783424e96bd589f0232daf6 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:13.684414 24512 raft_consensus.cc:374] T acbd240a2783424e96bd589f0232daf6 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:13.684505 24512 raft_consensus.cc:729] T acbd240a2783424e96bd589f0232daf6 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:54:13.684975 24512 consensus_queue.cc:260] T acbd240a2783424e96bd589f0232daf6 P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:13.685595 20494 raft_consensus.cc:2932] T 68bc23294ac64c9486bffebbdb16b508 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:13.688181 17604 raft_consensus.cc:2932] T 2714919062f5465c8cde1a3c4720978a P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:13.690109 24515 tablet_bootstrap.cc:492] T 5e3695e8ad0a44de9ba23ae3610aacec P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:54:13.692179 24515 tablet_bootstrap.cc:654] T 5e3695e8ad0a44de9ba23ae3610aacec P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:13.694226 20628 consensus_queue.cc:237] T 68bc23294ac64c9486bffebbdb16b508 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } }
I20240502 01:54:13.695155 17333 catalog_manager.cc:5484] T 2714919062f5465c8cde1a3c4720978a P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:13.696627 17605 raft_consensus.cc:1261] T 68bc23294ac64c9486bffebbdb16b508 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:13.697203 24415 consensus_queue.cc:1035] T 68bc23294ac64c9486bffebbdb16b508 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:13.700956 24415 raft_consensus.cc:2932] T 68bc23294ac64c9486bffebbdb16b508 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:13.701990 17605 raft_consensus.cc:2932] T 68bc23294ac64c9486bffebbdb16b508 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:13.711867 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 68bc23294ac64c9486bffebbdb16b508 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:13.712966 20231 raft_consensus.cc:2932] T 2714919062f5465c8cde1a3c4720978a P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:13.714164 17333 catalog_manager.cc:5484] T 68bc23294ac64c9486bffebbdb16b508 P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: config changed from index 3 to 4, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:13.716348 24545 consensus_queue.cc:1035] T 4bb42883b1a0475b863d008bd01fa70a P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:13.715260 20498 raft_consensus.cc:2932] T 2714919062f5465c8cde1a3c4720978a P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:13.718091 20211 tablet_service.cc:1514] Processing DeleteTablet for tablet 68bc23294ac64c9486bffebbdb16b508 with delete_type TABLET_DATA_TOMBSTONED (TS fdc2ac372492454c85387589a91f63af not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44624
I20240502 01:54:13.719872 20362 consensus_queue.cc:237] T 2714919062f5465c8cde1a3c4720978a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } }
I20240502 01:54:13.721973 17604 raft_consensus.cc:1261] T 2714919062f5465c8cde1a3c4720978a P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:13.722512 23908 consensus_queue.cc:1035] T 2714919062f5465c8cde1a3c4720978a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:13.725399 23908 raft_consensus.cc:2932] T 2714919062f5465c8cde1a3c4720978a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:13.729722 24512 ts_tablet_manager.cc:1398] T acbd240a2783424e96bd589f0232daf6 P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.046s user 0.002s sys 0.000s
I20240502 01:54:13.730594 17606 raft_consensus.cc:2932] T 2714919062f5465c8cde1a3c4720978a P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:13.731535 24512 tablet_bootstrap.cc:492] T 5e3695e8ad0a44de9ba23ae3610aacec P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:54:13.733254 24512 tablet_bootstrap.cc:654] T 5e3695e8ad0a44de9ba23ae3610aacec P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:13.735309 24533 tablet_bootstrap.cc:492] T 277c6ae5c8dc4f8eae261bf06890d793 P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:54:13.735869 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2714919062f5465c8cde1a3c4720978a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:13.738003 24533 ts_tablet_manager.cc:1367] T 277c6ae5c8dc4f8eae261bf06890d793 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.068s user 0.003s sys 0.000s
I20240502 01:54:13.738750 24533 raft_consensus.cc:348] T 277c6ae5c8dc4f8eae261bf06890d793 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:13.739024 24533 raft_consensus.cc:374] T 277c6ae5c8dc4f8eae261bf06890d793 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:13.739117 24533 raft_consensus.cc:729] T 277c6ae5c8dc4f8eae261bf06890d793 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:54:13.739363 24533 consensus_queue.cc:260] T 277c6ae5c8dc4f8eae261bf06890d793 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:13.741547 24533 ts_tablet_manager.cc:1398] T 277c6ae5c8dc4f8eae261bf06890d793 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:54:13.745229 24517 tablet_bootstrap.cc:492] T acbd240a2783424e96bd589f0232daf6 P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:54:13.745448 24517 ts_tablet_manager.cc:1367] T acbd240a2783424e96bd589f0232daf6 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.080s user 0.004s sys 0.000s
I20240502 01:54:13.746721 24517 raft_consensus.cc:348] T acbd240a2783424e96bd589f0232daf6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:13.746979 24517 raft_consensus.cc:374] T acbd240a2783424e96bd589f0232daf6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:13.747116 24517 raft_consensus.cc:729] T acbd240a2783424e96bd589f0232daf6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:54:13.749920 24548 tablet_replica.cc:330] T 68bc23294ac64c9486bffebbdb16b508 P fdc2ac372492454c85387589a91f63af: stopping tablet replica
I20240502 01:54:13.750646 20494 raft_consensus.cc:1261] T 68bc23294ac64c9486bffebbdb16b508 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:13.751420 24415 consensus_queue.cc:1035] T 68bc23294ac64c9486bffebbdb16b508 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:13.753558 24548 raft_consensus.cc:2229] T 68bc23294ac64c9486bffebbdb16b508 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:13.753892 24548 raft_consensus.cc:2258] T 68bc23294ac64c9486bffebbdb16b508 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:13.755524 24548 ts_tablet_manager.cc:1875] T 68bc23294ac64c9486bffebbdb16b508 P fdc2ac372492454c85387589a91f63af: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:13.756165 24515 tablet_bootstrap.cc:492] T 5e3695e8ad0a44de9ba23ae3610aacec P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:54:13.756470 24517 consensus_queue.cc:260] T acbd240a2783424e96bd589f0232daf6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:13.760618 17333 catalog_manager.cc:5484] T 2714919062f5465c8cde1a3c4720978a P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: config changed from index 3 to 4, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:13.761610 24517 ts_tablet_manager.cc:1398] T acbd240a2783424e96bd589f0232daf6 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240502 01:54:13.761876 24515 ts_tablet_manager.cc:1367] T 5e3695e8ad0a44de9ba23ae3610aacec P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.072s user 0.004s sys 0.000s
I20240502 01:54:13.763573 24515 raft_consensus.cc:348] T 5e3695e8ad0a44de9ba23ae3610aacec P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
W20240502 01:54:13.769877 20433 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.42.72:42411, user_credentials={real_user=slave}} blocked reactor thread for 53908.3us
I20240502 01:54:13.782227 24515 raft_consensus.cc:374] T 5e3695e8ad0a44de9ba23ae3610aacec P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:13.782686 24515 raft_consensus.cc:729] T 5e3695e8ad0a44de9ba23ae3610aacec P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:54:13.783569 20211 tablet_service.cc:1514] Processing DeleteTablet for tablet 2714919062f5465c8cde1a3c4720978a with delete_type TABLET_DATA_TOMBSTONED (TS fdc2ac372492454c85387589a91f63af not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44624
I20240502 01:54:13.783761 24515 consensus_queue.cc:260] T 5e3695e8ad0a44de9ba23ae3610aacec P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:13.787284 24515 ts_tablet_manager.cc:1398] T 5e3695e8ad0a44de9ba23ae3610aacec P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.024s user 0.001s sys 0.000s
I20240502 01:54:13.788595 24515 tablet_bootstrap.cc:492] T 277c6ae5c8dc4f8eae261bf06890d793 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:54:13.790921 24515 tablet_bootstrap.cc:654] T 277c6ae5c8dc4f8eae261bf06890d793 P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:13.798619 20497 raft_consensus.cc:1261] T 2714919062f5465c8cde1a3c4720978a P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:13.798775 24167 raft_consensus.cc:482] T 277c6ae5c8dc4f8eae261bf06890d793 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:13.798916 24167 raft_consensus.cc:504] T 277c6ae5c8dc4f8eae261bf06890d793 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:13.799502 23908 consensus_queue.cc:1035] T 2714919062f5465c8cde1a3c4720978a P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:13.799604 24167 leader_election.cc:288] T 277c6ae5c8dc4f8eae261bf06890d793 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:54:13.800374 20232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "277c6ae5c8dc4f8eae261bf06890d793" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:54:13.800268 20494 raft_consensus.cc:2932] T 68bc23294ac64c9486bffebbdb16b508 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
W20240502 01:54:13.801030 17803 leader_election.cc:341] T 277c6ae5c8dc4f8eae261bf06890d793 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:13.800391 17476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "277c6ae5c8dc4f8eae261bf06890d793" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
W20240502 01:54:13.802522 17802 leader_election.cc:341] T 277c6ae5c8dc4f8eae261bf06890d793 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:13.802640 17802 leader_election.cc:302] T 277c6ae5c8dc4f8eae261bf06890d793 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb31bceba76d466db24b9341c04900c3; no voters: 4da690c0fa13407a85c0689ce8de9ade, fdc2ac372492454c85387589a91f63af
I20240502 01:54:13.803081 24167 raft_consensus.cc:2726] T 277c6ae5c8dc4f8eae261bf06890d793 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:54:13.806355 24513 tablet_bootstrap.cc:492] T 5e3695e8ad0a44de9ba23ae3610aacec P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:54:13.806600 24513 ts_tablet_manager.cc:1367] T 5e3695e8ad0a44de9ba23ae3610aacec P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.281s user 0.024s sys 0.008s
I20240502 01:54:13.809981 20497 raft_consensus.cc:2932] T 2714919062f5465c8cde1a3c4720978a P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:13.814067 24515 tablet_bootstrap.cc:492] T 277c6ae5c8dc4f8eae261bf06890d793 P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:54:13.814297 24515 ts_tablet_manager.cc:1367] T 277c6ae5c8dc4f8eae261bf06890d793 P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20240502 01:54:13.814988 24512 tablet_bootstrap.cc:492] T 5e3695e8ad0a44de9ba23ae3610aacec P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:54:13.815085 24515 raft_consensus.cc:348] T 277c6ae5c8dc4f8eae261bf06890d793 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:13.815238 24512 ts_tablet_manager.cc:1367] T 5e3695e8ad0a44de9ba23ae3610aacec P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.084s user 0.003s sys 0.005s
I20240502 01:54:13.815352 24515 raft_consensus.cc:374] T 277c6ae5c8dc4f8eae261bf06890d793 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:13.815443 24515 raft_consensus.cc:729] T 277c6ae5c8dc4f8eae261bf06890d793 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:54:13.815701 24515 consensus_queue.cc:260] T 277c6ae5c8dc4f8eae261bf06890d793 P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:13.816324 24515 ts_tablet_manager.cc:1398] T 277c6ae5c8dc4f8eae261bf06890d793 P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:54:13.807693 24513 raft_consensus.cc:348] T 5e3695e8ad0a44de9ba23ae3610aacec P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:13.816305 24512 raft_consensus.cc:348] T 5e3695e8ad0a44de9ba23ae3610aacec P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:13.816586 24512 raft_consensus.cc:374] T 5e3695e8ad0a44de9ba23ae3610aacec P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:13.816643 24513 raft_consensus.cc:374] T 5e3695e8ad0a44de9ba23ae3610aacec P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:13.816709 24512 raft_consensus.cc:729] T 5e3695e8ad0a44de9ba23ae3610aacec P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:54:13.816803 24513 raft_consensus.cc:729] T 5e3695e8ad0a44de9ba23ae3610aacec P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:54:13.817005 24512 consensus_queue.cc:260] T 5e3695e8ad0a44de9ba23ae3610aacec P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:13.817165 24513 consensus_queue.cc:260] T 5e3695e8ad0a44de9ba23ae3610aacec P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:13.818157 24513 ts_tablet_manager.cc:1398] T 5e3695e8ad0a44de9ba23ae3610aacec P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240502 01:54:13.818511 24513 tablet_bootstrap.cc:492] T 277c6ae5c8dc4f8eae261bf06890d793 P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:54:13.819113 20628 raft_consensus.cc:1261] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [term 2 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:54:13.819811 24364 consensus_queue.cc:1035] T cf0237a4c65b495ba892871dd850bed9 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:13.821229 24512 ts_tablet_manager.cc:1398] T 5e3695e8ad0a44de9ba23ae3610aacec P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 01:54:13.829149 24548 ts_tablet_manager.cc:1888] T 68bc23294ac64c9486bffebbdb16b508 P fdc2ac372492454c85387589a91f63af: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:54:13.829754 24548 log.cc:1196] T 68bc23294ac64c9486bffebbdb16b508 P fdc2ac372492454c85387589a91f63af: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal/wals/68bc23294ac64c9486bffebbdb16b508
I20240502 01:54:13.830643 24548 tablet_replica.cc:330] T 2714919062f5465c8cde1a3c4720978a P fdc2ac372492454c85387589a91f63af: stopping tablet replica
I20240502 01:54:13.830909 24548 raft_consensus.cc:2229] T 2714919062f5465c8cde1a3c4720978a P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:13.830991 17322 catalog_manager.cc:4830] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): tablet 68bc23294ac64c9486bffebbdb16b508 (table rebalancer_extra_table_28 [id=736004bf70524edd905a6bb90bb25cc0]) successfully deleted
I20240502 01:54:13.839829 24548 raft_consensus.cc:2258] T 2714919062f5465c8cde1a3c4720978a P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:13.839915 24513 tablet_bootstrap.cc:654] T 277c6ae5c8dc4f8eae261bf06890d793 P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:13.849437 24548 ts_tablet_manager.cc:1875] T 2714919062f5465c8cde1a3c4720978a P fdc2ac372492454c85387589a91f63af: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:13.858806 24513 tablet_bootstrap.cc:492] T 277c6ae5c8dc4f8eae261bf06890d793 P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:54:13.859179 24513 ts_tablet_manager.cc:1367] T 277c6ae5c8dc4f8eae261bf06890d793 P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.041s user 0.000s sys 0.007s
I20240502 01:54:13.877413 24513 raft_consensus.cc:348] T 277c6ae5c8dc4f8eae261bf06890d793 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:13.877641 24513 raft_consensus.cc:374] T 277c6ae5c8dc4f8eae261bf06890d793 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:13.877714 24513 raft_consensus.cc:729] T 277c6ae5c8dc4f8eae261bf06890d793 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:54:13.877939 24513 consensus_queue.cc:260] T 277c6ae5c8dc4f8eae261bf06890d793 P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:13.878504 24513 ts_tablet_manager.cc:1398] T 277c6ae5c8dc4f8eae261bf06890d793 P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.017s user 0.000s sys 0.002s
I20240502 01:54:13.878823 24513 tablet_bootstrap.cc:492] T dc5b634d198f4408b30d859addd79de3 P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:54:13.880412 24513 tablet_bootstrap.cc:654] T dc5b634d198f4408b30d859addd79de3 P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:13.891120 24548 ts_tablet_manager.cc:1888] T 2714919062f5465c8cde1a3c4720978a P fdc2ac372492454c85387589a91f63af: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:54:13.894490 24212 raft_consensus.cc:482] T 277c6ae5c8dc4f8eae261bf06890d793 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:13.894685 24212 raft_consensus.cc:504] T 277c6ae5c8dc4f8eae261bf06890d793 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:13.895476 24212 leader_election.cc:288] T 277c6ae5c8dc4f8eae261bf06890d793 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:54:13.896623 17867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "277c6ae5c8dc4f8eae261bf06890d793" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:54:13.896879 17867 raft_consensus.cc:2454] T 277c6ae5c8dc4f8eae261bf06890d793 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 0.
I20240502 01:54:13.897569 17412 leader_election.cc:302] T 277c6ae5c8dc4f8eae261bf06890d793 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:54:13.898480 24212 raft_consensus.cc:2781] T 277c6ae5c8dc4f8eae261bf06890d793 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:13.898584 24212 raft_consensus.cc:482] T 277c6ae5c8dc4f8eae261bf06890d793 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:13.898692 24212 raft_consensus.cc:3037] T 277c6ae5c8dc4f8eae261bf06890d793 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:13.910815 24212 raft_consensus.cc:504] T 277c6ae5c8dc4f8eae261bf06890d793 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:13.912114 17867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "277c6ae5c8dc4f8eae261bf06890d793" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:54:13.912349 17867 raft_consensus.cc:3037] T 277c6ae5c8dc4f8eae261bf06890d793 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:13.915091 17867 raft_consensus.cc:2454] T 277c6ae5c8dc4f8eae261bf06890d793 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
I20240502 01:54:13.915709 17412 leader_election.cc:302] T 277c6ae5c8dc4f8eae261bf06890d793 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:54:13.915978 24476 raft_consensus.cc:2781] T 277c6ae5c8dc4f8eae261bf06890d793 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:13.921443 24212 leader_election.cc:288] T 277c6ae5c8dc4f8eae261bf06890d793 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Requested vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:54:13.932196 24476 raft_consensus.cc:686] T 277c6ae5c8dc4f8eae261bf06890d793 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Becoming Leader. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Running, Role: LEADER
I20240502 01:54:13.932843 24476 consensus_queue.cc:237] T 277c6ae5c8dc4f8eae261bf06890d793 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:13.896626 20231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "277c6ae5c8dc4f8eae261bf06890d793" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:54:13.933907 20231 raft_consensus.cc:2454] T 277c6ae5c8dc4f8eae261bf06890d793 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 0.
I20240502 01:54:13.934381 24548 log.cc:1196] T 2714919062f5465c8cde1a3c4720978a P fdc2ac372492454c85387589a91f63af: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal/wals/2714919062f5465c8cde1a3c4720978a
I20240502 01:54:13.921602 20232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "277c6ae5c8dc4f8eae261bf06890d793" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:54:13.936295 20232 raft_consensus.cc:3037] T 277c6ae5c8dc4f8eae261bf06890d793 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:13.936337 17322 catalog_manager.cc:4830] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): tablet 2714919062f5465c8cde1a3c4720978a (table rebalancer_extra_table_20 [id=0e1c9ce53d7c47f7a42d19e60342c557]) successfully deleted
I20240502 01:54:13.937744 24513 tablet_bootstrap.cc:492] T dc5b634d198f4408b30d859addd79de3 P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:54:13.937909 24513 ts_tablet_manager.cc:1367] T dc5b634d198f4408b30d859addd79de3 P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.059s user 0.002s sys 0.007s
I20240502 01:54:13.938783 24513 raft_consensus.cc:348] T dc5b634d198f4408b30d859addd79de3 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:13.939002 24513 raft_consensus.cc:374] T dc5b634d198f4408b30d859addd79de3 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:13.939088 24513 raft_consensus.cc:729] T dc5b634d198f4408b30d859addd79de3 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:54:13.939327 24513 consensus_queue.cc:260] T dc5b634d198f4408b30d859addd79de3 P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:13.940333 20232 raft_consensus.cc:2454] T 277c6ae5c8dc4f8eae261bf06890d793 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
I20240502 01:54:13.944964 24513 ts_tablet_manager.cc:1398] T dc5b634d198f4408b30d859addd79de3 P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:54:13.957937 17867 raft_consensus.cc:1261] T cf0237a4c65b495ba892871dd850bed9 P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:54:13.955320 17328 catalog_manager.cc:5484] T 277c6ae5c8dc4f8eae261bf06890d793 P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: term changed from 0 to 1, leader changed from <none> to 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65). New cstate: current_term: 1 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:13.958508 23967 consensus_queue.cc:1035] T cf0237a4c65b495ba892871dd850bed9 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:13.969812 20497 raft_consensus.cc:1261] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:54:13.970345 24203 consensus_queue.cc:1035] T cf0237a4c65b495ba892871dd850bed9 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:13.971310 24453 raft_consensus.cc:482] T dc5b634d198f4408b30d859addd79de3 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:13.971446 24453 raft_consensus.cc:504] T dc5b634d198f4408b30d859addd79de3 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:13.972102 24453 leader_election.cc:288] T dc5b634d198f4408b30d859addd79de3 P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:54:13.975095 20628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc5b634d198f4408b30d859addd79de3" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091" is_pre_election: true
I20240502 01:54:13.975327 20628 raft_consensus.cc:2454] T dc5b634d198f4408b30d859addd79de3 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 0.
I20240502 01:54:13.976644 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc5b634d198f4408b30d859addd79de3" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:54:13.976846 17735 raft_consensus.cc:2454] T dc5b634d198f4408b30d859addd79de3 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 0.
I20240502 01:54:13.977051 20167 leader_election.cc:302] T dc5b634d198f4408b30d859addd79de3 P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f19cc6cb1e76460aa6711e49a5775091, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:54:13.977319 24453 raft_consensus.cc:2781] T dc5b634d198f4408b30d859addd79de3 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:13.977452 24453 raft_consensus.cc:482] T dc5b634d198f4408b30d859addd79de3 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:13.977538 24453 raft_consensus.cc:3037] T dc5b634d198f4408b30d859addd79de3 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:13.979571 24453 raft_consensus.cc:504] T dc5b634d198f4408b30d859addd79de3 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:13.980132 24453 leader_election.cc:288] T dc5b634d198f4408b30d859addd79de3 P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 election: Requested vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:54:13.981725 20628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc5b634d198f4408b30d859addd79de3" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091"
I20240502 01:54:13.981756 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc5b634d198f4408b30d859addd79de3" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:54:13.981921 17735 raft_consensus.cc:3037] T dc5b634d198f4408b30d859addd79de3 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:13.981925 20628 raft_consensus.cc:3037] T dc5b634d198f4408b30d859addd79de3 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:13.983965 17735 raft_consensus.cc:2454] T dc5b634d198f4408b30d859addd79de3 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 1.
I20240502 01:54:13.984270 20167 leader_election.cc:302] T dc5b634d198f4408b30d859addd79de3 P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dfa1152987644bd781b46c7e42a7a685, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:54:13.984504 24453 raft_consensus.cc:2781] T dc5b634d198f4408b30d859addd79de3 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:13.984627 24453 raft_consensus.cc:686] T dc5b634d198f4408b30d859addd79de3 P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Becoming Leader. State: Replica: fdc2ac372492454c85387589a91f63af, State: Running, Role: LEADER
I20240502 01:54:13.984766 20628 raft_consensus.cc:2454] T dc5b634d198f4408b30d859addd79de3 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 1.
I20240502 01:54:13.984903 24453 consensus_queue.cc:237] T dc5b634d198f4408b30d859addd79de3 P fdc2ac372492454c85387589a91f63af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:13.988619 17326 catalog_manager.cc:5484] T dc5b634d198f4408b30d859addd79de3 P fdc2ac372492454c85387589a91f63af reported cstate change: term changed from 0 to 1, leader changed from <none> to fdc2ac372492454c85387589a91f63af (127.11.42.69). New cstate: current_term: 1 leader_uuid: "fdc2ac372492454c85387589a91f63af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:14.001742 20628 raft_consensus.cc:1261] T 5df83f83d2674155afba93c2d992af0a P f19cc6cb1e76460aa6711e49a5775091 [term 2 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:54:14.002312 24564 consensus_queue.cc:1035] T 5df83f83d2674155afba93c2d992af0a P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:14.018260 17604 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4aefc61cb95f4e949e7a23a1c677749e"
dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53554
I20240502 01:54:14.018447 17604 raft_consensus.cc:595] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:54:14.018488 24545 raft_consensus.cc:482] T acbd240a2783424e96bd589f0232daf6 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:14.018676 24545 raft_consensus.cc:504] T acbd240a2783424e96bd589f0232daf6 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:14.019416 24545 leader_election.cc:288] T acbd240a2783424e96bd589f0232daf6 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:54:14.019801 20362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acbd240a2783424e96bd589f0232daf6" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:54:14.020016 20362 raft_consensus.cc:2454] T acbd240a2783424e96bd589f0232daf6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 0.
I20240502 01:54:14.020627 20433 leader_election.cc:302] T acbd240a2783424e96bd589f0232daf6 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 6a36c952c51542c0bf7ea3ec2857a9f9; no voters:
I20240502 01:54:14.020869 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acbd240a2783424e96bd589f0232daf6" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:54:14.021118 17735 raft_consensus.cc:2454] T acbd240a2783424e96bd589f0232daf6 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 0.
I20240502 01:54:14.022310 24545 raft_consensus.cc:2781] T acbd240a2783424e96bd589f0232daf6 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:14.022401 24545 raft_consensus.cc:482] T acbd240a2783424e96bd589f0232daf6 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:14.022496 24545 raft_consensus.cc:3037] T acbd240a2783424e96bd589f0232daf6 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:14.024684 24545 raft_consensus.cc:504] T acbd240a2783424e96bd589f0232daf6 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:14.025513 20362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acbd240a2783424e96bd589f0232daf6" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
I20240502 01:54:14.025727 20362 raft_consensus.cc:3037] T acbd240a2783424e96bd589f0232daf6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:14.025926 24545 leader_election.cc:288] T acbd240a2783424e96bd589f0232daf6 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 election: Requested vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:54:14.027762 20362 raft_consensus.cc:2454] T acbd240a2783424e96bd589f0232daf6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 1.
I20240502 01:54:14.028165 20433 leader_election.cc:302] T acbd240a2783424e96bd589f0232daf6 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 6a36c952c51542c0bf7ea3ec2857a9f9; no voters:
I20240502 01:54:14.028858 17737 raft_consensus.cc:1261] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:54:14.028493 24545 raft_consensus.cc:2781] T acbd240a2783424e96bd589f0232daf6 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:14.029419 24564 consensus_queue.cc:1035] T 5df83f83d2674155afba93c2d992af0a P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:14.029796 24545 raft_consensus.cc:686] T acbd240a2783424e96bd589f0232daf6 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Running, Role: LEADER
I20240502 01:54:14.030081 24545 consensus_queue.cc:237] T acbd240a2783424e96bd589f0232daf6 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:14.032037 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acbd240a2783424e96bd589f0232daf6" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:54:14.032223 17735 raft_consensus.cc:3037] T acbd240a2783424e96bd589f0232daf6 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:14.035001 17735 raft_consensus.cc:2454] T acbd240a2783424e96bd589f0232daf6 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 1.
I20240502 01:54:14.047717 17326 catalog_manager.cc:5484] T acbd240a2783424e96bd589f0232daf6 P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71). New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:14.057227 20232 raft_consensus.cc:1261] T 5df83f83d2674155afba93c2d992af0a P fdc2ac372492454c85387589a91f63af [term 2 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:54:14.057859 24212 consensus_queue.cc:1035] T 5df83f83d2674155afba93c2d992af0a P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:14.095750 17604 consensus_queue.cc:237] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
W20240502 01:54:14.098387 17543 consensus_peers.cc:485] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): Couldn't send request to peer 2f306549407b4bb281ccf2994035e4f1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1fdd104f1a28446cbc25d4ca16d03d7d. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:14.098379 20362 raft_consensus.cc:1261] T 1fdd104f1a28446cbc25d4ca16d03d7d P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:14.098853 17737 raft_consensus.cc:1261] T 1fdd104f1a28446cbc25d4ca16d03d7d P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:14.098824 23807 consensus_queue.cc:1035] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:14.099375 23807 consensus_queue.cc:1035] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:14.101902 24598 raft_consensus.cc:2932] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:54:14.102545 17737 raft_consensus.cc:2932] T 1fdd104f1a28446cbc25d4ca16d03d7d P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:54:14.102648 20362 raft_consensus.cc:2932] T 1fdd104f1a28446cbc25d4ca16d03d7d P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:54:14.110136 17326 catalog_manager.cc:5484] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: config changed from index 6 to 7, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:14.118855 23807 raft_consensus.cc:979] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29: : Instructing follower f19cc6cb1e76460aa6711e49a5775091 to start an election
I20240502 01:54:14.118995 23807 raft_consensus.cc:1067] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Signalling peer f19cc6cb1e76460aa6711e49a5775091 to start an election
I20240502 01:54:14.121124 20628 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4aefc61cb95f4e949e7a23a1c677749e"
dest_uuid: "f19cc6cb1e76460aa6711e49a5775091"
from {username='slave'} at 127.11.42.66:54577
I20240502 01:54:14.121327 20628 raft_consensus.cc:482] T 4aefc61cb95f4e949e7a23a1c677749e P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:14.121410 20628 raft_consensus.cc:3037] T 4aefc61cb95f4e949e7a23a1c677749e P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:14.126366 20628 raft_consensus.cc:504] T 4aefc61cb95f4e949e7a23a1c677749e P f19cc6cb1e76460aa6711e49a5775091 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:14.127732 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aefc61cb95f4e949e7a23a1c677749e" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:54:14.127912 17604 raft_consensus.cc:3032] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:14.127985 17604 raft_consensus.cc:729] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Running, Role: LEADER
I20240502 01:54:14.127933 20628 leader_election.cc:288] T 4aefc61cb95f4e949e7a23a1c677749e P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 2 election: Requested vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:54:14.128139 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aefc61cb95f4e949e7a23a1c677749e" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:54:14.128327 17737 raft_consensus.cc:3037] T 4aefc61cb95f4e949e7a23a1c677749e P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:14.128240 17604 consensus_queue.cc:260] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:14.128648 17604 raft_consensus.cc:3037] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:14.131187 17604 raft_consensus.cc:2454] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 2.
I20240502 01:54:14.132088 17737 raft_consensus.cc:2454] T 4aefc61cb95f4e949e7a23a1c677749e P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 2.
I20240502 01:54:14.132397 17867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aefc61cb95f4e949e7a23a1c677749e" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:54:14.132544 20564 leader_election.cc:302] T 4aefc61cb95f4e949e7a23a1c677749e P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, dfa1152987644bd781b46c7e42a7a685, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:54:14.132652 17867 raft_consensus.cc:3037] T 4aefc61cb95f4e949e7a23a1c677749e P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:14.133065 24127 raft_consensus.cc:2781] T 4aefc61cb95f4e949e7a23a1c677749e P f19cc6cb1e76460aa6711e49a5775091 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:14.133219 24127 raft_consensus.cc:686] T 4aefc61cb95f4e949e7a23a1c677749e P f19cc6cb1e76460aa6711e49a5775091 [term 2 LEADER]: Becoming Leader. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Running, Role: LEADER
I20240502 01:54:14.133517 24127 consensus_queue.cc:237] T 4aefc61cb95f4e949e7a23a1c677749e P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:14.135425 17867 raft_consensus.cc:2454] T 4aefc61cb95f4e949e7a23a1c677749e P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 2.
I20240502 01:54:14.144006 17326 catalog_manager.cc:5484] T 4aefc61cb95f4e949e7a23a1c677749e P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: term changed from 1 to 2, leader changed from 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) to f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72). New cstate: current_term: 2 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:14.152755 20232 consensus_queue.cc:237] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
W20240502 01:54:14.155311 20167 consensus_peers.cc:485] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3e4c8f98b5e48e3ba6affc2df08943a. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:14.155481 20362 raft_consensus.cc:1261] T f3e4c8f98b5e48e3ba6affc2df08943a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:14.155625 20497 raft_consensus.cc:1261] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:14.156056 24453 consensus_queue.cc:1035] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:14.156626 24172 consensus_queue.cc:1035] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:14.159222 24453 raft_consensus.cc:2932] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:54:14.162426 20497 raft_consensus.cc:2932] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:54:14.173154 17326 catalog_manager.cc:5484] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af reported cstate change: config changed from index -1 to 2, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New cstate: current_term: 1 leader_uuid: "fdc2ac372492454c85387589a91f63af" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:14.175377 20362 raft_consensus.cc:2932] T f3e4c8f98b5e48e3ba6affc2df08943a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:54:14.181469 17328 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:54:14.187995 24172 raft_consensus.cc:482] T 5e3695e8ad0a44de9ba23ae3610aacec P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:14.188186 24172 raft_consensus.cc:504] T 5e3695e8ad0a44de9ba23ae3610aacec P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:14.188951 24172 leader_election.cc:288] T 5e3695e8ad0a44de9ba23ae3610aacec P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:54:14.189445 20497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e3695e8ad0a44de9ba23ae3610aacec" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
I20240502 01:54:14.189654 20497 raft_consensus.cc:2454] T 5e3695e8ad0a44de9ba23ae3610aacec P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 0.
I20240502 01:54:14.190428 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e3695e8ad0a44de9ba23ae3610aacec" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:54:14.190605 20168 leader_election.cc:302] T 5e3695e8ad0a44de9ba23ae3610aacec P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:54:14.190640 17475 raft_consensus.cc:2454] T 5e3695e8ad0a44de9ba23ae3610aacec P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 0.
I20240502 01:54:14.191208 17737 raft_consensus.cc:1261] T dc5b634d198f4408b30d859addd79de3 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:14.191906 20628 raft_consensus.cc:1261] T dc5b634d198f4408b30d859addd79de3 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:14.193552 24172 raft_consensus.cc:2781] T 5e3695e8ad0a44de9ba23ae3610aacec P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:14.193650 24172 raft_consensus.cc:482] T 5e3695e8ad0a44de9ba23ae3610aacec P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:14.193756 24172 raft_consensus.cc:3037] T 5e3695e8ad0a44de9ba23ae3610aacec P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:14.196259 24172 raft_consensus.cc:504] T 5e3695e8ad0a44de9ba23ae3610aacec P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:14.196919 24172 leader_election.cc:288] T 5e3695e8ad0a44de9ba23ae3610aacec P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 election: Requested vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:54:14.197188 20497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e3695e8ad0a44de9ba23ae3610aacec" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
I20240502 01:54:14.197367 20497 raft_consensus.cc:3037] T 5e3695e8ad0a44de9ba23ae3610aacec P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:14.200248 24172 consensus_queue.cc:1035] T dc5b634d198f4408b30d859addd79de3 P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:14.201568 24172 consensus_queue.cc:1035] T dc5b634d198f4408b30d859addd79de3 P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:14.205262 20497 raft_consensus.cc:2454] T 5e3695e8ad0a44de9ba23ae3610aacec P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 1.
I20240502 01:54:14.205703 20168 leader_election.cc:302] T 5e3695e8ad0a44de9ba23ae3610aacec P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:54:14.206151 24611 raft_consensus.cc:2781] T 5e3695e8ad0a44de9ba23ae3610aacec P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:14.206346 24611 raft_consensus.cc:686] T 5e3695e8ad0a44de9ba23ae3610aacec P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Becoming Leader. State: Replica: fdc2ac372492454c85387589a91f63af, State: Running, Role: LEADER
I20240502 01:54:14.206645 24611 consensus_queue.cc:237] T 5e3695e8ad0a44de9ba23ae3610aacec P fdc2ac372492454c85387589a91f63af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:14.217150 17326 catalog_manager.cc:5484] T 5e3695e8ad0a44de9ba23ae3610aacec P fdc2ac372492454c85387589a91f63af reported cstate change: term changed from 0 to 1, leader changed from <none> to fdc2ac372492454c85387589a91f63af (127.11.42.69). New cstate: current_term: 1 leader_uuid: "fdc2ac372492454c85387589a91f63af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:14.223299 20497 consensus_queue.cc:237] T 4bb42883b1a0475b863d008bd01fa70a P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } }
I20240502 01:54:14.225457 17604 raft_consensus.cc:1261] T 4bb42883b1a0475b863d008bd01fa70a P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:54:14.226190 20628 raft_consensus.cc:1261] T 4bb42883b1a0475b863d008bd01fa70a P f19cc6cb1e76460aa6711e49a5775091 [term 2 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:54:14.227519 24547 consensus_queue.cc:1035] T 4bb42883b1a0475b863d008bd01fa70a P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:54:14.228269 20497 consensus_queue.cc:237] T ee418fc4028a422eabac02ed63ea9047 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } }
I20240502 01:54:14.228335 24547 consensus_queue.cc:1035] T 4bb42883b1a0475b863d008bd01fa70a P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:54:14.229992 24547 raft_consensus.cc:2932] T 4bb42883b1a0475b863d008bd01fa70a P 2f306549407b4bb281ccf2994035e4f1 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:14.231577 20628 raft_consensus.cc:2932] T 4bb42883b1a0475b863d008bd01fa70a P f19cc6cb1e76460aa6711e49a5775091 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:14.231961 17604 raft_consensus.cc:2932] T 4bb42883b1a0475b863d008bd01fa70a P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:14.235934 17322 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4bb42883b1a0475b863d008bd01fa70a with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:14.236838 17328 catalog_manager.cc:5484] T 4bb42883b1a0475b863d008bd01fa70a P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: config changed from index 8 to 10, VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) evicted. New cstate: current_term: 2 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 01:54:14.239101 20167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.42.67:36115, user_credentials={real_user=slave}} blocked reactor thread for 46499.8us
I20240502 01:54:14.241940 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e3695e8ad0a44de9ba23ae3610aacec" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:54:14.242161 17475 raft_consensus.cc:3037] T 5e3695e8ad0a44de9ba23ae3610aacec P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:54:14.242173 20434 consensus_peers.cc:485] T ee418fc4028a422eabac02ed63ea9047 P 2f306549407b4bb281ccf2994035e4f1 -> Peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): Couldn't send request to peer bb31bceba76d466db24b9341c04900c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ee418fc4028a422eabac02ed63ea9047. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:14.244303 17475 raft_consensus.cc:2454] T 5e3695e8ad0a44de9ba23ae3610aacec P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 1.
I20240502 01:54:14.246730 20362 raft_consensus.cc:1261] T ee418fc4028a422eabac02ed63ea9047 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:14.247385 23694 consensus_queue.cc:1035] T ee418fc4028a422eabac02ed63ea9047 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:14.250329 17847 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bb42883b1a0475b863d008bd01fa70a with delete_type TABLET_DATA_TOMBSTONED (TS bb31bceba76d466db24b9341c04900c3 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:47226
I20240502 01:54:14.251263 17326 catalog_manager.cc:5576] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115) does not have the latest schema for tablet dc5b634d198f4408b30d859addd79de3 (table rebalancer_dynamic_table [id=6dc57802503d4ec5b7d530db90daec99]). Expected version 1 got 0
I20240502 01:54:14.254006 17475 raft_consensus.cc:1261] T ee418fc4028a422eabac02ed63ea9047 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:14.254995 23694 consensus_queue.cc:1035] T ee418fc4028a422eabac02ed63ea9047 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:14.259771 24626 tablet_replica.cc:330] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3: stopping tablet replica
I20240502 01:54:14.259917 23694 raft_consensus.cc:2932] T ee418fc4028a422eabac02ed63ea9047 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } } }
I20240502 01:54:14.260481 24626 raft_consensus.cc:2229] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:14.261197 24626 raft_consensus.cc:2258] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:14.263027 24626 ts_tablet_manager.cc:1875] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:14.264966 17476 raft_consensus.cc:2932] T ee418fc4028a422eabac02ed63ea9047 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } } }
I20240502 01:54:14.268532 17328 catalog_manager.cc:5484] T ee418fc4028a422eabac02ed63ea9047 P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: config changed from index -1 to 2, NON_VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) added. New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:14.269920 17333 catalog_manager.cc:5576] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411) does not have the latest schema for tablet dc5b634d198f4408b30d859addd79de3 (table rebalancer_dynamic_table [id=6dc57802503d4ec5b7d530db90daec99]). Expected version 1 got 0
I20240502 01:54:14.288264 24631 tablet.cc:1722] T dc5b634d198f4408b30d859addd79de3 P dfa1152987644bd781b46c7e42a7a685: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:14.289487 24631 tablet.cc:1620] T dc5b634d198f4408b30d859addd79de3 P dfa1152987644bd781b46c7e42a7a685: MemRowSet was empty: no flush needed.
I20240502 01:54:14.291903 17328 catalog_manager.cc:5576] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115) does not have the latest schema for tablet dc5b634d198f4408b30d859addd79de3 (table rebalancer_dynamic_table [id=6dc57802503d4ec5b7d530db90daec99]). Expected version 1 got 0
I20240502 01:54:14.294127 24628 tablet.cc:1722] T dc5b634d198f4408b30d859addd79de3 P fdc2ac372492454c85387589a91f63af: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:14.296500 24626 ts_tablet_manager.cc:1888] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 01:54:14.297391 24628 tablet.cc:1620] T dc5b634d198f4408b30d859addd79de3 P fdc2ac372492454c85387589a91f63af: MemRowSet was empty: no flush needed.
I20240502 01:54:14.296988 20362 raft_consensus.cc:2932] T ee418fc4028a422eabac02ed63ea9047 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } } }
I20240502 01:54:14.301287 17330 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=6dc57802503d4ec5b7d530db90daec99] alter complete (version 1)
I20240502 01:54:14.313558 24630 tablet.cc:1722] T dc5b634d198f4408b30d859addd79de3 P f19cc6cb1e76460aa6711e49a5775091: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:14.314078 24630 tablet.cc:1620] T dc5b634d198f4408b30d859addd79de3 P f19cc6cb1e76460aa6711e49a5775091: MemRowSet was empty: no flush needed.
I20240502 01:54:14.314661 24626 log.cc:1196] T 4bb42883b1a0475b863d008bd01fa70a P bb31bceba76d466db24b9341c04900c3: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/wals/4bb42883b1a0475b863d008bd01fa70a
I20240502 01:54:14.317348 17328 catalog_manager.cc:5576] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411) does not have the latest schema for tablet dc5b634d198f4408b30d859addd79de3 (table rebalancer_dynamic_table [id=6dc57802503d4ec5b7d530db90daec99]). Expected version 1 got 0
I20240502 01:54:14.347124 17322 catalog_manager.cc:4830] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): tablet 4bb42883b1a0475b863d008bd01fa70a (table rebalance_test_table_17 [id=c6474d0c5256497eb6fa5844f3530b62]) successfully deleted
I20240502 01:54:14.348883 24564 consensus_queue.cc:1035] T 277c6ae5c8dc4f8eae261bf06890d793 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:14.355175 24212 consensus_queue.cc:1035] T 277c6ae5c8dc4f8eae261bf06890d793 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:14.361981 17735 consensus_queue.cc:237] T 29d8cb76dc514e7b9002bd2291d8ee8d P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } }
I20240502 01:54:14.364416 20362 raft_consensus.cc:1261] T 29d8cb76dc514e7b9002bd2291d8ee8d P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:54:14.364601 17673 consensus_peers.cc:485] T 29d8cb76dc514e7b9002bd2291d8ee8d P dfa1152987644bd781b46c7e42a7a685 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 29d8cb76dc514e7b9002bd2291d8ee8d. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:14.364691 20497 raft_consensus.cc:1261] T 29d8cb76dc514e7b9002bd2291d8ee8d P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:14.365047 24364 consensus_queue.cc:1035] T 29d8cb76dc514e7b9002bd2291d8ee8d P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:14.365283 24571 consensus_queue.cc:1035] T 29d8cb76dc514e7b9002bd2291d8ee8d P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:14.367910 24364 raft_consensus.cc:2932] T 29d8cb76dc514e7b9002bd2291d8ee8d P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:54:14.370249 20498 raft_consensus.cc:2932] T 29d8cb76dc514e7b9002bd2291d8ee8d P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:54:14.373780 20362 raft_consensus.cc:2932] T 29d8cb76dc514e7b9002bd2291d8ee8d P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:54:14.375553 17330 catalog_manager.cc:5484] T 29d8cb76dc514e7b9002bd2291d8ee8d P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: config changed from index -1 to 2, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:54:14.426663 17328 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:54:14.435684 24628 tablet.cc:1722] T dc5b634d198f4408b30d859addd79de3 P fdc2ac372492454c85387589a91f63af: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:14.436178 24628 tablet.cc:1620] T dc5b634d198f4408b30d859addd79de3 P fdc2ac372492454c85387589a91f63af: MemRowSet was empty: no flush needed.
I20240502 01:54:14.436437 24631 tablet.cc:1722] T dc5b634d198f4408b30d859addd79de3 P dfa1152987644bd781b46c7e42a7a685: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:14.436872 24631 tablet.cc:1620] T dc5b634d198f4408b30d859addd79de3 P dfa1152987644bd781b46c7e42a7a685: MemRowSet was empty: no flush needed.
I20240502 01:54:14.438679 24630 tablet.cc:1722] T dc5b634d198f4408b30d859addd79de3 P f19cc6cb1e76460aa6711e49a5775091: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:14.439129 24630 tablet.cc:1620] T dc5b634d198f4408b30d859addd79de3 P f19cc6cb1e76460aa6711e49a5775091: MemRowSet was empty: no flush needed.
I20240502 01:54:14.447086 24547 consensus_queue.cc:1035] T acbd240a2783424e96bd589f0232daf6 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:14.448298 17330 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=6dc57802503d4ec5b7d530db90daec99] alter complete (version 2)
I20240502 01:54:14.462391 24545 consensus_queue.cc:1035] T acbd240a2783424e96bd589f0232daf6 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:14.487782 20628 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1b14ac3413004127b72904b631ec1b94"
dest_uuid: "f19cc6cb1e76460aa6711e49a5775091"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45766
I20240502 01:54:14.487987 20628 raft_consensus.cc:595] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:54:14.528216 17604 raft_consensus.cc:1261] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:54:14.528764 24127 consensus_queue.cc:1035] T 4aefc61cb95f4e949e7a23a1c677749e P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:14.551725 17330 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:54:14.555775 17735 raft_consensus.cc:1261] T 4aefc61cb95f4e949e7a23a1c677749e P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:54:14.556773 24127 consensus_queue.cc:1035] T 4aefc61cb95f4e949e7a23a1c677749e P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:14.564849 17475 consensus_queue.cc:237] T 5df83f83d2674155afba93c2d992af0a P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } }
I20240502 01:54:14.566046 17867 raft_consensus.cc:1261] T 4aefc61cb95f4e949e7a23a1c677749e P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:54:14.567551 24125 consensus_queue.cc:1035] T 4aefc61cb95f4e949e7a23a1c677749e P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:14.568269 20625 raft_consensus.cc:1261] T 5df83f83d2674155afba93c2d992af0a P f19cc6cb1e76460aa6711e49a5775091 [term 2 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:54:14.568902 24476 consensus_queue.cc:1035] T 5df83f83d2674155afba93c2d992af0a P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:54:14.570001 20231 raft_consensus.cc:1261] T 5df83f83d2674155afba93c2d992af0a P fdc2ac372492454c85387589a91f63af [term 2 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:54:14.570712 24476 consensus_queue.cc:1035] T 5df83f83d2674155afba93c2d992af0a P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:54:14.573041 24628 tablet.cc:1722] T dc5b634d198f4408b30d859addd79de3 P fdc2ac372492454c85387589a91f63af: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:14.574280 24564 raft_consensus.cc:2932] T 5df83f83d2674155afba93c2d992af0a P 4da690c0fa13407a85c0689ce8de9ade [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:14.574934 24631 tablet.cc:1722] T dc5b634d198f4408b30d859addd79de3 P dfa1152987644bd781b46c7e42a7a685: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:14.577090 24630 tablet.cc:1722] T dc5b634d198f4408b30d859addd79de3 P f19cc6cb1e76460aa6711e49a5775091: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:14.587996 20231 raft_consensus.cc:2932] T 5df83f83d2674155afba93c2d992af0a P fdc2ac372492454c85387589a91f63af [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:14.593627 20625 raft_consensus.cc:2932] T 5df83f83d2674155afba93c2d992af0a P f19cc6cb1e76460aa6711e49a5775091 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:14.594156 24125 raft_consensus.cc:979] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091: : Instructing follower dfa1152987644bd781b46c7e42a7a685 to start an election
I20240502 01:54:14.594291 24125 raft_consensus.cc:1067] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Signalling peer dfa1152987644bd781b46c7e42a7a685 to start an election
I20240502 01:54:14.595777 17735 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1b14ac3413004127b72904b631ec1b94"
dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
from {username='slave'} at 127.11.42.72:53665
I20240502 01:54:14.595959 17735 raft_consensus.cc:482] T 1b14ac3413004127b72904b631ec1b94 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:14.596037 17735 raft_consensus.cc:3037] T 1b14ac3413004127b72904b631ec1b94 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:14.597121 17333 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table_new_name [id=6dc57802503d4ec5b7d530db90daec99] alter complete (version 3)
I20240502 01:54:14.600069 17735 raft_consensus.cc:504] T 1b14ac3413004127b72904b631ec1b94 P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:14.601310 20232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b14ac3413004127b72904b631ec1b94" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:54:14.601529 20232 raft_consensus.cc:3037] T 1b14ac3413004127b72904b631ec1b94 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:14.604386 20232 raft_consensus.cc:2454] T 1b14ac3413004127b72904b631ec1b94 P fdc2ac372492454c85387589a91f63af [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 2.
I20240502 01:54:14.605930 20628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b14ac3413004127b72904b631ec1b94" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "f19cc6cb1e76460aa6711e49a5775091"
I20240502 01:54:14.606161 20628 raft_consensus.cc:3032] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:14.606240 20628 raft_consensus.cc:729] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Running, Role: LEADER
I20240502 01:54:14.606499 20628 consensus_queue.cc:260] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:14.606931 20628 raft_consensus.cc:3037] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:14.608181 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5df83f83d2674155afba93c2d992af0a with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:14.611018 17326 catalog_manager.cc:5484] T 5df83f83d2674155afba93c2d992af0a P fdc2ac372492454c85387589a91f63af reported cstate change: config changed from index 8 to 10, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New cstate: current_term: 2 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:14.612243 20628 raft_consensus.cc:2454] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 2.
I20240502 01:54:14.613565 17672 leader_election.cc:302] T 1b14ac3413004127b72904b631ec1b94 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: dfa1152987644bd781b46c7e42a7a685, f19cc6cb1e76460aa6711e49a5775091, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:54:14.613904 24364 raft_consensus.cc:2781] T 1b14ac3413004127b72904b631ec1b94 P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:14.614039 24364 raft_consensus.cc:686] T 1b14ac3413004127b72904b631ec1b94 P dfa1152987644bd781b46c7e42a7a685 [term 2 LEADER]: Becoming Leader. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Running, Role: LEADER
I20240502 01:54:14.614331 24364 consensus_queue.cc:237] T 1b14ac3413004127b72904b631ec1b94 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:14.616200 17716 tablet_service.cc:1514] Processing DeleteTablet for tablet 5df83f83d2674155afba93c2d992af0a with delete_type TABLET_DATA_TOMBSTONED (TS dfa1152987644bd781b46c7e42a7a685 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:52198
I20240502 01:54:14.616627 17737 consensus_queue.cc:237] T cf0237a4c65b495ba892871dd850bed9 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } }
I20240502 01:54:14.618494 17864 raft_consensus.cc:1261] T cf0237a4c65b495ba892871dd850bed9 P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 01:54:14.619122 20497 raft_consensus.cc:1261] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 01:54:14.620432 24571 consensus_queue.cc:1035] T cf0237a4c65b495ba892871dd850bed9 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:14.620954 24571 consensus_queue.cc:1035] T cf0237a4c65b495ba892871dd850bed9 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.002s
I20240502 01:54:14.628909 17735 leader_election.cc:288] T 1b14ac3413004127b72904b631ec1b94 P dfa1152987644bd781b46c7e42a7a685 [CANDIDATE]: Term 2 election: Requested vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:54:14.631058 17605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b14ac3413004127b72904b631ec1b94" candidate_uuid: "dfa1152987644bd781b46c7e42a7a685" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:54:14.631274 17605 raft_consensus.cc:3037] T 1b14ac3413004127b72904b631ec1b94 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:14.633822 17605 raft_consensus.cc:2454] T 1b14ac3413004127b72904b631ec1b94 P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa1152987644bd781b46c7e42a7a685 in term 2.
I20240502 01:54:14.632761 24203 raft_consensus.cc:2932] T cf0237a4c65b495ba892871dd850bed9 P dfa1152987644bd781b46c7e42a7a685 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:14.638171 24620 consensus_queue.cc:1035] T 5e3695e8ad0a44de9ba23ae3610aacec P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:14.638717 20497 raft_consensus.cc:2932] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:14.641288 17864 raft_consensus.cc:2932] T cf0237a4c65b495ba892871dd850bed9 P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:14.643128 24727 ts_tablet_manager.cc:902] T f3e4c8f98b5e48e3ba6affc2df08943a P f19cc6cb1e76460aa6711e49a5775091: Initiating tablet copy from peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:54:14.650622 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cf0237a4c65b495ba892871dd850bed9 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:14.652464 17330 catalog_manager.cc:5484] T cf0237a4c65b495ba892871dd850bed9 P bb31bceba76d466db24b9341c04900c3 reported cstate change: config changed from index 6 to 8, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New cstate: current_term: 2 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:14.657356 20609 tablet_service.cc:1514] Processing DeleteTablet for tablet cf0237a4c65b495ba892871dd850bed9 with delete_type TABLET_DATA_TOMBSTONED (TS f19cc6cb1e76460aa6711e49a5775091 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:33698
I20240502 01:54:14.665577 17334 catalog_manager.cc:5484] T 1b14ac3413004127b72904b631ec1b94 P dfa1152987644bd781b46c7e42a7a685 reported cstate change: term changed from 1 to 2, leader changed from f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) to dfa1152987644bd781b46c7e42a7a685 (127.11.42.67). New cstate: current_term: 2 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:14.667657 24726 tablet_replica.cc:330] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685: stopping tablet replica
I20240502 01:54:14.676563 24620 consensus_queue.cc:1035] T 5e3695e8ad0a44de9ba23ae3610aacec P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:14.678243 24730 tablet_replica.cc:330] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091: stopping tablet replica
I20240502 01:54:14.679080 24733 ts_tablet_manager.cc:902] T 1fdd104f1a28446cbc25d4ca16d03d7d P 2f306549407b4bb281ccf2994035e4f1: Initiating tablet copy from peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:54:14.678678 24730 raft_consensus.cc:2229] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:14.681715 24730 raft_consensus.cc:2258] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:14.682770 24730 ts_tablet_manager.cc:1875] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:14.668412 24726 raft_consensus.cc:2229] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:14.684477 24726 raft_consensus.cc:2258] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:14.688503 24726 ts_tablet_manager.cc:1875] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:14.700925 17334 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:14.701210 17334 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:14.708313 24727 tablet_copy_client.cc:308] T f3e4c8f98b5e48e3ba6affc2df08943a P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.69:34339
I20240502 01:54:14.711102 24730 ts_tablet_manager.cc:1888] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240502 01:54:14.711333 24730 log.cc:1196] T cf0237a4c65b495ba892871dd850bed9 P f19cc6cb1e76460aa6711e49a5775091: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal/wals/cf0237a4c65b495ba892871dd850bed9
I20240502 01:54:14.713189 17321 catalog_manager.cc:4830] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): tablet cf0237a4c65b495ba892871dd850bed9 (table rebalancer_extra_table_2 [id=2444c341261d4eb29c484622291d0517]) successfully deleted
I20240502 01:54:14.713743 20253 tablet_copy_service.cc:140] P fdc2ac372492454c85387589a91f63af: Received BeginTabletCopySession request for tablet f3e4c8f98b5e48e3ba6affc2df08943a from peer f19cc6cb1e76460aa6711e49a5775091 ({username='slave'} at 127.11.42.72:41729)
I20240502 01:54:14.713949 20253 tablet_copy_service.cc:161] P fdc2ac372492454c85387589a91f63af: Beginning new tablet copy session on tablet f3e4c8f98b5e48e3ba6affc2df08943a from peer f19cc6cb1e76460aa6711e49a5775091 at {username='slave'} at 127.11.42.72:41729: session id = f19cc6cb1e76460aa6711e49a5775091-f3e4c8f98b5e48e3ba6affc2df08943a
I20240502 01:54:14.715469 24733 tablet_copy_client.cc:308] T 1fdd104f1a28446cbc25d4ca16d03d7d P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.66:38531
I20240502 01:54:14.716446 17627 tablet_copy_service.cc:140] P 3fceccefba274b1dbbd8216af8b0bc29: Received BeginTabletCopySession request for tablet 1fdd104f1a28446cbc25d4ca16d03d7d from peer 2f306549407b4bb281ccf2994035e4f1 ({username='slave'} at 127.11.42.71:49447)
I20240502 01:54:14.716560 17627 tablet_copy_service.cc:161] P 3fceccefba274b1dbbd8216af8b0bc29: Beginning new tablet copy session on tablet 1fdd104f1a28446cbc25d4ca16d03d7d from peer 2f306549407b4bb281ccf2994035e4f1 at {username='slave'} at 127.11.42.71:49447: session id = 2f306549407b4bb281ccf2994035e4f1-1fdd104f1a28446cbc25d4ca16d03d7d
I20240502 01:54:14.716862 20253 tablet_copy_source_session.cc:205] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:14.718905 17627 tablet_copy_source_session.cc:205] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:14.720674 24733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fdd104f1a28446cbc25d4ca16d03d7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:14.720837 24727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3e4c8f98b5e48e3ba6affc2df08943a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:14.727890 24733 tablet_copy_client.cc:783] T 1fdd104f1a28446cbc25d4ca16d03d7d P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:14.728132 24733 tablet_copy_client.cc:647] T 1fdd104f1a28446cbc25d4ca16d03d7d P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:14.729611 24726 ts_tablet_manager.cc:1888] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 01:54:14.729717 24726 log.cc:1196] T 5df83f83d2674155afba93c2d992af0a P dfa1152987644bd781b46c7e42a7a685: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/wals/5df83f83d2674155afba93c2d992af0a
I20240502 01:54:14.730423 17321 catalog_manager.cc:4830] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): tablet 5df83f83d2674155afba93c2d992af0a (table rebalance_test_table_7 [id=c6c3a53b24874f31aa38118254253dcb]) successfully deleted
I20240502 01:54:14.731496 24733 tablet_copy_client.cc:523] T 1fdd104f1a28446cbc25d4ca16d03d7d P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:14.732169 24727 tablet_copy_client.cc:783] T f3e4c8f98b5e48e3ba6affc2df08943a P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:14.732445 24727 tablet_copy_client.cc:647] T f3e4c8f98b5e48e3ba6affc2df08943a P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:14.734306 24727 tablet_copy_client.cc:523] T f3e4c8f98b5e48e3ba6affc2df08943a P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:14.736415 24733 tablet_bootstrap.cc:492] T 1fdd104f1a28446cbc25d4ca16d03d7d P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:54:14.738101 24727 tablet_bootstrap.cc:492] T f3e4c8f98b5e48e3ba6affc2df08943a P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:54:14.738270 17334 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet dc5b634d198f4408b30d859addd79de3
I20240502 01:54:14.740346 20211 tablet_service.cc:1514] Processing DeleteTablet for tablet dc5b634d198f4408b30d859addd79de3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:14 UTC) from {username='slave'} at 127.0.0.1:44624
I20240502 01:54:14.750867 20609 tablet_service.cc:1514] Processing DeleteTablet for tablet dc5b634d198f4408b30d859addd79de3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:14 UTC) from {username='slave'} at 127.0.0.1:33698
I20240502 01:54:14.751143 24730 tablet_replica.cc:330] T dc5b634d198f4408b30d859addd79de3 P f19cc6cb1e76460aa6711e49a5775091: stopping tablet replica
I20240502 01:54:14.751351 24730 raft_consensus.cc:2229] T dc5b634d198f4408b30d859addd79de3 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:14.751549 24730 raft_consensus.cc:2258] T dc5b634d198f4408b30d859addd79de3 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:14.752425 24730 ts_tablet_manager.cc:1875] T dc5b634d198f4408b30d859addd79de3 P f19cc6cb1e76460aa6711e49a5775091: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:14.754091 17716 tablet_service.cc:1514] Processing DeleteTablet for tablet dc5b634d198f4408b30d859addd79de3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:14 UTC) from {username='slave'} at 127.0.0.1:52198
I20240502 01:54:14.754307 24726 tablet_replica.cc:330] T dc5b634d198f4408b30d859addd79de3 P dfa1152987644bd781b46c7e42a7a685: stopping tablet replica
I20240502 01:54:14.754457 24726 raft_consensus.cc:2229] T dc5b634d198f4408b30d859addd79de3 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:14.756659 24726 raft_consensus.cc:2258] T dc5b634d198f4408b30d859addd79de3 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:14.759397 24730 ts_tablet_manager.cc:1888] T dc5b634d198f4408b30d859addd79de3 P f19cc6cb1e76460aa6711e49a5775091: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:14.759552 24730 log.cc:1196] T dc5b634d198f4408b30d859addd79de3 P f19cc6cb1e76460aa6711e49a5775091: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal/wals/dc5b634d198f4408b30d859addd79de3
I20240502 01:54:14.760041 24730 ts_tablet_manager.cc:1909] T dc5b634d198f4408b30d859addd79de3 P f19cc6cb1e76460aa6711e49a5775091: Deleting consensus metadata
I20240502 01:54:14.761045 17321 catalog_manager.cc:4830] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): tablet dc5b634d198f4408b30d859addd79de3 (table rebalancer_dynamic_table_new_name [id=6dc57802503d4ec5b7d530db90daec99]) successfully deleted
I20240502 01:54:14.761536 24742 tablet_replica.cc:330] T dc5b634d198f4408b30d859addd79de3 P fdc2ac372492454c85387589a91f63af: stopping tablet replica
I20240502 01:54:14.761849 24742 raft_consensus.cc:2229] T dc5b634d198f4408b30d859addd79de3 P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:14.761874 24726 ts_tablet_manager.cc:1875] T dc5b634d198f4408b30d859addd79de3 P dfa1152987644bd781b46c7e42a7a685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:14.763231 24742 raft_consensus.cc:2258] T dc5b634d198f4408b30d859addd79de3 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:14.764168 24742 ts_tablet_manager.cc:1875] T dc5b634d198f4408b30d859addd79de3 P fdc2ac372492454c85387589a91f63af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:14.782877 24746 ts_tablet_manager.cc:902] T ee418fc4028a422eabac02ed63ea9047 P bb31bceba76d466db24b9341c04900c3: Initiating tablet copy from peer 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:54:14.792047 24726 ts_tablet_manager.cc:1888] T dc5b634d198f4408b30d859addd79de3 P dfa1152987644bd781b46c7e42a7a685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:14.795696 24726 log.cc:1196] T dc5b634d198f4408b30d859addd79de3 P dfa1152987644bd781b46c7e42a7a685: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/wals/dc5b634d198f4408b30d859addd79de3
I20240502 01:54:14.796408 24726 ts_tablet_manager.cc:1909] T dc5b634d198f4408b30d859addd79de3 P dfa1152987644bd781b46c7e42a7a685: Deleting consensus metadata
I20240502 01:54:14.798220 24742 ts_tablet_manager.cc:1888] T dc5b634d198f4408b30d859addd79de3 P fdc2ac372492454c85387589a91f63af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:14.799903 17321 catalog_manager.cc:4830] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): tablet dc5b634d198f4408b30d859addd79de3 (table rebalancer_dynamic_table_new_name [id=6dc57802503d4ec5b7d530db90daec99]) successfully deleted
I20240502 01:54:14.800484 24727 tablet_bootstrap.cc:492] T f3e4c8f98b5e48e3ba6affc2df08943a P f19cc6cb1e76460aa6711e49a5775091: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:14.800804 24742 log.cc:1196] T dc5b634d198f4408b30d859addd79de3 P fdc2ac372492454c85387589a91f63af: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal/wals/dc5b634d198f4408b30d859addd79de3
I20240502 01:54:14.801254 24727 tablet_bootstrap.cc:492] T f3e4c8f98b5e48e3ba6affc2df08943a P f19cc6cb1e76460aa6711e49a5775091: Bootstrap complete.
I20240502 01:54:14.801487 24727 ts_tablet_manager.cc:1367] T f3e4c8f98b5e48e3ba6affc2df08943a P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.063s user 0.009s sys 0.007s
I20240502 01:54:14.801915 24742 ts_tablet_manager.cc:1909] T dc5b634d198f4408b30d859addd79de3 P fdc2ac372492454c85387589a91f63af: Deleting consensus metadata
I20240502 01:54:14.803011 24727 raft_consensus.cc:348] T f3e4c8f98b5e48e3ba6affc2df08943a P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:54:14.803252 24727 raft_consensus.cc:729] T f3e4c8f98b5e48e3ba6affc2df08943a P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: LEARNER
I20240502 01:54:14.803470 24727 consensus_queue.cc:260] T f3e4c8f98b5e48e3ba6affc2df08943a P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:54:14.804036 24727 ts_tablet_manager.cc:1398] T f3e4c8f98b5e48e3ba6affc2df08943a P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:54:14.808842 20253 tablet_copy_service.cc:342] P fdc2ac372492454c85387589a91f63af: Request end of tablet copy session f19cc6cb1e76460aa6711e49a5775091-f3e4c8f98b5e48e3ba6affc2df08943a received from {username='slave'} at 127.11.42.72:41729
I20240502 01:54:14.809041 20253 tablet_copy_service.cc:433] P fdc2ac372492454c85387589a91f63af: ending tablet copy session f19cc6cb1e76460aa6711e49a5775091-f3e4c8f98b5e48e3ba6affc2df08943a on tablet f3e4c8f98b5e48e3ba6affc2df08943a with peer f19cc6cb1e76460aa6711e49a5775091
I20240502 01:54:14.809551 17322 catalog_manager.cc:4830] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): tablet dc5b634d198f4408b30d859addd79de3 (table rebalancer_dynamic_table_new_name [id=6dc57802503d4ec5b7d530db90daec99]) successfully deleted
I20240502 01:54:14.827236 24746 tablet_copy_client.cc:308] T ee418fc4028a422eabac02ed63ea9047 P bb31bceba76d466db24b9341c04900c3: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.71:38623
I20240502 01:54:14.828068 20518 tablet_copy_service.cc:140] P 2f306549407b4bb281ccf2994035e4f1: Received BeginTabletCopySession request for tablet ee418fc4028a422eabac02ed63ea9047 from peer bb31bceba76d466db24b9341c04900c3 ({username='slave'} at 127.11.42.68:49559)
I20240502 01:54:14.828241 20518 tablet_copy_service.cc:161] P 2f306549407b4bb281ccf2994035e4f1: Beginning new tablet copy session on tablet ee418fc4028a422eabac02ed63ea9047 from peer bb31bceba76d466db24b9341c04900c3 at {username='slave'} at 127.11.42.68:49559: session id = bb31bceba76d466db24b9341c04900c3-ee418fc4028a422eabac02ed63ea9047
I20240502 01:54:14.831477 20518 tablet_copy_source_session.cc:205] T ee418fc4028a422eabac02ed63ea9047 P 2f306549407b4bb281ccf2994035e4f1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:14.834796 24746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee418fc4028a422eabac02ed63ea9047. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:14.841022 24746 tablet_copy_client.cc:783] T ee418fc4028a422eabac02ed63ea9047 P bb31bceba76d466db24b9341c04900c3: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:14.841243 24746 tablet_copy_client.cc:647] T ee418fc4028a422eabac02ed63ea9047 P bb31bceba76d466db24b9341c04900c3: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:14.848881 17334 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_extra_table_50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:14.859431 24746 tablet_copy_client.cc:523] T ee418fc4028a422eabac02ed63ea9047 P bb31bceba76d466db24b9341c04900c3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:14.861294 24733 tablet_bootstrap.cc:492] T 1fdd104f1a28446cbc25d4ca16d03d7d P 2f306549407b4bb281ccf2994035e4f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:14.861714 24733 tablet_bootstrap.cc:492] T 1fdd104f1a28446cbc25d4ca16d03d7d P 2f306549407b4bb281ccf2994035e4f1: Bootstrap complete.
I20240502 01:54:14.861982 24733 ts_tablet_manager.cc:1367] T 1fdd104f1a28446cbc25d4ca16d03d7d P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.126s user 0.049s sys 0.000s
I20240502 01:54:14.863344 20345 tablet_service.cc:1467] Processing CreateTablet for tablet b9d82510030b4503ac892175a8bb1bb7 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=b994be1897424e729195614225fa1ffb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:14.863785 20345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9d82510030b4503ac892175a8bb1bb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:14.865813 17716 tablet_service.cc:1467] Processing CreateTablet for tablet b9d82510030b4503ac892175a8bb1bb7 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=b994be1897424e729195614225fa1ffb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:14.866209 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9d82510030b4503ac892175a8bb1bb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:14.867410 24746 tablet_bootstrap.cc:492] T ee418fc4028a422eabac02ed63ea9047 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:54:14.868145 17334 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:54:14.869417 20609 tablet_service.cc:1467] Processing CreateTablet for tablet b9d82510030b4503ac892175a8bb1bb7 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=b994be1897424e729195614225fa1ffb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:14.869803 20609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9d82510030b4503ac892175a8bb1bb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:14.873097 17585 tablet_service.cc:1467] Processing CreateTablet for tablet 049f5bd8759c488789b7c396985be3f9 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=b994be1897424e729195614225fa1ffb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:14.873478 17585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 049f5bd8759c488789b7c396985be3f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:14.873555 20610 tablet_service.cc:1467] Processing CreateTablet for tablet 049f5bd8759c488789b7c396985be3f9 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=b994be1897424e729195614225fa1ffb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:14.873942 20610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 049f5bd8759c488789b7c396985be3f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:14.877612 20211 tablet_service.cc:1467] Processing CreateTablet for tablet 049f5bd8759c488789b7c396985be3f9 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=b994be1897424e729195614225fa1ffb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:14.878008 20211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 049f5bd8759c488789b7c396985be3f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:14.879319 24733 raft_consensus.cc:348] T 1fdd104f1a28446cbc25d4ca16d03d7d P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
I20240502 01:54:14.879608 24733 raft_consensus.cc:729] T 1fdd104f1a28446cbc25d4ca16d03d7d P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: LEARNER
I20240502 01:54:14.879806 24733 consensus_queue.cc:260] T 1fdd104f1a28446cbc25d4ca16d03d7d P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
I20240502 01:54:14.881227 24733 ts_tablet_manager.cc:1398] T 1fdd104f1a28446cbc25d4ca16d03d7d P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240502 01:54:14.882512 24757 tablet_bootstrap.cc:492] T 049f5bd8759c488789b7c396985be3f9 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:54:14.886999 24757 tablet_bootstrap.cc:654] T 049f5bd8759c488789b7c396985be3f9 P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:14.888739 17627 tablet_copy_service.cc:342] P 3fceccefba274b1dbbd8216af8b0bc29: Request end of tablet copy session 2f306549407b4bb281ccf2994035e4f1-1fdd104f1a28446cbc25d4ca16d03d7d received from {username='slave'} at 127.11.42.71:49447
I20240502 01:54:14.888842 17627 tablet_copy_service.cc:433] P 3fceccefba274b1dbbd8216af8b0bc29: ending tablet copy session 2f306549407b4bb281ccf2994035e4f1-1fdd104f1a28446cbc25d4ca16d03d7d on tablet 1fdd104f1a28446cbc25d4ca16d03d7d with peer 2f306549407b4bb281ccf2994035e4f1
I20240502 01:54:14.902485 20346 tablet_service.cc:1467] Processing CreateTablet for tablet 52b83e43cae646a29bbf3b826fbbcfb6 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=b994be1897424e729195614225fa1ffb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:14.902935 20346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52b83e43cae646a29bbf3b826fbbcfb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:14.906313 17847 tablet_service.cc:1467] Processing CreateTablet for tablet 52b83e43cae646a29bbf3b826fbbcfb6 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=b994be1897424e729195614225fa1ffb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:14.906713 17847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52b83e43cae646a29bbf3b826fbbcfb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:14.909922 24762 tablet_bootstrap.cc:492] T b9d82510030b4503ac892175a8bb1bb7 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:54:14.910542 20212 tablet_service.cc:1467] Processing CreateTablet for tablet 52b83e43cae646a29bbf3b826fbbcfb6 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=b994be1897424e729195614225fa1ffb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:14.910972 20212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52b83e43cae646a29bbf3b826fbbcfb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:14.915230 24759 tablet_bootstrap.cc:492] T 049f5bd8759c488789b7c396985be3f9 P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:54:14.918507 24762 tablet_bootstrap.cc:654] T b9d82510030b4503ac892175a8bb1bb7 P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:14.920105 24763 tablet_bootstrap.cc:492] T 52b83e43cae646a29bbf3b826fbbcfb6 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:54:14.927605 17456 tablet_service.cc:1467] Processing CreateTablet for tablet 4ecf3b8482d04b37baf761305f933afb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4193c1e64b834e889d5218c65ef0bba3]), partition=
I20240502 01:54:14.927963 17456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ecf3b8482d04b37baf761305f933afb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:14.928298 24763 tablet_bootstrap.cc:654] T 52b83e43cae646a29bbf3b826fbbcfb6 P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:14.937003 24760 tablet_bootstrap.cc:492] T b9d82510030b4503ac892175a8bb1bb7 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:54:14.942833 24761 tablet_bootstrap.cc:492] T 049f5bd8759c488789b7c396985be3f9 P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:54:14.945674 24767 tablet_bootstrap.cc:492] T 4ecf3b8482d04b37baf761305f933afb P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:54:14.947060 17847 tablet_service.cc:1467] Processing CreateTablet for tablet 4ecf3b8482d04b37baf761305f933afb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4193c1e64b834e889d5218c65ef0bba3]), partition=
I20240502 01:54:14.947289 24767 tablet_bootstrap.cc:654] T 4ecf3b8482d04b37baf761305f933afb P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:14.947558 17847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ecf3b8482d04b37baf761305f933afb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:14.952598 24759 tablet_bootstrap.cc:654] T 049f5bd8759c488789b7c396985be3f9 P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:14.956897 24767 tablet_bootstrap.cc:492] T 4ecf3b8482d04b37baf761305f933afb P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:54:14.957721 24767 ts_tablet_manager.cc:1367] T 4ecf3b8482d04b37baf761305f933afb P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20240502 01:54:14.958649 24757 tablet_bootstrap.cc:492] T 049f5bd8759c488789b7c396985be3f9 P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:54:14.958798 24757 ts_tablet_manager.cc:1367] T 049f5bd8759c488789b7c396985be3f9 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.076s user 0.001s sys 0.013s
I20240502 01:54:14.958570 24767 raft_consensus.cc:348] T 4ecf3b8482d04b37baf761305f933afb P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:14.959651 24757 raft_consensus.cc:348] T 049f5bd8759c488789b7c396985be3f9 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:14.959846 24757 raft_consensus.cc:374] T 049f5bd8759c488789b7c396985be3f9 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:14.959913 24757 raft_consensus.cc:729] T 049f5bd8759c488789b7c396985be3f9 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:54:14.959882 24767 raft_consensus.cc:374] T 4ecf3b8482d04b37baf761305f933afb P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:14.960140 24767 raft_consensus.cc:729] T 4ecf3b8482d04b37baf761305f933afb P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:54:14.960134 24757 consensus_queue.cc:260] T 049f5bd8759c488789b7c396985be3f9 P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:14.960763 24757 ts_tablet_manager.cc:1398] T 049f5bd8759c488789b7c396985be3f9 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:54:14.961282 24764 ts_tablet_manager.cc:902] T 29d8cb76dc514e7b9002bd2291d8ee8d P fdc2ac372492454c85387589a91f63af: Initiating tablet copy from peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:54:14.962200 24767 consensus_queue.cc:260] T 4ecf3b8482d04b37baf761305f933afb P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:14.962816 24759 tablet_bootstrap.cc:492] T 049f5bd8759c488789b7c396985be3f9 P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:54:14.963243 24767 ts_tablet_manager.cc:1398] T 4ecf3b8482d04b37baf761305f933afb P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 01:54:14.965099 24759 ts_tablet_manager.cc:1367] T 049f5bd8759c488789b7c396985be3f9 P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.050s user 0.004s sys 0.000s
I20240502 01:54:14.966478 24759 raft_consensus.cc:348] T 049f5bd8759c488789b7c396985be3f9 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:14.966758 24759 raft_consensus.cc:374] T 049f5bd8759c488789b7c396985be3f9 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:14.966907 24759 raft_consensus.cc:729] T 049f5bd8759c488789b7c396985be3f9 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:54:14.967192 24759 consensus_queue.cc:260] T 049f5bd8759c488789b7c396985be3f9 P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:14.967834 24759 ts_tablet_manager.cc:1398] T 049f5bd8759c488789b7c396985be3f9 P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:14.968464 24746 tablet_bootstrap.cc:492] T ee418fc4028a422eabac02ed63ea9047 P bb31bceba76d466db24b9341c04900c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:14.968909 24746 tablet_bootstrap.cc:492] T ee418fc4028a422eabac02ed63ea9047 P bb31bceba76d466db24b9341c04900c3: Bootstrap complete.
I20240502 01:54:14.973368 24746 ts_tablet_manager.cc:1367] T ee418fc4028a422eabac02ed63ea9047 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.106s user 0.025s sys 0.000s
I20240502 01:54:14.975014 24746 raft_consensus.cc:348] T ee418fc4028a422eabac02ed63ea9047 P bb31bceba76d466db24b9341c04900c3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } }
I20240502 01:54:14.975503 24746 raft_consensus.cc:729] T ee418fc4028a422eabac02ed63ea9047 P bb31bceba76d466db24b9341c04900c3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: LEARNER
I20240502 01:54:14.975881 24746 consensus_queue.cc:260] T ee418fc4028a422eabac02ed63ea9047 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } }
I20240502 01:54:14.983539 24761 tablet_bootstrap.cc:654] T 049f5bd8759c488789b7c396985be3f9 P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:14.976930 20610 tablet_service.cc:1467] Processing CreateTablet for tablet 4ecf3b8482d04b37baf761305f933afb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4193c1e64b834e889d5218c65ef0bba3]), partition=
I20240502 01:54:14.985154 20610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ecf3b8482d04b37baf761305f933afb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:14.990231 24762 tablet_bootstrap.cc:492] T b9d82510030b4503ac892175a8bb1bb7 P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:54:14.990434 24762 ts_tablet_manager.cc:1367] T b9d82510030b4503ac892175a8bb1bb7 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.081s user 0.012s sys 0.024s
I20240502 01:54:14.991196 24762 raft_consensus.cc:348] T b9d82510030b4503ac892175a8bb1bb7 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:14.991461 24762 raft_consensus.cc:374] T b9d82510030b4503ac892175a8bb1bb7 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:14.991573 24762 raft_consensus.cc:729] T b9d82510030b4503ac892175a8bb1bb7 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:54:14.991623 20625 consensus_queue.cc:237] T 4aefc61cb95f4e949e7a23a1c677749e P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:14.991827 24762 consensus_queue.cc:260] T b9d82510030b4503ac892175a8bb1bb7 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:14.993875 24762 ts_tablet_manager.cc:1398] T b9d82510030b4503ac892175a8bb1bb7 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 01:54:14.994099 17737 raft_consensus.cc:1261] T 4aefc61cb95f4e949e7a23a1c677749e P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:54:14.995055 24127 consensus_queue.cc:1035] T 4aefc61cb95f4e949e7a23a1c677749e P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:54:14.997895 17864 raft_consensus.cc:1261] T 4aefc61cb95f4e949e7a23a1c677749e P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:54:14.998502 24127 consensus_queue.cc:1035] T 4aefc61cb95f4e949e7a23a1c677749e P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:54:14.999698 24127 raft_consensus.cc:2932] T 4aefc61cb95f4e949e7a23a1c677749e P f19cc6cb1e76460aa6711e49a5775091 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:15.002354 24760 tablet_bootstrap.cc:654] T b9d82510030b4503ac892175a8bb1bb7 P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:15.005190 17737 raft_consensus.cc:2932] T 4aefc61cb95f4e949e7a23a1c677749e P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:15.009383 17864 raft_consensus.cc:2932] T 4aefc61cb95f4e949e7a23a1c677749e P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:15.014014 20232 raft_consensus.cc:1261] T 1b14ac3413004127b72904b631ec1b94 P fdc2ac372492454c85387589a91f63af [term 2 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:54:15.014955 24476 raft_consensus.cc:482] T 4ecf3b8482d04b37baf761305f933afb P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:15.015093 24476 raft_consensus.cc:504] T 4ecf3b8482d04b37baf761305f933afb P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:15.015724 24476 leader_election.cc:288] T 4ecf3b8482d04b37baf761305f933afb P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:54:15.015828 17334 catalog_manager.cc:5484] T 4aefc61cb95f4e949e7a23a1c677749e P bb31bceba76d466db24b9341c04900c3 reported cstate change: config changed from index 8 to 10, VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) evicted. New cstate: current_term: 2 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:15.016460 24759 tablet_bootstrap.cc:492] T 52b83e43cae646a29bbf3b826fbbcfb6 P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:54:15.017062 24364 consensus_queue.cc:1035] T 1b14ac3413004127b72904b631ec1b94 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:54:15.018226 17411 leader_election.cc:341] T 4ecf3b8482d04b37baf761305f933afb P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Not found: Tablet not found: 4ecf3b8482d04b37baf761305f933afb
I20240502 01:54:15.018246 17604 raft_consensus.cc:1261] T 1b14ac3413004127b72904b631ec1b94 P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:54:15.018546 17864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ecf3b8482d04b37baf761305f933afb" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
W20240502 01:54:15.019124 17412 leader_election.cc:341] T 4ecf3b8482d04b37baf761305f933afb P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:15.019224 17412 leader_election.cc:302] T 4ecf3b8482d04b37baf761305f933afb P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade; no voters: bb31bceba76d466db24b9341c04900c3, f19cc6cb1e76460aa6711e49a5775091
I20240502 01:54:15.019559 24212 raft_consensus.cc:2726] T 4ecf3b8482d04b37baf761305f933afb P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:54:15.019623 24759 tablet_bootstrap.cc:654] T 52b83e43cae646a29bbf3b826fbbcfb6 P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:15.019907 24364 consensus_queue.cc:1035] T 1b14ac3413004127b72904b631ec1b94 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:15.020794 17585 tablet_service.cc:1514] Processing DeleteTablet for tablet 4aefc61cb95f4e949e7a23a1c677749e with delete_type TABLET_DATA_TOMBSTONED (TS 3fceccefba274b1dbbd8216af8b0bc29 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:55232
I20240502 01:54:15.024843 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4aefc61cb95f4e949e7a23a1c677749e with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:15.039242 24746 ts_tablet_manager.cc:1398] T ee418fc4028a422eabac02ed63ea9047 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.066s user 0.001s sys 0.000s
I20240502 01:54:15.042860 24763 tablet_bootstrap.cc:492] T 52b83e43cae646a29bbf3b826fbbcfb6 P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:54:15.043486 24763 ts_tablet_manager.cc:1367] T 52b83e43cae646a29bbf3b826fbbcfb6 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.123s user 0.005s sys 0.006s
I20240502 01:54:15.043790 20518 tablet_copy_service.cc:342] P 2f306549407b4bb281ccf2994035e4f1: Request end of tablet copy session bb31bceba76d466db24b9341c04900c3-ee418fc4028a422eabac02ed63ea9047 received from {username='slave'} at 127.11.42.68:49559
I20240502 01:54:15.043941 20518 tablet_copy_service.cc:433] P 2f306549407b4bb281ccf2994035e4f1: ending tablet copy session bb31bceba76d466db24b9341c04900c3-ee418fc4028a422eabac02ed63ea9047 on tablet ee418fc4028a422eabac02ed63ea9047 with peer bb31bceba76d466db24b9341c04900c3
I20240502 01:54:15.044939 24763 raft_consensus.cc:348] T 52b83e43cae646a29bbf3b826fbbcfb6 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:15.048472 24763 raft_consensus.cc:374] T 52b83e43cae646a29bbf3b826fbbcfb6 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:15.048554 24763 raft_consensus.cc:729] T 52b83e43cae646a29bbf3b826fbbcfb6 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:54:15.049090 24763 consensus_queue.cc:260] T 52b83e43cae646a29bbf3b826fbbcfb6 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:15.056329 24760 tablet_bootstrap.cc:492] T b9d82510030b4503ac892175a8bb1bb7 P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:54:15.056547 24760 ts_tablet_manager.cc:1367] T b9d82510030b4503ac892175a8bb1bb7 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.120s user 0.004s sys 0.002s
I20240502 01:54:15.057407 24760 raft_consensus.cc:348] T b9d82510030b4503ac892175a8bb1bb7 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:15.057842 24763 ts_tablet_manager.cc:1398] T 52b83e43cae646a29bbf3b826fbbcfb6 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240502 01:54:15.058609 24761 tablet_bootstrap.cc:492] T 049f5bd8759c488789b7c396985be3f9 P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:54:15.059242 24761 ts_tablet_manager.cc:1367] T 049f5bd8759c488789b7c396985be3f9 P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.116s user 0.005s sys 0.000s
I20240502 01:54:15.061705 24763 tablet_bootstrap.cc:492] T 4ecf3b8482d04b37baf761305f933afb P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:54:15.063657 24763 tablet_bootstrap.cc:654] T 4ecf3b8482d04b37baf761305f933afb P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:15.064262 24775 tablet_replica.cc:330] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29: stopping tablet replica
I20240502 01:54:15.064597 24775 raft_consensus.cc:2229] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:15.065464 24759 tablet_bootstrap.cc:492] T 52b83e43cae646a29bbf3b826fbbcfb6 P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:54:15.075253 24761 raft_consensus.cc:348] T 049f5bd8759c488789b7c396985be3f9 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:15.075564 24761 raft_consensus.cc:374] T 049f5bd8759c488789b7c396985be3f9 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:15.075717 24761 raft_consensus.cc:729] T 049f5bd8759c488789b7c396985be3f9 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:54:15.076052 24761 consensus_queue.cc:260] T 049f5bd8759c488789b7c396985be3f9 P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:15.077355 24775 raft_consensus.cc:2258] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:15.077913 24759 ts_tablet_manager.cc:1367] T 52b83e43cae646a29bbf3b826fbbcfb6 P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.061s user 0.004s sys 0.000s
I20240502 01:54:15.078473 24775 ts_tablet_manager.cc:1875] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:15.078819 24760 raft_consensus.cc:374] T b9d82510030b4503ac892175a8bb1bb7 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:15.085114 24760 raft_consensus.cc:729] T b9d82510030b4503ac892175a8bb1bb7 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:54:15.085541 24760 consensus_queue.cc:260] T b9d82510030b4503ac892175a8bb1bb7 P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:15.086951 24759 raft_consensus.cc:348] T 52b83e43cae646a29bbf3b826fbbcfb6 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:15.090006 24775 ts_tablet_manager.cc:1888] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 01:54:15.090118 24775 log.cc:1196] T 4aefc61cb95f4e949e7a23a1c677749e P 3fceccefba274b1dbbd8216af8b0bc29: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/wals/4aefc61cb95f4e949e7a23a1c677749e
I20240502 01:54:15.091570 17319 catalog_manager.cc:4830] TS 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): tablet 4aefc61cb95f4e949e7a23a1c677749e (table rebalance_test_table_13 [id=dbf97af96f07462286eb1fabeddf25fa]) successfully deleted
I20240502 01:54:15.092346 24760 ts_tablet_manager.cc:1398] T b9d82510030b4503ac892175a8bb1bb7 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.036s user 0.002s sys 0.000s
I20240502 01:54:15.077064 24761 ts_tablet_manager.cc:1398] T 049f5bd8759c488789b7c396985be3f9 P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20240502 01:54:15.097421 24761 tablet_bootstrap.cc:492] T b9d82510030b4503ac892175a8bb1bb7 P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:54:15.097579 24759 raft_consensus.cc:374] T 52b83e43cae646a29bbf3b826fbbcfb6 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:15.099083 24760 tablet_bootstrap.cc:492] T 52b83e43cae646a29bbf3b826fbbcfb6 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:54:15.100755 24760 tablet_bootstrap.cc:654] T 52b83e43cae646a29bbf3b826fbbcfb6 P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:15.101645 24759 raft_consensus.cc:729] T 52b83e43cae646a29bbf3b826fbbcfb6 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:54:15.101961 24759 consensus_queue.cc:260] T 52b83e43cae646a29bbf3b826fbbcfb6 P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:15.102625 24759 ts_tablet_manager.cc:1398] T 52b83e43cae646a29bbf3b826fbbcfb6 P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.024s user 0.001s sys 0.000s
I20240502 01:54:15.087814 24764 tablet_copy_client.cc:308] T 29d8cb76dc514e7b9002bd2291d8ee8d P fdc2ac372492454c85387589a91f63af: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.67:36115
I20240502 01:54:15.097925 20625 raft_consensus.cc:1261] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091 [term 2 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:54:15.103935 17757 tablet_copy_service.cc:140] P dfa1152987644bd781b46c7e42a7a685: Received BeginTabletCopySession request for tablet 29d8cb76dc514e7b9002bd2291d8ee8d from peer fdc2ac372492454c85387589a91f63af ({username='slave'} at 127.11.42.69:44021)
I20240502 01:54:15.104071 17757 tablet_copy_service.cc:161] P dfa1152987644bd781b46c7e42a7a685: Beginning new tablet copy session on tablet 29d8cb76dc514e7b9002bd2291d8ee8d from peer fdc2ac372492454c85387589a91f63af at {username='slave'} at 127.11.42.69:44021: session id = fdc2ac372492454c85387589a91f63af-29d8cb76dc514e7b9002bd2291d8ee8d
I20240502 01:54:15.105171 24761 tablet_bootstrap.cc:654] T b9d82510030b4503ac892175a8bb1bb7 P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:15.109777 24571 consensus_queue.cc:1035] T 1b14ac3413004127b72904b631ec1b94 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:15.110754 17757 tablet_copy_source_session.cc:205] T 29d8cb76dc514e7b9002bd2291d8ee8d P dfa1152987644bd781b46c7e42a7a685: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:15.112336 24764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29d8cb76dc514e7b9002bd2291d8ee8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:15.113432 24760 tablet_bootstrap.cc:492] T 52b83e43cae646a29bbf3b826fbbcfb6 P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:54:15.113679 24760 ts_tablet_manager.cc:1367] T 52b83e43cae646a29bbf3b826fbbcfb6 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.003s
I20240502 01:54:15.114457 24760 raft_consensus.cc:348] T 52b83e43cae646a29bbf3b826fbbcfb6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:15.115936 24760 raft_consensus.cc:374] T 52b83e43cae646a29bbf3b826fbbcfb6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:15.116065 24760 raft_consensus.cc:729] T 52b83e43cae646a29bbf3b826fbbcfb6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:54:15.117316 17737 consensus_queue.cc:237] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
I20240502 01:54:15.118793 24764 tablet_copy_client.cc:783] T 29d8cb76dc514e7b9002bd2291d8ee8d P fdc2ac372492454c85387589a91f63af: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:15.119004 24764 tablet_copy_client.cc:647] T 29d8cb76dc514e7b9002bd2291d8ee8d P fdc2ac372492454c85387589a91f63af: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:15.120240 24760 consensus_queue.cc:260] T 52b83e43cae646a29bbf3b826fbbcfb6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:15.122649 24760 ts_tablet_manager.cc:1398] T 52b83e43cae646a29bbf3b826fbbcfb6 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240502 01:54:15.125360 24764 tablet_copy_client.cc:523] T 29d8cb76dc514e7b9002bd2291d8ee8d P fdc2ac372492454c85387589a91f63af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:15.128846 24764 tablet_bootstrap.cc:492] T 29d8cb76dc514e7b9002bd2291d8ee8d P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:54:15.131656 24761 tablet_bootstrap.cc:492] T b9d82510030b4503ac892175a8bb1bb7 P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:54:15.131929 24761 ts_tablet_manager.cc:1367] T b9d82510030b4503ac892175a8bb1bb7 P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.035s user 0.001s sys 0.003s
I20240502 01:54:15.132918 24761 raft_consensus.cc:348] T b9d82510030b4503ac892175a8bb1bb7 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:15.133234 24761 raft_consensus.cc:374] T b9d82510030b4503ac892175a8bb1bb7 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:15.133364 24761 raft_consensus.cc:729] T b9d82510030b4503ac892175a8bb1bb7 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:54:15.133677 24761 consensus_queue.cc:260] T b9d82510030b4503ac892175a8bb1bb7 P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:15.134099 24415 raft_consensus.cc:482] T b9d82510030b4503ac892175a8bb1bb7 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240502 01:54:15.134199 17673 consensus_peers.cc:485] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 -> Peer 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): Couldn't send request to peer 2f306549407b4bb281ccf2994035e4f1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 71cbc012efe14fdb937302796a9f84e5. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:15.134208 24415 raft_consensus.cc:504] T b9d82510030b4503ac892175a8bb1bb7 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:15.134871 24415 leader_election.cc:288] T b9d82510030b4503ac892175a8bb1bb7 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:54:15.138957 20362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d82510030b4503ac892175a8bb1bb7" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:54:15.139154 20362 raft_consensus.cc:2454] T b9d82510030b4503ac892175a8bb1bb7 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 0.
I20240502 01:54:15.139182 24763 tablet_bootstrap.cc:492] T 4ecf3b8482d04b37baf761305f933afb P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:54:15.139348 24763 ts_tablet_manager.cc:1367] T 4ecf3b8482d04b37baf761305f933afb P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.078s user 0.007s sys 0.002s
I20240502 01:54:15.139521 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d82510030b4503ac892175a8bb1bb7" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:54:15.139734 17735 raft_consensus.cc:2454] T b9d82510030b4503ac892175a8bb1bb7 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 0.
I20240502 01:54:15.139765 17475 raft_consensus.cc:1261] T 71cbc012efe14fdb937302796a9f84e5 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:15.140064 24763 raft_consensus.cc:348] T 4ecf3b8482d04b37baf761305f933afb P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:15.140255 24763 raft_consensus.cc:374] T 4ecf3b8482d04b37baf761305f933afb P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:15.140340 24763 raft_consensus.cc:729] T 4ecf3b8482d04b37baf761305f933afb P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:54:15.140550 24364 consensus_queue.cc:1035] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:15.144691 23611 raft_consensus.cc:482] T 52b83e43cae646a29bbf3b826fbbcfb6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:15.144840 23611 raft_consensus.cc:504] T 52b83e43cae646a29bbf3b826fbbcfb6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:15.145076 24763 consensus_queue.cc:260] T 4ecf3b8482d04b37baf761305f933afb P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:15.145431 23611 leader_election.cc:288] T 52b83e43cae646a29bbf3b826fbbcfb6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:54:15.146186 20564 leader_election.cc:302] T b9d82510030b4503ac892175a8bb1bb7 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:54:15.147011 20232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52b83e43cae646a29bbf3b826fbbcfb6" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:54:15.147214 20232 raft_consensus.cc:2454] T 52b83e43cae646a29bbf3b826fbbcfb6 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 0.
I20240502 01:54:15.147698 20303 leader_election.cc:302] T 52b83e43cae646a29bbf3b826fbbcfb6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:54:15.147861 17864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52b83e43cae646a29bbf3b826fbbcfb6" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:54:15.147965 23908 raft_consensus.cc:2781] T 52b83e43cae646a29bbf3b826fbbcfb6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:15.148057 23908 raft_consensus.cc:482] T 52b83e43cae646a29bbf3b826fbbcfb6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:15.148065 17864 raft_consensus.cc:2454] T 52b83e43cae646a29bbf3b826fbbcfb6 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 0.
I20240502 01:54:15.148135 23908 raft_consensus.cc:3037] T 52b83e43cae646a29bbf3b826fbbcfb6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:15.148372 24763 ts_tablet_manager.cc:1398] T 4ecf3b8482d04b37baf761305f933afb P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 01:54:15.149360 17736 consensus_queue.cc:237] T 1b14ac3413004127b72904b631ec1b94 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:15.150789 23908 raft_consensus.cc:504] T 52b83e43cae646a29bbf3b826fbbcfb6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:15.151453 23908 leader_election.cc:288] T 52b83e43cae646a29bbf3b826fbbcfb6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Requested vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:54:15.152101 24125 raft_consensus.cc:2781] T b9d82510030b4503ac892175a8bb1bb7 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:15.152207 24125 raft_consensus.cc:482] T b9d82510030b4503ac892175a8bb1bb7 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:15.152316 24125 raft_consensus.cc:3037] T b9d82510030b4503ac892175a8bb1bb7 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:15.155530 20231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52b83e43cae646a29bbf3b826fbbcfb6" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:54:15.155714 20231 raft_consensus.cc:3037] T 52b83e43cae646a29bbf3b826fbbcfb6 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:15.156422 17604 raft_consensus.cc:1261] T 1b14ac3413004127b72904b631ec1b94 P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:54:15.158408 20231 raft_consensus.cc:2454] T 52b83e43cae646a29bbf3b826fbbcfb6 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 1.
I20240502 01:54:15.159571 17867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52b83e43cae646a29bbf3b826fbbcfb6" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:54:15.159749 17867 raft_consensus.cc:3037] T 52b83e43cae646a29bbf3b826fbbcfb6 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:15.160730 20232 raft_consensus.cc:1261] T 1b14ac3413004127b72904b631ec1b94 P fdc2ac372492454c85387589a91f63af [term 2 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:54:15.160888 24764 tablet_bootstrap.cc:492] T 29d8cb76dc514e7b9002bd2291d8ee8d P fdc2ac372492454c85387589a91f63af: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:15.161516 24764 tablet_bootstrap.cc:492] T 29d8cb76dc514e7b9002bd2291d8ee8d P fdc2ac372492454c85387589a91f63af: Bootstrap complete.
I20240502 01:54:15.161748 24764 ts_tablet_manager.cc:1367] T 29d8cb76dc514e7b9002bd2291d8ee8d P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.033s user 0.010s sys 0.001s
I20240502 01:54:15.161756 20362 raft_consensus.cc:1261] T 71cbc012efe14fdb937302796a9f84e5 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:15.162292 17867 raft_consensus.cc:2454] T 52b83e43cae646a29bbf3b826fbbcfb6 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 1.
I20240502 01:54:15.162292 20303 leader_election.cc:302] T 52b83e43cae646a29bbf3b826fbbcfb6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:54:15.162557 23611 raft_consensus.cc:2781] T 52b83e43cae646a29bbf3b826fbbcfb6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:15.162691 23611 raft_consensus.cc:686] T 52b83e43cae646a29bbf3b826fbbcfb6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Becoming Leader. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Running, Role: LEADER
I20240502 01:54:15.162616 24764 raft_consensus.cc:348] T 29d8cb76dc514e7b9002bd2291d8ee8d P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } }
I20240502 01:54:15.162698 24203 consensus_queue.cc:1035] T 1b14ac3413004127b72904b631ec1b94 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:15.162860 24764 raft_consensus.cc:729] T 29d8cb76dc514e7b9002bd2291d8ee8d P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: LEARNER
I20240502 01:54:15.162914 23611 consensus_queue.cc:237] T 52b83e43cae646a29bbf3b826fbbcfb6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:15.163115 24203 consensus_queue.cc:1035] T 1b14ac3413004127b72904b631ec1b94 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:15.163069 24764 consensus_queue.cc:260] T 29d8cb76dc514e7b9002bd2291d8ee8d P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } }
I20240502 01:54:15.165052 24764 ts_tablet_manager.cc:1398] T 29d8cb76dc514e7b9002bd2291d8ee8d P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:15.173540 17604 raft_consensus.cc:2932] T 1b14ac3413004127b72904b631ec1b94 P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:15.175565 20232 raft_consensus.cc:2932] T 1b14ac3413004127b72904b631ec1b94 P fdc2ac372492454c85387589a91f63af [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:15.179656 24125 raft_consensus.cc:504] T b9d82510030b4503ac892175a8bb1bb7 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:15.180624 24125 leader_election.cc:288] T b9d82510030b4503ac892175a8bb1bb7 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 election: Requested vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:54:15.182199 24203 consensus_queue.cc:1035] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:15.183362 24761 ts_tablet_manager.cc:1398] T b9d82510030b4503ac892175a8bb1bb7 P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.051s user 0.000s sys 0.002s
I20240502 01:54:15.183434 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d82510030b4503ac892175a8bb1bb7" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:54:15.183378 24203 raft_consensus.cc:2932] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:54:15.185981 20362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d82510030b4503ac892175a8bb1bb7" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
I20240502 01:54:15.186187 20362 raft_consensus.cc:3037] T b9d82510030b4503ac892175a8bb1bb7 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:15.186364 17757 tablet_copy_service.cc:342] P dfa1152987644bd781b46c7e42a7a685: Request end of tablet copy session fdc2ac372492454c85387589a91f63af-29d8cb76dc514e7b9002bd2291d8ee8d received from {username='slave'} at 127.11.42.69:44021
I20240502 01:54:15.186455 17757 tablet_copy_service.cc:433] P dfa1152987644bd781b46c7e42a7a685: ending tablet copy session fdc2ac372492454c85387589a91f63af-29d8cb76dc514e7b9002bd2291d8ee8d on tablet 29d8cb76dc514e7b9002bd2291d8ee8d with peer fdc2ac372492454c85387589a91f63af
I20240502 01:54:15.188153 17326 catalog_manager.cc:5484] T 1b14ac3413004127b72904b631ec1b94 P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: config changed from index 3 to 5, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New cstate: current_term: 2 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:15.189078 20362 raft_consensus.cc:2454] T b9d82510030b4503ac892175a8bb1bb7 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 1.
I20240502 01:54:15.189822 24761 tablet_bootstrap.cc:492] T 4ecf3b8482d04b37baf761305f933afb P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:54:15.190537 20564 leader_election.cc:302] T b9d82510030b4503ac892175a8bb1bb7 P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:54:15.191112 24127 raft_consensus.cc:2781] T b9d82510030b4503ac892175a8bb1bb7 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:15.191313 24127 raft_consensus.cc:686] T b9d82510030b4503ac892175a8bb1bb7 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Becoming Leader. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Running, Role: LEADER
I20240502 01:54:15.191596 24127 consensus_queue.cc:237] T b9d82510030b4503ac892175a8bb1bb7 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:15.192847 20610 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b14ac3413004127b72904b631ec1b94 with delete_type TABLET_DATA_TOMBSTONED (TS f19cc6cb1e76460aa6711e49a5775091 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33698
I20240502 01:54:15.194020 17475 raft_consensus.cc:2932] T 71cbc012efe14fdb937302796a9f84e5 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:54:15.200289 23967 raft_consensus.cc:2932] T 1b14ac3413004127b72904b631ec1b94 P dfa1152987644bd781b46c7e42a7a685 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:15.200748 17326 catalog_manager.cc:5484] T 71cbc012efe14fdb937302796a9f84e5 P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: config changed from index 6 to 7, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:54:15.205103 24730 tablet_replica.cc:330] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091: stopping tablet replica
I20240502 01:54:15.205370 24730 raft_consensus.cc:2229] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:15.205564 24730 raft_consensus.cc:2258] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:15.206552 24730 ts_tablet_manager.cc:1875] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:15.214182 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1b14ac3413004127b72904b631ec1b94 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:15.218256 24730 ts_tablet_manager.cc:1888] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 01:54:15.218380 17326 catalog_manager.cc:5484] T b9d82510030b4503ac892175a8bb1bb7 P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: term changed from 0 to 1, leader changed from <none> to f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72). New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:15.220248 24730 log.cc:1196] T 1b14ac3413004127b72904b631ec1b94 P f19cc6cb1e76460aa6711e49a5775091: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal/wals/1b14ac3413004127b72904b631ec1b94
I20240502 01:54:15.221143 17321 catalog_manager.cc:4830] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): tablet 1b14ac3413004127b72904b631ec1b94 (table rebalancer_extra_table_10 [id=96aabcc026484fffa9ea0c0dc8d37340]) successfully deleted
I20240502 01:54:15.226373 24761 tablet_bootstrap.cc:654] T 4ecf3b8482d04b37baf761305f933afb P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:15.235891 20498 raft_consensus.cc:1203] T 1fdd104f1a28446cbc25d4ca16d03d7d P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 01:54:15.240854 24761 tablet_bootstrap.cc:492] T 4ecf3b8482d04b37baf761305f933afb P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:54:15.241267 24761 ts_tablet_manager.cc:1367] T 4ecf3b8482d04b37baf761305f933afb P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.052s user 0.000s sys 0.003s
I20240502 01:54:15.241997 24761 raft_consensus.cc:348] T 4ecf3b8482d04b37baf761305f933afb P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:15.242264 24761 raft_consensus.cc:374] T 4ecf3b8482d04b37baf761305f933afb P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:15.242420 24761 raft_consensus.cc:729] T 4ecf3b8482d04b37baf761305f933afb P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:54:15.242722 24761 consensus_queue.cc:260] T 4ecf3b8482d04b37baf761305f933afb P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:15.247905 17326 catalog_manager.cc:5484] T 52b83e43cae646a29bbf3b826fbbcfb6 P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70). New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:15.248950 24761 ts_tablet_manager.cc:1398] T 4ecf3b8482d04b37baf761305f933afb P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240502 01:54:15.256536 20362 raft_consensus.cc:2932] T 71cbc012efe14fdb937302796a9f84e5 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:54:15.287364 17867 raft_consensus.cc:1203] T ee418fc4028a422eabac02ed63ea9047 P bb31bceba76d466db24b9341c04900c3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:15.294106 24598 raft_consensus.cc:482] T 049f5bd8759c488789b7c396985be3f9 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:15.294260 24598 raft_consensus.cc:504] T 049f5bd8759c488789b7c396985be3f9 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:15.294910 24598 leader_election.cc:288] T 049f5bd8759c488789b7c396985be3f9 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:54:15.295923 20625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "049f5bd8759c488789b7c396985be3f9" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091" is_pre_election: true
I20240502 01:54:15.296133 20625 raft_consensus.cc:2454] T 049f5bd8759c488789b7c396985be3f9 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:54:15.296597 17542 leader_election.cc:302] T 049f5bd8759c488789b7c396985be3f9 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:54:15.296895 24598 raft_consensus.cc:2781] T 049f5bd8759c488789b7c396985be3f9 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:15.296958 24598 raft_consensus.cc:482] T 049f5bd8759c488789b7c396985be3f9 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:15.297027 24598 raft_consensus.cc:3037] T 049f5bd8759c488789b7c396985be3f9 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:15.303257 24598 raft_consensus.cc:504] T 049f5bd8759c488789b7c396985be3f9 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:15.305716 20625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "049f5bd8759c488789b7c396985be3f9" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091"
I20240502 01:54:15.305898 20625 raft_consensus.cc:3037] T 049f5bd8759c488789b7c396985be3f9 P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:15.310081 20625 raft_consensus.cc:2454] T 049f5bd8759c488789b7c396985be3f9 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:54:15.310487 17542 leader_election.cc:302] T 049f5bd8759c488789b7c396985be3f9 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:54:15.310714 24735 raft_consensus.cc:2781] T 049f5bd8759c488789b7c396985be3f9 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:15.313150 20627 raft_consensus.cc:1203] T f3e4c8f98b5e48e3ba6affc2df08943a P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:15.319079 20232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "049f5bd8759c488789b7c396985be3f9" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:54:15.319317 20232 raft_consensus.cc:2454] T 049f5bd8759c488789b7c396985be3f9 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:54:15.323015 24598 leader_election.cc:288] T 049f5bd8759c488789b7c396985be3f9 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Requested vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:54:15.323228 20232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "049f5bd8759c488789b7c396985be3f9" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:54:15.323432 20232 raft_consensus.cc:3037] T 049f5bd8759c488789b7c396985be3f9 P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:15.326083 20232 raft_consensus.cc:2454] T 049f5bd8759c488789b7c396985be3f9 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:54:15.328671 24735 raft_consensus.cc:686] T 049f5bd8759c488789b7c396985be3f9 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Becoming Leader. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Running, Role: LEADER
I20240502 01:54:15.328948 24735 consensus_queue.cc:237] T 049f5bd8759c488789b7c396985be3f9 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:15.328931 20362 consensus_queue.cc:237] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:54:15.341348 20625 raft_consensus.cc:1261] T 8eed067f639542d487de971761e91931 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:54:15.341804 20302 consensus_peers.cc:485] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8eed067f639542d487de971761e91931. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:15.342299 20232 raft_consensus.cc:1261] T 8eed067f639542d487de971761e91931 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:15.342769 23908 consensus_queue.cc:1035] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 01:54:15.344169 23908 consensus_queue.cc:1035] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:15.347194 17326 catalog_manager.cc:5484] T 049f5bd8759c488789b7c396985be3f9 P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66). New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:15.351688 23908 raft_consensus.cc:2932] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:15.353178 20627 raft_consensus.cc:2932] T 8eed067f639542d487de971761e91931 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:15.355029 20232 raft_consensus.cc:2932] T 8eed067f639542d487de971761e91931 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:15.360540 17330 catalog_manager.cc:5484] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:15.373739 20232 consensus_queue.cc:237] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:54:15.375934 20362 raft_consensus.cc:1261] T 61ab80ce377a44ac95a9c574e9a615f2 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:15.375949 17867 raft_consensus.cc:1261] T 61ab80ce377a44ac95a9c574e9a615f2 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:54:15.376159 20167 consensus_peers.cc:485] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 61ab80ce377a44ac95a9c574e9a615f2. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:15.376490 24811 consensus_queue.cc:1035] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:15.376775 24814 consensus_queue.cc:1035] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:15.380172 24811 raft_consensus.cc:2932] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:15.380803 20362 raft_consensus.cc:2932] T 61ab80ce377a44ac95a9c574e9a615f2 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:15.386904 17334 catalog_manager.cc:5484] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af reported cstate change: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New cstate: current_term: 1 leader_uuid: "fdc2ac372492454c85387589a91f63af" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:15.400981 17864 raft_consensus.cc:2932] T 61ab80ce377a44ac95a9c574e9a615f2 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:15.417826 24167 raft_consensus.cc:482] T 4ecf3b8482d04b37baf761305f933afb P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:15.417996 24167 raft_consensus.cc:504] T 4ecf3b8482d04b37baf761305f933afb P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:15.418691 24167 leader_election.cc:288] T 4ecf3b8482d04b37baf761305f933afb P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:54:15.419314 17476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ecf3b8482d04b37baf761305f933afb" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:54:15.419409 20627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ecf3b8482d04b37baf761305f933afb" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091" is_pre_election: true
I20240502 01:54:15.419500 17476 raft_consensus.cc:2454] T 4ecf3b8482d04b37baf761305f933afb P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 0.
I20240502 01:54:15.419636 20627 raft_consensus.cc:2454] T 4ecf3b8482d04b37baf761305f933afb P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 0.
I20240502 01:54:15.419871 17802 leader_election.cc:302] T 4ecf3b8482d04b37baf761305f933afb P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:54:15.420373 24167 raft_consensus.cc:2781] T 4ecf3b8482d04b37baf761305f933afb P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:15.420495 24167 raft_consensus.cc:482] T 4ecf3b8482d04b37baf761305f933afb P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:15.420583 24167 raft_consensus.cc:3037] T 4ecf3b8482d04b37baf761305f933afb P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:15.423066 24167 raft_consensus.cc:504] T 4ecf3b8482d04b37baf761305f933afb P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:15.423717 24167 leader_election.cc:288] T 4ecf3b8482d04b37baf761305f933afb P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 election: Requested vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:54:15.424160 20627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ecf3b8482d04b37baf761305f933afb" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091"
I20240502 01:54:15.424187 17476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ecf3b8482d04b37baf761305f933afb" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:54:15.424396 20627 raft_consensus.cc:3037] T 4ecf3b8482d04b37baf761305f933afb P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:15.424928 17476 raft_consensus.cc:3037] T 4ecf3b8482d04b37baf761305f933afb P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:15.427021 20627 raft_consensus.cc:2454] T 4ecf3b8482d04b37baf761305f933afb P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 1.
I20240502 01:54:15.427429 17802 leader_election.cc:302] T 4ecf3b8482d04b37baf761305f933afb P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb31bceba76d466db24b9341c04900c3, f19cc6cb1e76460aa6711e49a5775091; no voters:
I20240502 01:54:15.427564 17476 raft_consensus.cc:2454] T 4ecf3b8482d04b37baf761305f933afb P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 1.
I20240502 01:54:15.427721 24167 raft_consensus.cc:2781] T 4ecf3b8482d04b37baf761305f933afb P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:15.427874 24167 raft_consensus.cc:686] T 4ecf3b8482d04b37baf761305f933afb P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Becoming Leader. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Running, Role: LEADER
I20240502 01:54:15.428159 24167 consensus_queue.cc:237] T 4ecf3b8482d04b37baf761305f933afb P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:15.442955 17334 catalog_manager.cc:5484] T 4ecf3b8482d04b37baf761305f933afb P bb31bceba76d466db24b9341c04900c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb31bceba76d466db24b9341c04900c3 (127.11.42.68). New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:15.475606 20232 raft_consensus.cc:1203] T 29d8cb76dc514e7b9002bd2291d8ee8d P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:15.528872 17864 consensus_queue.cc:237] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:54:15.531337 20231 raft_consensus.cc:1261] T 0618761d30f34cfe9650d13c9275cdcb P fdc2ac372492454c85387589a91f63af [term 2 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 01:54:15.531858 17737 raft_consensus.cc:1261] T 0618761d30f34cfe9650d13c9275cdcb P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20240502 01:54:15.531903 17802 consensus_peers.cc:485] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0618761d30f34cfe9650d13c9275cdcb. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:15.532058 24888 consensus_queue.cc:1035] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:15.532359 24167 consensus_queue.cc:1035] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:15.535039 24167 raft_consensus.cc:2932] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:54:15.537979 17737 raft_consensus.cc:2932] T 0618761d30f34cfe9650d13c9275cdcb P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:54:15.541247 17334 catalog_manager.cc:5484] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 reported cstate change: config changed from index 7 to 8, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New cstate: current_term: 2 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:15.546875 20231 raft_consensus.cc:2932] T 0618761d30f34cfe9650d13c9275cdcb P fdc2ac372492454c85387589a91f63af [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:54:15.628218 24818 raft_consensus.cc:1050] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29: attempting to promote NON_VOTER 2f306549407b4bb281ccf2994035e4f1 to VOTER
I20240502 01:54:15.629093 24818 consensus_queue.cc:237] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } }
I20240502 01:54:15.638025 20497 raft_consensus.cc:1261] T 1fdd104f1a28446cbc25d4ca16d03d7d P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:15.638718 24735 consensus_queue.cc:1035] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:15.639997 20362 raft_consensus.cc:1261] T 1fdd104f1a28446cbc25d4ca16d03d7d P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:15.640249 17737 raft_consensus.cc:1261] T 1fdd104f1a28446cbc25d4ca16d03d7d P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:15.643150 24598 consensus_queue.cc:1035] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:15.643659 24598 consensus_queue.cc:1035] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:15.647275 17326 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:54:15.663574 23908 consensus_queue.cc:1035] T 52b83e43cae646a29bbf3b826fbbcfb6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:15.698931 17476 raft_consensus.cc:1261] T 4ecf3b8482d04b37baf761305f933afb P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:15.699141 20625 raft_consensus.cc:1261] T 4ecf3b8482d04b37baf761305f933afb P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:15.700245 24888 consensus_queue.cc:1035] T 4ecf3b8482d04b37baf761305f933afb P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:15.701494 24167 consensus_queue.cc:1035] T 4ecf3b8482d04b37baf761305f933afb P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:15.703202 23611 consensus_queue.cc:1035] T 52b83e43cae646a29bbf3b826fbbcfb6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:15.729586 17326 catalog_manager.cc:5576] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411) does not have the latest schema for tablet 4ecf3b8482d04b37baf761305f933afb (table rebalancer_dynamic_table [id=4193c1e64b834e889d5218c65ef0bba3]). Expected version 1 got 0
I20240502 01:54:15.733708 24918 raft_consensus.cc:2932] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:15.743754 17735 raft_consensus.cc:2932] T 1fdd104f1a28446cbc25d4ca16d03d7d P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:15.748845 24811 raft_consensus.cc:1050] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af: attempting to promote NON_VOTER f19cc6cb1e76460aa6711e49a5775091 to VOTER
I20240502 01:54:15.749706 24811 consensus_queue.cc:237] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } }
I20240502 01:54:15.755540 24932 tablet.cc:1722] T 4ecf3b8482d04b37baf761305f933afb P bb31bceba76d466db24b9341c04900c3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:15.756052 24932 tablet.cc:1620] T 4ecf3b8482d04b37baf761305f933afb P bb31bceba76d466db24b9341c04900c3: MemRowSet was empty: no flush needed.
I20240502 01:54:15.755949 17334 catalog_manager.cc:5484] T 1fdd104f1a28446cbc25d4ca16d03d7d P dfa1152987644bd781b46c7e42a7a685 reported cstate change: config changed from index 7 to 8, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:15.758258 24933 tablet.cc:1722] T 4ecf3b8482d04b37baf761305f933afb P f19cc6cb1e76460aa6711e49a5775091: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:15.763818 24933 tablet.cc:1620] T 4ecf3b8482d04b37baf761305f933afb P f19cc6cb1e76460aa6711e49a5775091: MemRowSet was empty: no flush needed.
I20240502 01:54:15.782981 20362 raft_consensus.cc:2932] T 1fdd104f1a28446cbc25d4ca16d03d7d P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:15.788702 24547 raft_consensus.cc:1050] T ee418fc4028a422eabac02ed63ea9047 P 2f306549407b4bb281ccf2994035e4f1: attempting to promote NON_VOTER bb31bceba76d466db24b9341c04900c3 to VOTER
I20240502 01:54:15.789582 24547 consensus_queue.cc:237] T ee418fc4028a422eabac02ed63ea9047 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } }
I20240502 01:54:15.791962 17735 raft_consensus.cc:3037] T b9d82510030b4503ac892175a8bb1bb7 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:15.795378 17326 catalog_manager.cc:5576] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411) does not have the latest schema for tablet 4ecf3b8482d04b37baf761305f933afb (table rebalancer_dynamic_table [id=4193c1e64b834e889d5218c65ef0bba3]). Expected version 1 got 0
I20240502 01:54:15.795467 24127 consensus_queue.cc:1035] T b9d82510030b4503ac892175a8bb1bb7 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:15.797247 24818 consensus_queue.cc:1035] T 049f5bd8759c488789b7c396985be3f9 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:15.803602 17326 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=4193c1e64b834e889d5218c65ef0bba3] alter complete (version 1)
I20240502 01:54:15.815497 24127 consensus_queue.cc:1035] T b9d82510030b4503ac892175a8bb1bb7 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:15.819449 20625 raft_consensus.cc:1261] T f3e4c8f98b5e48e3ba6affc2df08943a P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:15.820077 24813 consensus_queue.cc:1035] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:15.821918 20494 raft_consensus.cc:1261] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:15.822490 24814 consensus_queue.cc:1035] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:15.824784 20366 raft_consensus.cc:1261] T f3e4c8f98b5e48e3ba6affc2df08943a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:15.825071 24928 ts_tablet_manager.cc:902] T 71cbc012efe14fdb937302796a9f84e5 P 2f306549407b4bb281ccf2994035e4f1: Initiating tablet copy from peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:54:15.825350 24814 consensus_queue.cc:1035] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:15.835254 17864 raft_consensus.cc:1261] T ee418fc4028a422eabac02ed63ea9047 P bb31bceba76d466db24b9341c04900c3 [term 1 LEARNER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:15.835803 24545 consensus_queue.cc:1035] T ee418fc4028a422eabac02ed63ea9047 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:15.841492 20498 raft_consensus.cc:2932] T 1fdd104f1a28446cbc25d4ca16d03d7d P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:15.853780 17605 consensus_queue.cc:237] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } }
I20240502 01:54:15.860158 20362 raft_consensus.cc:1261] T 1fdd104f1a28446cbc25d4ca16d03d7d P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:15.862049 24818 consensus_queue.cc:1035] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:15.861395 20366 raft_consensus.cc:1261] T ee418fc4028a422eabac02ed63ea9047 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:15.862746 24545 consensus_queue.cc:1035] T ee418fc4028a422eabac02ed63ea9047 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:15.864435 17475 raft_consensus.cc:1261] T ee418fc4028a422eabac02ed63ea9047 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:15.864516 24598 raft_consensus.cc:2932] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:15.865507 24547 consensus_queue.cc:1035] T ee418fc4028a422eabac02ed63ea9047 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:15.872855 17330 catalog_manager.cc:5484] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: config changed from index 8 to 9, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:15.876727 17334 catalog_manager.cc:5576] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321) does not have the latest schema for tablet 4ecf3b8482d04b37baf761305f933afb (table rebalancer_dynamic_table [id=4193c1e64b834e889d5218c65ef0bba3]). Expected version 1 got 0
I20240502 01:54:15.879776 17716 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fdd104f1a28446cbc25d4ca16d03d7d with delete_type TABLET_DATA_TOMBSTONED (TS dfa1152987644bd781b46c7e42a7a685 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:52198
I20240502 01:54:15.879942 17319 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1fdd104f1a28446cbc25d4ca16d03d7d with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:15.883773 17330 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_extra_table_52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:15.890532 24951 ts_tablet_manager.cc:902] T 61ab80ce377a44ac95a9c574e9a615f2 P dfa1152987644bd781b46c7e42a7a685: Initiating tablet copy from peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:54:15.896935 24545 raft_consensus.cc:2932] T ee418fc4028a422eabac02ed63ea9047 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:15.903170 24814 raft_consensus.cc:2932] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:15.903676 17473 raft_consensus.cc:2932] T ee418fc4028a422eabac02ed63ea9047 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:15.903851 20496 raft_consensus.cc:2932] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:15.910071 24735 consensus_queue.cc:1035] T 049f5bd8759c488789b7c396985be3f9 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:15.913599 20477 tablet_service.cc:1467] Processing CreateTablet for tablet 25ad701a0fec4e80b37df5dec598796a (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=2532ab32527b409cb1b76975ab07f04f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:15.914034 20477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25ad701a0fec4e80b37df5dec598796a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:15.914328 17333 catalog_manager.cc:5484] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af reported cstate change: config changed from index 2 to 3, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fdc2ac372492454c85387589a91f63af" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:15.917680 17330 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:54:15.921131 17456 tablet_service.cc:1467] Processing CreateTablet for tablet 25ad701a0fec4e80b37df5dec598796a (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=2532ab32527b409cb1b76975ab07f04f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:15.921511 17456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25ad701a0fec4e80b37df5dec598796a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:15.922304 20211 tablet_service.cc:1467] Processing CreateTablet for tablet 25ad701a0fec4e80b37df5dec598796a (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=2532ab32527b409cb1b76975ab07f04f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:15.922675 20211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25ad701a0fec4e80b37df5dec598796a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:15.926609 17715 tablet_service.cc:1467] Processing CreateTablet for tablet 43572b8f8fba433ab00b046c769d886d (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=2532ab32527b409cb1b76975ab07f04f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:15.927012 17715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43572b8f8fba433ab00b046c769d886d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:15.926966 17585 tablet_service.cc:1467] Processing CreateTablet for tablet 43572b8f8fba433ab00b046c769d886d (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=2532ab32527b409cb1b76975ab07f04f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:15.927368 17585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43572b8f8fba433ab00b046c769d886d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:15.927951 17847 tablet_service.cc:1467] Processing CreateTablet for tablet 7efd29396c7c4fb597be2251cdcdd48f (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=2532ab32527b409cb1b76975ab07f04f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:15.928337 17847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7efd29396c7c4fb597be2251cdcdd48f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:15.930135 17454 tablet_service.cc:1467] Processing CreateTablet for tablet 7efd29396c7c4fb597be2251cdcdd48f (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=2532ab32527b409cb1b76975ab07f04f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:15.930485 17454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7efd29396c7c4fb597be2251cdcdd48f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:15.936455 20212 tablet_service.cc:1467] Processing CreateTablet for tablet 7efd29396c7c4fb597be2251cdcdd48f (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=2532ab32527b409cb1b76975ab07f04f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:15.936901 20212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7efd29396c7c4fb597be2251cdcdd48f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:15.939680 24932 tablet.cc:1722] T 4ecf3b8482d04b37baf761305f933afb P bb31bceba76d466db24b9341c04900c3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:15.940238 24932 tablet.cc:1620] T 4ecf3b8482d04b37baf761305f933afb P bb31bceba76d466db24b9341c04900c3: MemRowSet was empty: no flush needed.
I20240502 01:54:15.940177 24933 tablet.cc:1722] T 4ecf3b8482d04b37baf761305f933afb P f19cc6cb1e76460aa6711e49a5775091: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:15.940625 24933 tablet.cc:1620] T 4ecf3b8482d04b37baf761305f933afb P f19cc6cb1e76460aa6711e49a5775091: MemRowSet was empty: no flush needed.
I20240502 01:54:15.952428 24954 tablet.cc:1722] T 4ecf3b8482d04b37baf761305f933afb P 4da690c0fa13407a85c0689ce8de9ade: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:15.953280 24954 tablet.cc:1620] T 4ecf3b8482d04b37baf761305f933afb P 4da690c0fa13407a85c0689ce8de9ade: MemRowSet was empty: no flush needed.
I20240502 01:54:15.956367 24952 tablet_replica.cc:330] T 1fdd104f1a28446cbc25d4ca16d03d7d P dfa1152987644bd781b46c7e42a7a685: stopping tablet replica
I20240502 01:54:15.956616 24952 raft_consensus.cc:2229] T 1fdd104f1a28446cbc25d4ca16d03d7d P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:15.957288 24952 raft_consensus.cc:2258] T 1fdd104f1a28446cbc25d4ca16d03d7d P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:15.960114 17334 catalog_manager.cc:5576] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083) does not have the latest schema for tablet 4ecf3b8482d04b37baf761305f933afb (table rebalancer_dynamic_table [id=4193c1e64b834e889d5218c65ef0bba3]). Expected version 2 got 1
I20240502 01:54:15.964320 17328 catalog_manager.cc:5576] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321) does not have the latest schema for tablet 4ecf3b8482d04b37baf761305f933afb (table rebalancer_dynamic_table [id=4193c1e64b834e889d5218c65ef0bba3]). Expected version 2 got 0
I20240502 01:54:15.965493 20361 raft_consensus.cc:2932] T f3e4c8f98b5e48e3ba6affc2df08943a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:15.966624 20345 tablet_service.cc:1467] Processing CreateTablet for tablet 43572b8f8fba433ab00b046c769d886d (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=2532ab32527b409cb1b76975ab07f04f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:15.966842 24967 tablet_bootstrap.cc:492] T 7efd29396c7c4fb597be2251cdcdd48f P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:54:15.969425 24964 tablet_bootstrap.cc:492] T 43572b8f8fba433ab00b046c769d886d P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:54:15.970443 20345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43572b8f8fba433ab00b046c769d886d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:15.971210 24964 tablet_bootstrap.cc:654] T 43572b8f8fba433ab00b046c769d886d P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:15.974982 24952 ts_tablet_manager.cc:1875] T 1fdd104f1a28446cbc25d4ca16d03d7d P dfa1152987644bd781b46c7e42a7a685: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:15.981447 24967 tablet_bootstrap.cc:654] T 7efd29396c7c4fb597be2251cdcdd48f P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:15.984369 24963 tablet_bootstrap.cc:492] T 43572b8f8fba433ab00b046c769d886d P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:54:15.985143 24966 tablet_bootstrap.cc:492] T 25ad701a0fec4e80b37df5dec598796a P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:54:15.985213 24961 tablet_bootstrap.cc:492] T 25ad701a0fec4e80b37df5dec598796a P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
W20240502 01:54:15.988432 24948 log.cc:924] Time spent T ee418fc4028a422eabac02ed63ea9047 P 6a36c952c51542c0bf7ea3ec2857a9f9: Append to log took a long time: real 0.096s user 0.000s sys 0.000s
I20240502 01:54:15.996028 20364 raft_consensus.cc:2932] T ee418fc4028a422eabac02ed63ea9047 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:15.995566 20362 raft_consensus.cc:2932] T 1fdd104f1a28446cbc25d4ca16d03d7d P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:16.006825 23967 raft_consensus.cc:1050] T 29d8cb76dc514e7b9002bd2291d8ee8d P dfa1152987644bd781b46c7e42a7a685: attempting to promote NON_VOTER fdc2ac372492454c85387589a91f63af to VOTER
I20240502 01:54:16.006172 17333 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=4193c1e64b834e889d5218c65ef0bba3] alter complete (version 2)
I20240502 01:54:16.007721 23967 consensus_queue.cc:237] T 29d8cb76dc514e7b9002bd2291d8ee8d P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } }
I20240502 01:54:16.011927 24972 tablet.cc:1722] T 4ecf3b8482d04b37baf761305f933afb P 4da690c0fa13407a85c0689ce8de9ade: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:16.012413 24966 tablet_bootstrap.cc:654] T 25ad701a0fec4e80b37df5dec598796a P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:16.012508 24972 tablet.cc:1620] T 4ecf3b8482d04b37baf761305f933afb P 4da690c0fa13407a85c0689ce8de9ade: MemRowSet was empty: no flush needed.
I20240502 01:54:16.014634 24961 tablet_bootstrap.cc:654] T 25ad701a0fec4e80b37df5dec598796a P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
W20240502 01:54:16.015419 20564 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.42.67:36115, user_credentials={real_user=slave}} blocked reactor thread for 118758us
I20240502 01:54:16.015517 24965 tablet_bootstrap.cc:492] T 7efd29396c7c4fb597be2251cdcdd48f P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:54:16.016314 17334 catalog_manager.cc:5484] T ee418fc4028a422eabac02ed63ea9047 P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: config changed from index 2 to 3, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:16.015512 24967 tablet_bootstrap.cc:492] T 7efd29396c7c4fb597be2251cdcdd48f P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:54:16.017477 24967 ts_tablet_manager.cc:1367] T 7efd29396c7c4fb597be2251cdcdd48f P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.051s user 0.004s sys 0.000s
I20240502 01:54:16.018144 24967 raft_consensus.cc:348] T 7efd29396c7c4fb597be2251cdcdd48f P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:16.019475 17867 raft_consensus.cc:2932] T ee418fc4028a422eabac02ed63ea9047 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:16.023102 20625 raft_consensus.cc:2932] T f3e4c8f98b5e48e3ba6affc2df08943a P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:16.023629 24969 ts_tablet_manager.cc:902] T 8eed067f639542d487de971761e91931 P dfa1152987644bd781b46c7e42a7a685: Initiating tablet copy from peer 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:54:16.023766 24967 raft_consensus.cc:374] T 7efd29396c7c4fb597be2251cdcdd48f P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:16.027644 24964 tablet_bootstrap.cc:492] T 43572b8f8fba433ab00b046c769d886d P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:54:16.027843 24964 ts_tablet_manager.cc:1367] T 43572b8f8fba433ab00b046c769d886d P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.059s user 0.002s sys 0.003s
I20240502 01:54:16.029290 24967 raft_consensus.cc:729] T 7efd29396c7c4fb597be2251cdcdd48f P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:54:16.029515 24970 tablet_bootstrap.cc:492] T 43572b8f8fba433ab00b046c769d886d P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:54:16.029697 24967 consensus_queue.cc:260] T 7efd29396c7c4fb597be2251cdcdd48f P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:16.031201 24970 tablet_bootstrap.cc:654] T 43572b8f8fba433ab00b046c769d886d P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:16.032593 24967 ts_tablet_manager.cc:1398] T 7efd29396c7c4fb597be2251cdcdd48f P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240502 01:54:16.033908 24963 tablet_bootstrap.cc:654] T 43572b8f8fba433ab00b046c769d886d P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:16.039021 24966 tablet_bootstrap.cc:492] T 25ad701a0fec4e80b37df5dec598796a P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:54:16.039227 24966 ts_tablet_manager.cc:1367] T 25ad701a0fec4e80b37df5dec598796a P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.055s user 0.003s sys 0.004s
I20240502 01:54:16.028532 24964 raft_consensus.cc:348] T 43572b8f8fba433ab00b046c769d886d P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:16.041852 24964 raft_consensus.cc:374] T 43572b8f8fba433ab00b046c769d886d P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:16.041923 24964 raft_consensus.cc:729] T 43572b8f8fba433ab00b046c769d886d P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:54:16.041757 20496 consensus_queue.cc:237] T ee418fc4028a422eabac02ed63ea9047 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } }
I20240502 01:54:16.042121 24964 consensus_queue.cc:260] T 43572b8f8fba433ab00b046c769d886d P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:16.042470 24966 raft_consensus.cc:348] T 25ad701a0fec4e80b37df5dec598796a P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:16.042649 24964 ts_tablet_manager.cc:1398] T 43572b8f8fba433ab00b046c769d886d P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.015s user 0.000s sys 0.001s
I20240502 01:54:16.042776 24966 raft_consensus.cc:374] T 25ad701a0fec4e80b37df5dec598796a P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:16.042938 24966 raft_consensus.cc:729] T 25ad701a0fec4e80b37df5dec598796a P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:54:16.043262 24966 consensus_queue.cc:260] T 25ad701a0fec4e80b37df5dec598796a P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:16.043435 24952 ts_tablet_manager.cc:1888] T 1fdd104f1a28446cbc25d4ca16d03d7d P dfa1152987644bd781b46c7e42a7a685: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:54:16.044947 24952 log.cc:1196] T 1fdd104f1a28446cbc25d4ca16d03d7d P dfa1152987644bd781b46c7e42a7a685: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/wals/1fdd104f1a28446cbc25d4ca16d03d7d
I20240502 01:54:16.045403 24965 tablet_bootstrap.cc:654] T 7efd29396c7c4fb597be2251cdcdd48f P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:16.045969 24966 ts_tablet_manager.cc:1398] T 25ad701a0fec4e80b37df5dec598796a P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240502 01:54:16.045960 17321 catalog_manager.cc:4830] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): tablet 1fdd104f1a28446cbc25d4ca16d03d7d (table rebalance_test_table_13 [id=dbf97af96f07462286eb1fabeddf25fa]) successfully deleted
I20240502 01:54:16.049168 17867 raft_consensus.cc:1261] T ee418fc4028a422eabac02ed63ea9047 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:16.049525 17475 raft_consensus.cc:1261] T ee418fc4028a422eabac02ed63ea9047 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:16.050408 24958 consensus_queue.cc:1035] T ee418fc4028a422eabac02ed63ea9047 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 01:54:16.052485 24966 tablet_bootstrap.cc:492] T 7efd29396c7c4fb597be2251cdcdd48f P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:54:16.053004 24959 consensus_queue.cc:1035] T ee418fc4028a422eabac02ed63ea9047 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:16.056730 24966 tablet_bootstrap.cc:654] T 7efd29396c7c4fb597be2251cdcdd48f P fdc2ac372492454c85387589a91f63af: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:16.057479 24957 raft_consensus.cc:2932] T ee418fc4028a422eabac02ed63ea9047 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:16.058074 17475 raft_consensus.cc:2932] T ee418fc4028a422eabac02ed63ea9047 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:16.061266 24951 tablet_copy_client.cc:308] T 61ab80ce377a44ac95a9c574e9a615f2 P dfa1152987644bd781b46c7e42a7a685: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.69:34339
I20240502 01:54:16.061633 24928 tablet_copy_client.cc:308] T 71cbc012efe14fdb937302796a9f84e5 P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.67:36115
I20240502 01:54:16.062615 17757 tablet_copy_service.cc:140] P dfa1152987644bd781b46c7e42a7a685: Received BeginTabletCopySession request for tablet 71cbc012efe14fdb937302796a9f84e5 from peer 2f306549407b4bb281ccf2994035e4f1 ({username='slave'} at 127.11.42.71:49495)
I20240502 01:54:16.062741 17757 tablet_copy_service.cc:161] P dfa1152987644bd781b46c7e42a7a685: Beginning new tablet copy session on tablet 71cbc012efe14fdb937302796a9f84e5 from peer 2f306549407b4bb281ccf2994035e4f1 at {username='slave'} at 127.11.42.71:49495: session id = 2f306549407b4bb281ccf2994035e4f1-71cbc012efe14fdb937302796a9f84e5
I20240502 01:54:16.062899 20253 tablet_copy_service.cc:140] P fdc2ac372492454c85387589a91f63af: Received BeginTabletCopySession request for tablet 61ab80ce377a44ac95a9c574e9a615f2 from peer dfa1152987644bd781b46c7e42a7a685 ({username='slave'} at 127.11.42.67:38077)
I20240502 01:54:16.063023 20253 tablet_copy_service.cc:161] P fdc2ac372492454c85387589a91f63af: Beginning new tablet copy session on tablet 61ab80ce377a44ac95a9c574e9a615f2 from peer dfa1152987644bd781b46c7e42a7a685 at {username='slave'} at 127.11.42.67:38077: session id = dfa1152987644bd781b46c7e42a7a685-61ab80ce377a44ac95a9c574e9a615f2
I20240502 01:54:16.065729 17757 tablet_copy_source_session.cc:205] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:16.067162 20253 tablet_copy_source_session.cc:205] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:16.067574 17330 catalog_manager.cc:5484] T ee418fc4028a422eabac02ed63ea9047 P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: config changed from index 3 to 4, VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) evicted. New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:16.070600 24966 tablet_bootstrap.cc:492] T 7efd29396c7c4fb597be2251cdcdd48f P fdc2ac372492454c85387589a91f63af: No bootstrap required, opened a new log
I20240502 01:54:16.070930 24966 ts_tablet_manager.cc:1367] T 7efd29396c7c4fb597be2251cdcdd48f P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.004s
I20240502 01:54:16.071872 24951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61ab80ce377a44ac95a9c574e9a615f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:16.072727 17867 raft_consensus.cc:2932] T ee418fc4028a422eabac02ed63ea9047 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:16.073312 20345 tablet_service.cc:1514] Processing DeleteTablet for tablet ee418fc4028a422eabac02ed63ea9047 with delete_type TABLET_DATA_TOMBSTONED (TS 6a36c952c51542c0bf7ea3ec2857a9f9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40196
I20240502 01:54:16.074275 24965 tablet_bootstrap.cc:492] T 7efd29396c7c4fb597be2251cdcdd48f P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:54:16.074550 24965 ts_tablet_manager.cc:1367] T 7efd29396c7c4fb597be2251cdcdd48f P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.059s user 0.003s sys 0.000s
I20240502 01:54:16.075450 24928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71cbc012efe14fdb937302796a9f84e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:16.075660 24965 raft_consensus.cc:348] T 7efd29396c7c4fb597be2251cdcdd48f P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:16.075906 24965 raft_consensus.cc:374] T 7efd29396c7c4fb597be2251cdcdd48f P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:16.076006 24965 raft_consensus.cc:729] T 7efd29396c7c4fb597be2251cdcdd48f P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:54:16.076565 24988 tablet_replica.cc:330] T ee418fc4028a422eabac02ed63ea9047 P 6a36c952c51542c0bf7ea3ec2857a9f9: stopping tablet replica
I20240502 01:54:16.076867 24965 consensus_queue.cc:260] T 7efd29396c7c4fb597be2251cdcdd48f P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:16.081720 24965 ts_tablet_manager.cc:1398] T 7efd29396c7c4fb597be2251cdcdd48f P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240502 01:54:16.081998 24988 raft_consensus.cc:2229] T ee418fc4028a422eabac02ed63ea9047 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:16.082821 17322 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ee418fc4028a422eabac02ed63ea9047 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:16.084017 24961 tablet_bootstrap.cc:492] T 25ad701a0fec4e80b37df5dec598796a P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:54:16.084233 24961 ts_tablet_manager.cc:1367] T 25ad701a0fec4e80b37df5dec598796a P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.101s user 0.013s sys 0.000s
I20240502 01:54:16.085070 24961 raft_consensus.cc:348] T 25ad701a0fec4e80b37df5dec598796a P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:16.085325 24961 raft_consensus.cc:374] T 25ad701a0fec4e80b37df5dec598796a P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:16.085450 24961 raft_consensus.cc:729] T 25ad701a0fec4e80b37df5dec598796a P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:54:16.085708 24961 consensus_queue.cc:260] T 25ad701a0fec4e80b37df5dec598796a P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:16.086333 24961 ts_tablet_manager.cc:1398] T 25ad701a0fec4e80b37df5dec598796a P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:54:16.087440 24969 tablet_copy_client.cc:308] T 8eed067f639542d487de971761e91931 P dfa1152987644bd781b46c7e42a7a685: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.70:39155
I20240502 01:54:16.089934 24965 tablet_bootstrap.cc:492] T 25ad701a0fec4e80b37df5dec598796a P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:54:16.090618 24966 raft_consensus.cc:348] T 7efd29396c7c4fb597be2251cdcdd48f P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:16.091428 24988 raft_consensus.cc:2258] T ee418fc4028a422eabac02ed63ea9047 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:16.091562 20387 tablet_copy_service.cc:140] P 6a36c952c51542c0bf7ea3ec2857a9f9: Received BeginTabletCopySession request for tablet 8eed067f639542d487de971761e91931 from peer dfa1152987644bd781b46c7e42a7a685 ({username='slave'} at 127.11.42.67:35525)
I20240502 01:54:16.091807 20387 tablet_copy_service.cc:161] P 6a36c952c51542c0bf7ea3ec2857a9f9: Beginning new tablet copy session on tablet 8eed067f639542d487de971761e91931 from peer dfa1152987644bd781b46c7e42a7a685 at {username='slave'} at 127.11.42.67:35525: session id = dfa1152987644bd781b46c7e42a7a685-8eed067f639542d487de971761e91931
I20240502 01:54:16.093195 24965 tablet_bootstrap.cc:654] T 25ad701a0fec4e80b37df5dec598796a P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:16.093570 20497 raft_consensus.cc:1261] T 1fdd104f1a28446cbc25d4ca16d03d7d P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:16.094038 20232 raft_consensus.cc:1261] T 29d8cb76dc514e7b9002bd2291d8ee8d P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:16.095178 24963 tablet_bootstrap.cc:492] T 43572b8f8fba433ab00b046c769d886d P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:54:16.095206 20361 raft_consensus.cc:1261] T 29d8cb76dc514e7b9002bd2291d8ee8d P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:16.095561 20387 tablet_copy_source_session.cc:205] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:16.095754 24797 consensus_queue.cc:1035] T 29d8cb76dc514e7b9002bd2291d8ee8d P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:16.096307 24797 consensus_queue.cc:1035] T 29d8cb76dc514e7b9002bd2291d8ee8d P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:16.099622 24988 ts_tablet_manager.cc:1875] T ee418fc4028a422eabac02ed63ea9047 P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:16.095302 24963 ts_tablet_manager.cc:1367] T 43572b8f8fba433ab00b046c769d886d P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.113s user 0.004s sys 0.002s
I20240502 01:54:16.100610 24963 raft_consensus.cc:348] T 43572b8f8fba433ab00b046c769d886d P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:16.100687 20496 raft_consensus.cc:1261] T 29d8cb76dc514e7b9002bd2291d8ee8d P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:16.100756 24963 raft_consensus.cc:374] T 43572b8f8fba433ab00b046c769d886d P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:16.101182 23967 consensus_queue.cc:1035] T 29d8cb76dc514e7b9002bd2291d8ee8d P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:16.101925 24598 consensus_queue.cc:1035] T 1fdd104f1a28446cbc25d4ca16d03d7d P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:16.103667 24963 raft_consensus.cc:729] T 43572b8f8fba433ab00b046c769d886d P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:54:16.103946 24963 consensus_queue.cc:260] T 43572b8f8fba433ab00b046c769d886d P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:16.104296 20498 raft_consensus.cc:2932] T 1fdd104f1a28446cbc25d4ca16d03d7d P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:16.107265 24965 tablet_bootstrap.cc:492] T 25ad701a0fec4e80b37df5dec598796a P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:54:16.107405 24965 ts_tablet_manager.cc:1367] T 25ad701a0fec4e80b37df5dec598796a P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.003s
I20240502 01:54:16.109062 24965 raft_consensus.cc:348] T 25ad701a0fec4e80b37df5dec598796a P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:16.109321 24965 raft_consensus.cc:374] T 25ad701a0fec4e80b37df5dec598796a P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:16.110038 17326 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 01:54:16.111804 24963 ts_tablet_manager.cc:1398] T 43572b8f8fba433ab00b046c769d886d P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
I20240502 01:54:16.116366 24988 ts_tablet_manager.cc:1888] T ee418fc4028a422eabac02ed63ea9047 P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:54:16.109422 24965 raft_consensus.cc:729] T 25ad701a0fec4e80b37df5dec598796a P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:54:16.116760 24965 consensus_queue.cc:260] T 25ad701a0fec4e80b37df5dec598796a P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:16.116492 24988 log.cc:1196] T ee418fc4028a422eabac02ed63ea9047 P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal/wals/ee418fc4028a422eabac02ed63ea9047
I20240502 01:54:16.118172 24932 tablet.cc:1722] T 4ecf3b8482d04b37baf761305f933afb P bb31bceba76d466db24b9341c04900c3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:16.118682 17321 catalog_manager.cc:4830] TS 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): tablet ee418fc4028a422eabac02ed63ea9047 (table rebalancer_extra_table_14 [id=54db4901eae145fbbc5c66c0719703f1]) successfully deleted
I20240502 01:54:16.118947 24965 ts_tablet_manager.cc:1398] T 25ad701a0fec4e80b37df5dec598796a P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240502 01:54:16.122390 24933 tablet.cc:1722] T 4ecf3b8482d04b37baf761305f933afb P f19cc6cb1e76460aa6711e49a5775091: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:16.094058 24966 raft_consensus.cc:374] T 7efd29396c7c4fb597be2251cdcdd48f P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:16.127812 24966 raft_consensus.cc:729] T 7efd29396c7c4fb597be2251cdcdd48f P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: FOLLOWER
I20240502 01:54:16.128159 24966 consensus_queue.cc:260] T 7efd29396c7c4fb597be2251cdcdd48f P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:16.128830 24966 ts_tablet_manager.cc:1398] T 7efd29396c7c4fb597be2251cdcdd48f P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.058s user 0.001s sys 0.001s
I20240502 01:54:16.129150 24972 tablet.cc:1722] T 4ecf3b8482d04b37baf761305f933afb P 4da690c0fa13407a85c0689ce8de9ade: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 01:54:16.128959 23967 raft_consensus.cc:2932] T 29d8cb76dc514e7b9002bd2291d8ee8d P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:16.129607 24969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eed067f639542d487de971761e91931. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:16.135571 20232 raft_consensus.cc:2932] T 29d8cb76dc514e7b9002bd2291d8ee8d P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:16.143375 17330 catalog_manager.cc:5484] T 29d8cb76dc514e7b9002bd2291d8ee8d P fdc2ac372492454c85387589a91f63af reported cstate change: config changed from index 2 to 3, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:16.146976 24970 tablet_bootstrap.cc:492] T 43572b8f8fba433ab00b046c769d886d P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:54:16.147337 24970 ts_tablet_manager.cc:1367] T 43572b8f8fba433ab00b046c769d886d P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.118s user 0.006s sys 0.000s
I20240502 01:54:16.145327 17334 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table_new_name [id=4193c1e64b834e889d5218c65ef0bba3] alter complete (version 3)
I20240502 01:54:16.155447 24970 raft_consensus.cc:348] T 43572b8f8fba433ab00b046c769d886d P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:16.155748 24970 raft_consensus.cc:374] T 43572b8f8fba433ab00b046c769d886d P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:16.155879 24970 raft_consensus.cc:729] T 43572b8f8fba433ab00b046c769d886d P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:54:16.156158 24970 consensus_queue.cc:260] T 43572b8f8fba433ab00b046c769d886d P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:16.156972 25001 ts_tablet_manager.cc:902] T 0618761d30f34cfe9650d13c9275cdcb P f19cc6cb1e76460aa6711e49a5775091: Initiating tablet copy from peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:54:16.159826 24951 tablet_copy_client.cc:783] T 61ab80ce377a44ac95a9c574e9a615f2 P dfa1152987644bd781b46c7e42a7a685: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:16.167088 24970 ts_tablet_manager.cc:1398] T 43572b8f8fba433ab00b046c769d886d P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
I20240502 01:54:16.168921 20496 raft_consensus.cc:2932] T 29d8cb76dc514e7b9002bd2291d8ee8d P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:16.171206 17735 consensus_queue.cc:237] T 29d8cb76dc514e7b9002bd2291d8ee8d P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } }
I20240502 01:54:16.168231 20361 raft_consensus.cc:2932] T 29d8cb76dc514e7b9002bd2291d8ee8d P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, fdc2ac372492454c85387589a91f63af (127.11.42.69) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:16.173074 20233 raft_consensus.cc:1261] T 29d8cb76dc514e7b9002bd2291d8ee8d P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:16.173731 24984 consensus_queue.cc:1035] T 29d8cb76dc514e7b9002bd2291d8ee8d P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:16.180757 24797 raft_consensus.cc:2932] T 29d8cb76dc514e7b9002bd2291d8ee8d P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:16.160041 24951 tablet_copy_client.cc:647] T 61ab80ce377a44ac95a9c574e9a615f2 P dfa1152987644bd781b46c7e42a7a685: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:16.185065 24918 raft_consensus.cc:482] T 43572b8f8fba433ab00b046c769d886d P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:16.185207 24918 raft_consensus.cc:504] T 43572b8f8fba433ab00b046c769d886d P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:16.187139 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43572b8f8fba433ab00b046c769d886d" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:54:16.187407 17735 raft_consensus.cc:2454] T 43572b8f8fba433ab00b046c769d886d P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:54:16.187858 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 29d8cb76dc514e7b9002bd2291d8ee8d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:16.188174 17542 leader_election.cc:302] T 43572b8f8fba433ab00b046c769d886d P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:54:16.188408 24598 raft_consensus.cc:2781] T 43572b8f8fba433ab00b046c769d886d P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:16.188504 24598 raft_consensus.cc:482] T 43572b8f8fba433ab00b046c769d886d P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:16.188565 24598 raft_consensus.cc:3037] T 43572b8f8fba433ab00b046c769d886d P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:16.188737 24918 leader_election.cc:288] T 43572b8f8fba433ab00b046c769d886d P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:54:16.189620 20364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43572b8f8fba433ab00b046c769d886d" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:54:16.189898 24951 tablet_copy_client.cc:523] T 61ab80ce377a44ac95a9c574e9a615f2 P dfa1152987644bd781b46c7e42a7a685: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:16.190044 20233 raft_consensus.cc:2932] T 29d8cb76dc514e7b9002bd2291d8ee8d P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:16.191046 24598 raft_consensus.cc:504] T 43572b8f8fba433ab00b046c769d886d P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:16.191646 24598 leader_election.cc:288] T 43572b8f8fba433ab00b046c769d886d P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Requested vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:54:16.192827 20367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43572b8f8fba433ab00b046c769d886d" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
I20240502 01:54:16.195151 20367 raft_consensus.cc:3037] T 43572b8f8fba433ab00b046c769d886d P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:16.196897 17334 catalog_manager.cc:5484] T 29d8cb76dc514e7b9002bd2291d8ee8d P fdc2ac372492454c85387589a91f63af reported cstate change: config changed from index 3 to 4, VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) evicted. New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:16.197831 20367 raft_consensus.cc:2454] T 43572b8f8fba433ab00b046c769d886d P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:54:16.199607 17542 leader_election.cc:302] T 43572b8f8fba433ab00b046c769d886d P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, 6a36c952c51542c0bf7ea3ec2857a9f9; no voters:
I20240502 01:54:16.200153 24918 raft_consensus.cc:2781] T 43572b8f8fba433ab00b046c769d886d P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:16.200372 24918 raft_consensus.cc:686] T 43572b8f8fba433ab00b046c769d886d P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Becoming Leader. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Running, Role: LEADER
I20240502 01:54:16.201332 20364 raft_consensus.cc:2362] T 43572b8f8fba433ab00b046c769d886d P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1. Re-sending same reply.
I20240502 01:54:16.201498 24212 raft_consensus.cc:482] T 7efd29396c7c4fb597be2251cdcdd48f P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:16.201611 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43572b8f8fba433ab00b046c769d886d" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:54:16.201625 24212 raft_consensus.cc:504] T 7efd29396c7c4fb597be2251cdcdd48f P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:16.201771 17735 raft_consensus.cc:3037] T 43572b8f8fba433ab00b046c769d886d P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:16.202236 24212 leader_election.cc:288] T 7efd29396c7c4fb597be2251cdcdd48f P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:54:16.202872 17867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7efd29396c7c4fb597be2251cdcdd48f" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:54:16.203071 17867 raft_consensus.cc:2454] T 7efd29396c7c4fb597be2251cdcdd48f P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 0.
I20240502 01:54:16.182858 24928 tablet_copy_client.cc:783] T 71cbc012efe14fdb937302796a9f84e5 P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:16.203505 24928 tablet_copy_client.cc:647] T 71cbc012efe14fdb937302796a9f84e5 P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:16.203514 17412 leader_election.cc:302] T 7efd29396c7c4fb597be2251cdcdd48f P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:54:16.204047 24212 raft_consensus.cc:2781] T 7efd29396c7c4fb597be2251cdcdd48f P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:16.204123 24212 raft_consensus.cc:482] T 7efd29396c7c4fb597be2251cdcdd48f P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:16.204182 24212 raft_consensus.cc:3037] T 7efd29396c7c4fb597be2251cdcdd48f P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:16.204339 17735 raft_consensus.cc:2454] T 43572b8f8fba433ab00b046c769d886d P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:54:16.206670 24212 raft_consensus.cc:504] T 7efd29396c7c4fb597be2251cdcdd48f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:16.207305 24928 tablet_copy_client.cc:523] T 71cbc012efe14fdb937302796a9f84e5 P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:16.208092 17867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7efd29396c7c4fb597be2251cdcdd48f" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:54:16.208257 17867 raft_consensus.cc:3037] T 7efd29396c7c4fb597be2251cdcdd48f P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:16.208838 24969 tablet_copy_client.cc:783] T 8eed067f639542d487de971761e91931 P dfa1152987644bd781b46c7e42a7a685: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:16.210619 24928 tablet_bootstrap.cc:492] T 71cbc012efe14fdb937302796a9f84e5 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:54:16.210872 17867 raft_consensus.cc:2454] T 7efd29396c7c4fb597be2251cdcdd48f P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
I20240502 01:54:16.210781 24969 tablet_copy_client.cc:647] T 8eed067f639542d487de971761e91931 P dfa1152987644bd781b46c7e42a7a685: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:16.211283 17412 leader_election.cc:302] T 7efd29396c7c4fb597be2251cdcdd48f P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:54:16.212221 20233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7efd29396c7c4fb597be2251cdcdd48f" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af" is_pre_election: true
I20240502 01:54:16.211997 24951 tablet_bootstrap.cc:492] T 61ab80ce377a44ac95a9c574e9a615f2 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:54:16.212379 20232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7efd29396c7c4fb597be2251cdcdd48f" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:54:16.212435 20233 raft_consensus.cc:2454] T 7efd29396c7c4fb597be2251cdcdd48f P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 0.
I20240502 01:54:16.213073 24969 tablet_copy_client.cc:523] T 8eed067f639542d487de971761e91931 P dfa1152987644bd781b46c7e42a7a685: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:16.232195 24951 tablet_bootstrap.cc:492] T 61ab80ce377a44ac95a9c574e9a615f2 P dfa1152987644bd781b46c7e42a7a685: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:16.232973 24951 tablet_bootstrap.cc:492] T 61ab80ce377a44ac95a9c574e9a615f2 P dfa1152987644bd781b46c7e42a7a685: Bootstrap complete.
I20240502 01:54:16.233721 24951 ts_tablet_manager.cc:1367] T 61ab80ce377a44ac95a9c574e9a615f2 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.001s
I20240502 01:54:16.234406 24212 leader_election.cc:288] T 7efd29396c7c4fb597be2251cdcdd48f P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Requested vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:54:16.234400 24951 raft_consensus.cc:348] T 61ab80ce377a44ac95a9c574e9a615f2 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:54:16.234624 24951 raft_consensus.cc:729] T 61ab80ce377a44ac95a9c574e9a615f2 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: LEARNER
I20240502 01:54:16.234772 24212 raft_consensus.cc:2781] T 7efd29396c7c4fb597be2251cdcdd48f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:16.234835 24951 consensus_queue.cc:260] T 61ab80ce377a44ac95a9c574e9a615f2 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:54:16.234972 24212 raft_consensus.cc:686] T 7efd29396c7c4fb597be2251cdcdd48f P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Becoming Leader. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Running, Role: LEADER
I20240502 01:54:16.237185 24951 ts_tablet_manager.cc:1398] T 61ab80ce377a44ac95a9c574e9a615f2 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:54:16.238775 20253 tablet_copy_service.cc:342] P fdc2ac372492454c85387589a91f63af: Request end of tablet copy session dfa1152987644bd781b46c7e42a7a685-61ab80ce377a44ac95a9c574e9a615f2 received from {username='slave'} at 127.11.42.67:38077
I20240502 01:54:16.238922 20253 tablet_copy_service.cc:433] P fdc2ac372492454c85387589a91f63af: ending tablet copy session dfa1152987644bd781b46c7e42a7a685-61ab80ce377a44ac95a9c574e9a615f2 on tablet 61ab80ce377a44ac95a9c574e9a615f2 with peer dfa1152987644bd781b46c7e42a7a685
I20240502 01:54:16.235311 24212 consensus_queue.cc:237] T 7efd29396c7c4fb597be2251cdcdd48f P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:16.247391 17328 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:16.247584 17328 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 01:54:16.248657 24918 consensus_queue.cc:237] T 43572b8f8fba433ab00b046c769d886d P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:16.252032 20345 tablet_service.cc:1514] Processing DeleteTablet for tablet 29d8cb76dc514e7b9002bd2291d8ee8d with delete_type TABLET_DATA_TOMBSTONED (TS 6a36c952c51542c0bf7ea3ec2857a9f9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40196
I20240502 01:54:16.252764 24988 tablet_replica.cc:330] T 29d8cb76dc514e7b9002bd2291d8ee8d P 6a36c952c51542c0bf7ea3ec2857a9f9: stopping tablet replica
I20240502 01:54:16.252985 24988 raft_consensus.cc:2229] T 29d8cb76dc514e7b9002bd2291d8ee8d P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:16.253217 24988 raft_consensus.cc:2258] T 29d8cb76dc514e7b9002bd2291d8ee8d P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:16.253448 17328 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: Sending DeleteTablet for 3 replicas of tablet 4ecf3b8482d04b37baf761305f933afb
I20240502 01:54:16.255234 17847 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ecf3b8482d04b37baf761305f933afb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:16 UTC) from {username='slave'} at 127.0.0.1:47226
I20240502 01:54:16.255646 24969 tablet_bootstrap.cc:492] T 8eed067f639542d487de971761e91931 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:54:16.255941 24988 ts_tablet_manager.cc:1875] T 29d8cb76dc514e7b9002bd2291d8ee8d P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:16.258318 17454 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ecf3b8482d04b37baf761305f933afb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:16 UTC) from {username='slave'} at 127.0.0.1:54520
I20240502 01:54:16.263837 20610 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ecf3b8482d04b37baf761305f933afb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 01:54:16 UTC) from {username='slave'} at 127.0.0.1:33698
I20240502 01:54:16.271697 17328 catalog_manager.cc:5484] T 43572b8f8fba433ab00b046c769d886d P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66). New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:16.280800 24988 ts_tablet_manager.cc:1888] T 29d8cb76dc514e7b9002bd2291d8ee8d P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:54:16.280963 24988 log.cc:1196] T 29d8cb76dc514e7b9002bd2291d8ee8d P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal/wals/29d8cb76dc514e7b9002bd2291d8ee8d
I20240502 01:54:16.281706 17321 catalog_manager.cc:4830] TS 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): tablet 29d8cb76dc514e7b9002bd2291d8ee8d (table rebalancer_extra_table_14 [id=54db4901eae145fbbc5c66c0719703f1]) successfully deleted
I20240502 01:54:16.289422 20497 raft_consensus.cc:1261] T 29d8cb76dc514e7b9002bd2291d8ee8d P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:16.290170 24797 consensus_queue.cc:1035] T 29d8cb76dc514e7b9002bd2291d8ee8d P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:16.290886 24969 tablet_bootstrap.cc:492] T 8eed067f639542d487de971761e91931 P dfa1152987644bd781b46c7e42a7a685: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:16.291251 24969 tablet_bootstrap.cc:492] T 8eed067f639542d487de971761e91931 P dfa1152987644bd781b46c7e42a7a685: Bootstrap complete.
I20240502 01:54:16.291945 24969 ts_tablet_manager.cc:1367] T 8eed067f639542d487de971761e91931 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.036s user 0.007s sys 0.000s
I20240502 01:54:16.292600 24969 raft_consensus.cc:348] T 8eed067f639542d487de971761e91931 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:54:16.292861 24969 raft_consensus.cc:729] T 8eed067f639542d487de971761e91931 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: LEARNER
I20240502 01:54:16.293066 24969 consensus_queue.cc:260] T 8eed067f639542d487de971761e91931 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:54:16.292784 20497 raft_consensus.cc:2932] T 29d8cb76dc514e7b9002bd2291d8ee8d P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } }
I20240502 01:54:16.293601 24969 ts_tablet_manager.cc:1398] T 8eed067f639542d487de971761e91931 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:54:16.294299 20387 tablet_copy_service.cc:342] P 6a36c952c51542c0bf7ea3ec2857a9f9: Request end of tablet copy session dfa1152987644bd781b46c7e42a7a685-8eed067f639542d487de971761e91931 received from {username='slave'} at 127.11.42.67:35525
I20240502 01:54:16.294441 20387 tablet_copy_service.cc:433] P 6a36c952c51542c0bf7ea3ec2857a9f9: ending tablet copy session dfa1152987644bd781b46c7e42a7a685-8eed067f639542d487de971761e91931 on tablet 8eed067f639542d487de971761e91931 with peer dfa1152987644bd781b46c7e42a7a685
I20240502 01:54:16.298388 25016 tablet_replica.cc:330] T 4ecf3b8482d04b37baf761305f933afb P f19cc6cb1e76460aa6711e49a5775091: stopping tablet replica
I20240502 01:54:16.298756 25016 raft_consensus.cc:2229] T 4ecf3b8482d04b37baf761305f933afb P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:16.298997 25016 raft_consensus.cc:2258] T 4ecf3b8482d04b37baf761305f933afb P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:16.300488 25014 tablet_replica.cc:330] T 4ecf3b8482d04b37baf761305f933afb P bb31bceba76d466db24b9341c04900c3: stopping tablet replica
I20240502 01:54:16.300755 25014 raft_consensus.cc:2229] T 4ecf3b8482d04b37baf761305f933afb P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 01:54:16.305186 25014 raft_consensus.cc:2258] T 4ecf3b8482d04b37baf761305f933afb P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:16.306730 25014 ts_tablet_manager.cc:1875] T 4ecf3b8482d04b37baf761305f933afb P bb31bceba76d466db24b9341c04900c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:16.306959 25016 ts_tablet_manager.cc:1875] T 4ecf3b8482d04b37baf761305f933afb P f19cc6cb1e76460aa6711e49a5775091: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:16.319775 25016 ts_tablet_manager.cc:1888] T 4ecf3b8482d04b37baf761305f933afb P f19cc6cb1e76460aa6711e49a5775091: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:16.320304 24811 raft_consensus.cc:482] T 25ad701a0fec4e80b37df5dec598796a P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:16.320456 24811 raft_consensus.cc:504] T 25ad701a0fec4e80b37df5dec598796a P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:16.321136 24811 leader_election.cc:288] T 25ad701a0fec4e80b37df5dec598796a P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:54:16.322235 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25ad701a0fec4e80b37df5dec598796a" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:54:16.322470 17475 raft_consensus.cc:2454] T 25ad701a0fec4e80b37df5dec598796a P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 0.
I20240502 01:54:16.322719 20497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25ad701a0fec4e80b37df5dec598796a" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
I20240502 01:54:16.322886 20167 leader_election.cc:302] T 25ad701a0fec4e80b37df5dec598796a P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:54:16.322995 20497 raft_consensus.cc:2454] T 25ad701a0fec4e80b37df5dec598796a P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 0.
I20240502 01:54:16.323210 24811 raft_consensus.cc:2781] T 25ad701a0fec4e80b37df5dec598796a P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:16.323325 24811 raft_consensus.cc:482] T 25ad701a0fec4e80b37df5dec598796a P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:16.323400 24811 raft_consensus.cc:3037] T 25ad701a0fec4e80b37df5dec598796a P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:16.324321 25015 tablet_replica.cc:330] T 4ecf3b8482d04b37baf761305f933afb P 4da690c0fa13407a85c0689ce8de9ade: stopping tablet replica
I20240502 01:54:16.324517 25015 raft_consensus.cc:2229] T 4ecf3b8482d04b37baf761305f933afb P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:16.324723 25015 raft_consensus.cc:2258] T 4ecf3b8482d04b37baf761305f933afb P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 01:54:16.324997 17412 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.42.68:44083, user_credentials={real_user=slave}} blocked reactor thread for 113675us
I20240502 01:54:16.326109 24811 raft_consensus.cc:504] T 25ad701a0fec4e80b37df5dec598796a P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:16.327157 24811 leader_election.cc:288] T 25ad701a0fec4e80b37df5dec598796a P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 election: Requested vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:54:16.328197 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25ad701a0fec4e80b37df5dec598796a" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:54:16.328393 17475 raft_consensus.cc:3037] T 25ad701a0fec4e80b37df5dec598796a P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:16.328738 17328 catalog_manager.cc:5484] T 7efd29396c7c4fb597be2251cdcdd48f P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: term changed from 0 to 1, leader changed from <none> to 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65). New cstate: current_term: 1 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:16.330025 20497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25ad701a0fec4e80b37df5dec598796a" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
I20240502 01:54:16.330168 25015 ts_tablet_manager.cc:1875] T 4ecf3b8482d04b37baf761305f933afb P 4da690c0fa13407a85c0689ce8de9ade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 01:54:16.330220 20497 raft_consensus.cc:3037] T 25ad701a0fec4e80b37df5dec598796a P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:16.331092 17475 raft_consensus.cc:2454] T 25ad701a0fec4e80b37df5dec598796a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 1.
I20240502 01:54:16.334722 20167 leader_election.cc:302] T 25ad701a0fec4e80b37df5dec598796a P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:54:16.335083 24814 raft_consensus.cc:2781] T 25ad701a0fec4e80b37df5dec598796a P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:16.335214 24814 raft_consensus.cc:686] T 25ad701a0fec4e80b37df5dec598796a P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Becoming Leader. State: Replica: fdc2ac372492454c85387589a91f63af, State: Running, Role: LEADER
I20240502 01:54:16.335480 24814 consensus_queue.cc:237] T 25ad701a0fec4e80b37df5dec598796a P fdc2ac372492454c85387589a91f63af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:16.342744 25014 ts_tablet_manager.cc:1888] T 4ecf3b8482d04b37baf761305f933afb P bb31bceba76d466db24b9341c04900c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:16.342929 25014 log.cc:1196] T 4ecf3b8482d04b37baf761305f933afb P bb31bceba76d466db24b9341c04900c3: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/wals/4ecf3b8482d04b37baf761305f933afb
I20240502 01:54:16.320156 25001 tablet_copy_client.cc:308] T 0618761d30f34cfe9650d13c9275cdcb P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.68:44083
I20240502 01:54:16.343169 25016 log.cc:1196] T 4ecf3b8482d04b37baf761305f933afb P f19cc6cb1e76460aa6711e49a5775091: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal/wals/4ecf3b8482d04b37baf761305f933afb
I20240502 01:54:16.343708 25016 ts_tablet_manager.cc:1909] T 4ecf3b8482d04b37baf761305f933afb P f19cc6cb1e76460aa6711e49a5775091: Deleting consensus metadata
I20240502 01:54:16.343967 17887 tablet_copy_service.cc:140] P bb31bceba76d466db24b9341c04900c3: Received BeginTabletCopySession request for tablet 0618761d30f34cfe9650d13c9275cdcb from peer f19cc6cb1e76460aa6711e49a5775091 ({username='slave'} at 127.11.42.72:53629)
I20240502 01:54:16.344122 17887 tablet_copy_service.cc:161] P bb31bceba76d466db24b9341c04900c3: Beginning new tablet copy session on tablet 0618761d30f34cfe9650d13c9275cdcb from peer f19cc6cb1e76460aa6711e49a5775091 at {username='slave'} at 127.11.42.72:53629: session id = f19cc6cb1e76460aa6711e49a5775091-0618761d30f34cfe9650d13c9275cdcb
I20240502 01:54:16.344158 25014 ts_tablet_manager.cc:1909] T 4ecf3b8482d04b37baf761305f933afb P bb31bceba76d466db24b9341c04900c3: Deleting consensus metadata
I20240502 01:54:16.345614 17322 catalog_manager.cc:4830] TS bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): tablet 4ecf3b8482d04b37baf761305f933afb (table rebalancer_dynamic_table_new_name [id=4193c1e64b834e889d5218c65ef0bba3]) successfully deleted
I20240502 01:54:16.346900 17887 tablet_copy_source_session.cc:205] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:16.348284 17326 catalog_manager.cc:5484] T 25ad701a0fec4e80b37df5dec598796a P fdc2ac372492454c85387589a91f63af reported cstate change: term changed from 0 to 1, leader changed from <none> to fdc2ac372492454c85387589a91f63af (127.11.42.69). New cstate: current_term: 1 leader_uuid: "fdc2ac372492454c85387589a91f63af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:16.351092 20497 raft_consensus.cc:2454] T 25ad701a0fec4e80b37df5dec598796a P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 1.
I20240502 01:54:16.352167 17321 catalog_manager.cc:4830] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): tablet 4ecf3b8482d04b37baf761305f933afb (table rebalancer_dynamic_table_new_name [id=4193c1e64b834e889d5218c65ef0bba3]) successfully deleted
I20240502 01:54:16.353430 25001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0618761d30f34cfe9650d13c9275cdcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:16.356010 17326 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 01:54:16.360379 25001 tablet_copy_client.cc:783] T 0618761d30f34cfe9650d13c9275cdcb P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:16.360944 25001 tablet_copy_client.cc:647] T 0618761d30f34cfe9650d13c9275cdcb P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:16.361101 20233 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f3e4c8f98b5e48e3ba6affc2df08943a"
dest_uuid: "fdc2ac372492454c85387589a91f63af"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48708
I20240502 01:54:16.361369 20233 raft_consensus.cc:595] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Received request to transfer leadership
I20240502 01:54:16.364063 25001 tablet_copy_client.cc:523] T 0618761d30f34cfe9650d13c9275cdcb P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:16.367754 25015 ts_tablet_manager.cc:1888] T 4ecf3b8482d04b37baf761305f933afb P 4da690c0fa13407a85c0689ce8de9ade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 01:54:16.369935 25015 log.cc:1196] T 4ecf3b8482d04b37baf761305f933afb P 4da690c0fa13407a85c0689ce8de9ade: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/wals/4ecf3b8482d04b37baf761305f933afb
I20240502 01:54:16.370227 17585 tablet_service.cc:1467] Processing CreateTablet for tablet 5824db6154734b0f8f9094ae46633771 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bcaa76f028574d38944cbe8a4de2342b]), partition=
I20240502 01:54:16.370278 20345 tablet_service.cc:1467] Processing CreateTablet for tablet 5824db6154734b0f8f9094ae46633771 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bcaa76f028574d38944cbe8a4de2342b]), partition=
I20240502 01:54:16.370486 25015 ts_tablet_manager.cc:1909] T 4ecf3b8482d04b37baf761305f933afb P 4da690c0fa13407a85c0689ce8de9ade: Deleting consensus metadata
I20240502 01:54:16.370589 17585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5824db6154734b0f8f9094ae46633771. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:16.370632 20345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5824db6154734b0f8f9094ae46633771. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:16.371547 25001 tablet_bootstrap.cc:492] T 0618761d30f34cfe9650d13c9275cdcb P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:54:16.374436 17321 catalog_manager.cc:4830] TS 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): tablet 4ecf3b8482d04b37baf761305f933afb (table rebalancer_dynamic_table_new_name [id=4193c1e64b834e889d5218c65ef0bba3]) successfully deleted
I20240502 01:54:16.375177 17715 tablet_service.cc:1467] Processing CreateTablet for tablet 5824db6154734b0f8f9094ae46633771 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bcaa76f028574d38944cbe8a4de2342b]), partition=
I20240502 01:54:16.375521 17715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5824db6154734b0f8f9094ae46633771. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:16.376734 24963 tablet_bootstrap.cc:492] T 5824db6154734b0f8f9094ae46633771 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:54:16.378441 24963 tablet_bootstrap.cc:654] T 5824db6154734b0f8f9094ae46633771 P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:16.382601 17737 raft_consensus.cc:1203] T 61ab80ce377a44ac95a9c574e9a615f2 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:16.391889 24963 tablet_bootstrap.cc:492] T 5824db6154734b0f8f9094ae46633771 P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:54:16.393779 24963 ts_tablet_manager.cc:1367] T 5824db6154734b0f8f9094ae46633771 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.004s
I20240502 01:54:16.394691 24963 raft_consensus.cc:348] T 5824db6154734b0f8f9094ae46633771 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:16.394975 24963 raft_consensus.cc:374] T 5824db6154734b0f8f9094ae46633771 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:16.395076 24963 raft_consensus.cc:729] T 5824db6154734b0f8f9094ae46633771 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:54:16.395419 24963 consensus_queue.cc:260] T 5824db6154734b0f8f9094ae46633771 P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:16.398262 24970 tablet_bootstrap.cc:492] T 5824db6154734b0f8f9094ae46633771 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:54:16.399787 24970 tablet_bootstrap.cc:654] T 5824db6154734b0f8f9094ae46633771 P 6a36c952c51542c0bf7ea3ec2857a9f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:16.400866 24813 raft_consensus.cc:979] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af: : Instructing follower f19cc6cb1e76460aa6711e49a5775091 to start an election
I20240502 01:54:16.401044 24964 tablet_bootstrap.cc:492] T 5824db6154734b0f8f9094ae46633771 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:54:16.401048 24813 raft_consensus.cc:1067] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Signalling peer f19cc6cb1e76460aa6711e49a5775091 to start an election
I20240502 01:54:16.402314 24964 tablet_bootstrap.cc:654] T 5824db6154734b0f8f9094ae46633771 P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:16.402380 20627 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f3e4c8f98b5e48e3ba6affc2df08943a"
dest_uuid: "f19cc6cb1e76460aa6711e49a5775091"
from {username='slave'} at 127.11.42.69:49895
I20240502 01:54:16.402580 20627 raft_consensus.cc:482] T f3e4c8f98b5e48e3ba6affc2df08943a P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:16.402704 20627 raft_consensus.cc:3037] T f3e4c8f98b5e48e3ba6affc2df08943a P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:16.407974 24970 tablet_bootstrap.cc:492] T 5824db6154734b0f8f9094ae46633771 P 6a36c952c51542c0bf7ea3ec2857a9f9: No bootstrap required, opened a new log
I20240502 01:54:16.408135 24970 ts_tablet_manager.cc:1367] T 5824db6154734b0f8f9094ae46633771 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 01:54:16.408859 24970 raft_consensus.cc:348] T 5824db6154734b0f8f9094ae46633771 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:16.409096 24970 raft_consensus.cc:374] T 5824db6154734b0f8f9094ae46633771 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:16.409173 24970 raft_consensus.cc:729] T 5824db6154734b0f8f9094ae46633771 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: FOLLOWER
I20240502 01:54:16.409299 20627 raft_consensus.cc:504] T f3e4c8f98b5e48e3ba6affc2df08943a P f19cc6cb1e76460aa6711e49a5775091 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } }
I20240502 01:54:16.409399 24970 consensus_queue.cc:260] T 5824db6154734b0f8f9094ae46633771 P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:16.409981 24970 ts_tablet_manager.cc:1398] T 5824db6154734b0f8f9094ae46633771 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:54:16.410780 17735 raft_consensus.cc:1203] T 8eed067f639542d487de971761e91931 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:16.410924 20233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3e4c8f98b5e48e3ba6affc2df08943a" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:54:16.411132 20233 raft_consensus.cc:3032] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:16.411253 20233 raft_consensus.cc:729] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Running, Role: LEADER
I20240502 01:54:16.411444 20497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3e4c8f98b5e48e3ba6affc2df08943a" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
I20240502 01:54:16.411651 20497 raft_consensus.cc:3037] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:16.411576 20233 consensus_queue.cc:260] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } }
I20240502 01:54:16.412021 20233 raft_consensus.cc:3037] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:16.412611 24964 tablet_bootstrap.cc:492] T 5824db6154734b0f8f9094ae46633771 P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:54:16.412727 24964 ts_tablet_manager.cc:1367] T 5824db6154734b0f8f9094ae46633771 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240502 01:54:16.413317 20364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3e4c8f98b5e48e3ba6affc2df08943a" candidate_uuid: "f19cc6cb1e76460aa6711e49a5775091" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
I20240502 01:54:16.413365 24964 raft_consensus.cc:348] T 5824db6154734b0f8f9094ae46633771 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:16.413489 20364 raft_consensus.cc:3037] T f3e4c8f98b5e48e3ba6affc2df08943a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:16.413524 24964 raft_consensus.cc:374] T 5824db6154734b0f8f9094ae46633771 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:16.413570 24964 raft_consensus.cc:729] T 5824db6154734b0f8f9094ae46633771 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:54:16.413713 20627 leader_election.cc:288] T f3e4c8f98b5e48e3ba6affc2df08943a P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 2 election: Requested vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:54:16.413729 24964 consensus_queue.cc:260] T 5824db6154734b0f8f9094ae46633771 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:16.414455 24964 ts_tablet_manager.cc:1398] T 5824db6154734b0f8f9094ae46633771 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:16.415217 20233 raft_consensus.cc:2454] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 2.
I20240502 01:54:16.416024 20364 raft_consensus.cc:2454] T f3e4c8f98b5e48e3ba6affc2df08943a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 2.
I20240502 01:54:16.420284 20564 leader_election.cc:302] T f3e4c8f98b5e48e3ba6affc2df08943a P f19cc6cb1e76460aa6711e49a5775091 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, f19cc6cb1e76460aa6711e49a5775091, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:54:16.420920 24962 raft_consensus.cc:2781] T f3e4c8f98b5e48e3ba6affc2df08943a P f19cc6cb1e76460aa6711e49a5775091 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:16.421311 24962 raft_consensus.cc:686] T f3e4c8f98b5e48e3ba6affc2df08943a P f19cc6cb1e76460aa6711e49a5775091 [term 2 LEADER]: Becoming Leader. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Running, Role: LEADER
I20240502 01:54:16.421696 24962 consensus_queue.cc:237] T f3e4c8f98b5e48e3ba6affc2df08943a P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } }
I20240502 01:54:16.430681 17326 catalog_manager.cc:5484] T f3e4c8f98b5e48e3ba6affc2df08943a P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: term changed from 1 to 2, leader changed from fdc2ac372492454c85387589a91f63af (127.11.42.69) to f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72). New cstate: current_term: 2 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:16.444120 24963 ts_tablet_manager.cc:1398] T 5824db6154734b0f8f9094ae46633771 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.050s user 0.001s sys 0.001s
I20240502 01:54:16.446136 24928 tablet_bootstrap.cc:492] T 71cbc012efe14fdb937302796a9f84e5 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:16.446712 24928 tablet_bootstrap.cc:492] T 71cbc012efe14fdb937302796a9f84e5 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap complete.
I20240502 01:54:16.457234 24928 ts_tablet_manager.cc:1367] T 71cbc012efe14fdb937302796a9f84e5 P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.247s user 0.053s sys 0.010s
I20240502 01:54:16.458091 24928 raft_consensus.cc:348] T 71cbc012efe14fdb937302796a9f84e5 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
I20240502 01:54:16.458436 24928 raft_consensus.cc:729] T 71cbc012efe14fdb937302796a9f84e5 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: LEARNER
I20240502 01:54:16.458709 24928 consensus_queue.cc:260] T 71cbc012efe14fdb937302796a9f84e5 P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
I20240502 01:54:16.460799 24928 ts_tablet_manager.cc:1398] T 71cbc012efe14fdb937302796a9f84e5 P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:54:16.462858 17757 tablet_copy_service.cc:342] P dfa1152987644bd781b46c7e42a7a685: Request end of tablet copy session 2f306549407b4bb281ccf2994035e4f1-71cbc012efe14fdb937302796a9f84e5 received from {username='slave'} at 127.11.42.71:49495
I20240502 01:54:16.462970 17757 tablet_copy_service.cc:433] P dfa1152987644bd781b46c7e42a7a685: ending tablet copy session 2f306549407b4bb281ccf2994035e4f1-71cbc012efe14fdb937302796a9f84e5 on tablet 71cbc012efe14fdb937302796a9f84e5 with peer 2f306549407b4bb281ccf2994035e4f1
I20240502 01:54:16.468365 20497 raft_consensus.cc:2454] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19cc6cb1e76460aa6711e49a5775091 in term 2.
I20240502 01:54:16.475749 25001 tablet_bootstrap.cc:492] T 0618761d30f34cfe9650d13c9275cdcb P f19cc6cb1e76460aa6711e49a5775091: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:16.476228 25001 tablet_bootstrap.cc:492] T 0618761d30f34cfe9650d13c9275cdcb P f19cc6cb1e76460aa6711e49a5775091: Bootstrap complete.
I20240502 01:54:16.476531 25001 ts_tablet_manager.cc:1367] T 0618761d30f34cfe9650d13c9275cdcb P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.105s user 0.041s sys 0.008s
I20240502 01:54:16.477409 25001 raft_consensus.cc:348] T 0618761d30f34cfe9650d13c9275cdcb P f19cc6cb1e76460aa6711e49a5775091 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:54:16.477624 25001 raft_consensus.cc:729] T 0618761d30f34cfe9650d13c9275cdcb P f19cc6cb1e76460aa6711e49a5775091 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: LEARNER
I20240502 01:54:16.477816 25001 consensus_queue.cc:260] T 0618761d30f34cfe9650d13c9275cdcb P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:54:16.478515 25001 ts_tablet_manager.cc:1398] T 0618761d30f34cfe9650d13c9275cdcb P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:16.480139 17887 tablet_copy_service.cc:342] P bb31bceba76d466db24b9341c04900c3: Request end of tablet copy session f19cc6cb1e76460aa6711e49a5775091-0618761d30f34cfe9650d13c9275cdcb received from {username='slave'} at 127.11.42.72:53629
I20240502 01:54:16.480250 17887 tablet_copy_service.cc:433] P bb31bceba76d466db24b9341c04900c3: ending tablet copy session f19cc6cb1e76460aa6711e49a5775091-0618761d30f34cfe9650d13c9275cdcb on tablet 0618761d30f34cfe9650d13c9275cdcb with peer f19cc6cb1e76460aa6711e49a5775091
I20240502 01:54:16.516134 20233 consensus_queue.cc:237] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:54:16.518790 20364 raft_consensus.cc:1261] T 88e68acb7de346f0ad165cadabd077be P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:54:16.519030 20167 consensus_peers.cc:485] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 88e68acb7de346f0ad165cadabd077be. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:16.519369 20497 raft_consensus.cc:1261] T 88e68acb7de346f0ad165cadabd077be P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:16.519359 25047 consensus_queue.cc:1035] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:16.519904 24814 consensus_queue.cc:1035] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:16.522104 24811 raft_consensus.cc:2932] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:16.522786 20364 raft_consensus.cc:2932] T 88e68acb7de346f0ad165cadabd077be P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:16.523128 20497 raft_consensus.cc:2932] T 88e68acb7de346f0ad165cadabd077be P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:16.529569 17328 catalog_manager.cc:5484] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af reported cstate change: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New cstate: current_term: 1 leader_uuid: "fdc2ac372492454c85387589a91f63af" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:16.649941 23908 raft_consensus.cc:482] T 5824db6154734b0f8f9094ae46633771 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:16.650141 23908 raft_consensus.cc:504] T 5824db6154734b0f8f9094ae46633771 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:16.650696 23908 leader_election.cc:288] T 5824db6154734b0f8f9094ae46633771 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:54:16.651352 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5824db6154734b0f8f9094ae46633771" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:54:16.651556 17737 raft_consensus.cc:2454] T 5824db6154734b0f8f9094ae46633771 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 0.
I20240502 01:54:16.651916 20302 leader_election.cc:302] T 5824db6154734b0f8f9094ae46633771 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:54:16.652168 23908 raft_consensus.cc:2781] T 5824db6154734b0f8f9094ae46633771 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:16.652302 23908 raft_consensus.cc:482] T 5824db6154734b0f8f9094ae46633771 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:16.652408 23908 raft_consensus.cc:3037] T 5824db6154734b0f8f9094ae46633771 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:16.654954 23908 raft_consensus.cc:504] T 5824db6154734b0f8f9094ae46633771 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:16.655619 23908 leader_election.cc:288] T 5824db6154734b0f8f9094ae46633771 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Requested vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:54:16.655916 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5824db6154734b0f8f9094ae46633771" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:54:16.655982 17605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5824db6154734b0f8f9094ae46633771" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:54:16.655982 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5824db6154734b0f8f9094ae46633771" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:54:16.656162 17604 raft_consensus.cc:2454] T 5824db6154734b0f8f9094ae46633771 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 0.
I20240502 01:54:16.656164 17737 raft_consensus.cc:3037] T 5824db6154734b0f8f9094ae46633771 P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:16.658613 17737 raft_consensus.cc:2454] T 5824db6154734b0f8f9094ae46633771 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 1.
I20240502 01:54:16.659246 20302 leader_election.cc:302] T 5824db6154734b0f8f9094ae46633771 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, dfa1152987644bd781b46c7e42a7a685; no voters: 3fceccefba274b1dbbd8216af8b0bc29
I20240502 01:54:16.659709 24817 raft_consensus.cc:2781] T 5824db6154734b0f8f9094ae46633771 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:16.659876 24817 raft_consensus.cc:686] T 5824db6154734b0f8f9094ae46633771 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Becoming Leader. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Running, Role: LEADER
I20240502 01:54:16.660148 24817 consensus_queue.cc:237] T 5824db6154734b0f8f9094ae46633771 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:16.664501 17333 catalog_manager.cc:5484] T 5824db6154734b0f8f9094ae46633771 P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70). New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:16.690286 24918 consensus_queue.cc:1035] T 43572b8f8fba433ab00b046c769d886d P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:16.694447 20233 raft_consensus.cc:3037] T 7efd29396c7c4fb597be2251cdcdd48f P fdc2ac372492454c85387589a91f63af [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:16.698062 25004 consensus_queue.cc:1035] T 7efd29396c7c4fb597be2251cdcdd48f P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:16.704747 24598 consensus_queue.cc:1035] T 43572b8f8fba433ab00b046c769d886d P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:16.705789 20627 raft_consensus.cc:1203] T 0618761d30f34cfe9650d13c9275cdcb P f19cc6cb1e76460aa6711e49a5775091 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240502 01:54:16.710951 24212 consensus_queue.cc:1035] T 7efd29396c7c4fb597be2251cdcdd48f P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:16.778268 17333 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 01:54:16.801265 25047 consensus_queue.cc:1035] T 25ad701a0fec4e80b37df5dec598796a P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:16.807974 20364 raft_consensus.cc:1261] T f3e4c8f98b5e48e3ba6affc2df08943a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:54:16.808584 24955 consensus_queue.cc:1035] T f3e4c8f98b5e48e3ba6affc2df08943a P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:16.811450 17604 raft_consensus.cc:3037] T 5824db6154734b0f8f9094ae46633771 P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:16.814414 25047 consensus_queue.cc:1035] T 25ad701a0fec4e80b37df5dec598796a P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:16.837821 17604 raft_consensus.cc:1261] T 5824db6154734b0f8f9094ae46633771 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:16.838461 24817 consensus_queue.cc:1035] T 5824db6154734b0f8f9094ae46633771 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:16.845304 17326 catalog_manager.cc:5576] TS 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531) does not have the latest schema for tablet 5824db6154734b0f8f9094ae46633771 (table rebalancer_dynamic_table [id=bcaa76f028574d38944cbe8a4de2342b]). Expected version 1 got 0
I20240502 01:54:16.845664 17737 raft_consensus.cc:1261] T 5824db6154734b0f8f9094ae46633771 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:16.846319 24817 consensus_queue.cc:1035] T 5824db6154734b0f8f9094ae46633771 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:16.858441 23611 raft_consensus.cc:1050] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9: attempting to promote NON_VOTER dfa1152987644bd781b46c7e42a7a685 to VOTER
I20240502 01:54:16.859664 23611 consensus_queue.cc:237] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:16.895519 17326 catalog_manager.cc:5576] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115) does not have the latest schema for tablet 5824db6154734b0f8f9094ae46633771 (table rebalancer_dynamic_table [id=bcaa76f028574d38944cbe8a4de2342b]). Expected version 1 got 0
I20240502 01:54:16.898177 20496 raft_consensus.cc:1203] T 71cbc012efe14fdb937302796a9f84e5 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 01:54:16.910838 17326 catalog_manager.cc:5576] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115) does not have the latest schema for tablet 5824db6154734b0f8f9094ae46633771 (table rebalancer_dynamic_table [id=bcaa76f028574d38944cbe8a4de2342b]). Expected version 1 got 0
I20240502 01:54:16.912628 17735 raft_consensus.cc:1261] T 8eed067f639542d487de971761e91931 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:16.913210 25136 consensus_queue.cc:1035] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:16.922362 24814 raft_consensus.cc:1050] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af: attempting to promote NON_VOTER dfa1152987644bd781b46c7e42a7a685 to VOTER
I20240502 01:54:16.923225 24814 consensus_queue.cc:237] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:16.928527 17328 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59784:
name: "rebalancer_extra_table_54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:16.939728 17847 tablet_service.cc:1467] Processing CreateTablet for tablet 0615786e80314766892988f3cff6a74d (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=4d9c9075126e4a0981cd79951b7f3c9c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:16.940188 17847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0615786e80314766892988f3cff6a74d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:16.940737 17585 tablet_service.cc:1467] Processing CreateTablet for tablet e61094fdf9324469afa7ecdc45fd55de (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=4d9c9075126e4a0981cd79951b7f3c9c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:16.941210 17585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e61094fdf9324469afa7ecdc45fd55de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:16.941274 17454 tablet_service.cc:1467] Processing CreateTablet for tablet 0615786e80314766892988f3cff6a74d (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=4d9c9075126e4a0981cd79951b7f3c9c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:16.941654 17454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0615786e80314766892988f3cff6a74d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:16.942747 17715 tablet_service.cc:1467] Processing CreateTablet for tablet e61094fdf9324469afa7ecdc45fd55de (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=4d9c9075126e4a0981cd79951b7f3c9c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:16.943109 17715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e61094fdf9324469afa7ecdc45fd55de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:16.946352 20610 tablet_service.cc:1467] Processing CreateTablet for tablet 0615786e80314766892988f3cff6a74d (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=4d9c9075126e4a0981cd79951b7f3c9c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:16.946774 20610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0615786e80314766892988f3cff6a74d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:16.948974 17456 tablet_service.cc:1467] Processing CreateTablet for tablet 1423ea912bfc4472acb2b6798b3a99e9 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=4d9c9075126e4a0981cd79951b7f3c9c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:16.949359 17456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1423ea912bfc4472acb2b6798b3a99e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:16.949723 20233 raft_consensus.cc:1261] T 8eed067f639542d487de971761e91931 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:16.951390 20627 raft_consensus.cc:1261] T 8eed067f639542d487de971761e91931 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:16.953102 23908 consensus_queue.cc:1035] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:16.954567 17845 tablet_service.cc:1467] Processing CreateTablet for tablet 1423ea912bfc4472acb2b6798b3a99e9 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=4d9c9075126e4a0981cd79951b7f3c9c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:16.954588 17846 tablet_service.cc:1467] Processing CreateTablet for tablet e61094fdf9324469afa7ecdc45fd55de (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=4d9c9075126e4a0981cd79951b7f3c9c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:16.954995 17845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1423ea912bfc4472acb2b6798b3a99e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:16.955212 17846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e61094fdf9324469afa7ecdc45fd55de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:16.957979 24963 tablet_bootstrap.cc:492] T e61094fdf9324469afa7ecdc45fd55de P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:54:16.959599 24963 tablet_bootstrap.cc:654] T e61094fdf9324469afa7ecdc45fd55de P 3fceccefba274b1dbbd8216af8b0bc29: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:16.960570 23908 consensus_queue.cc:1035] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:16.960741 20477 tablet_service.cc:1467] Processing CreateTablet for tablet 1423ea912bfc4472acb2b6798b3a99e9 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=4d9c9075126e4a0981cd79951b7f3c9c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:16.961154 20477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1423ea912bfc4472acb2b6798b3a99e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:16.986518 25151 tablet_bootstrap.cc:492] T 1423ea912bfc4472acb2b6798b3a99e9 P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:54:16.988718 25151 tablet_bootstrap.cc:654] T 1423ea912bfc4472acb2b6798b3a99e9 P 2f306549407b4bb281ccf2994035e4f1: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:16.990540 25143 tablet_bootstrap.cc:492] T e61094fdf9324469afa7ecdc45fd55de P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:54:16.993085 25148 tablet_bootstrap.cc:492] T e61094fdf9324469afa7ecdc45fd55de P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:54:16.994189 24963 tablet_bootstrap.cc:492] T e61094fdf9324469afa7ecdc45fd55de P 3fceccefba274b1dbbd8216af8b0bc29: No bootstrap required, opened a new log
I20240502 01:54:16.994268 20232 raft_consensus.cc:1261] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af [term 2 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:54:16.994613 25148 tablet_bootstrap.cc:654] T e61094fdf9324469afa7ecdc45fd55de P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:16.995023 24963 ts_tablet_manager.cc:1367] T e61094fdf9324469afa7ecdc45fd55de P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.009s
I20240502 01:54:16.994954 24955 consensus_queue.cc:1035] T f3e4c8f98b5e48e3ba6affc2df08943a P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:16.995913 25130 tablet.cc:1722] T 5824db6154734b0f8f9094ae46633771 P 3fceccefba274b1dbbd8216af8b0bc29: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:16.996222 25143 tablet_bootstrap.cc:654] T e61094fdf9324469afa7ecdc45fd55de P dfa1152987644bd781b46c7e42a7a685: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:16.996400 25130 tablet.cc:1620] T 5824db6154734b0f8f9094ae46633771 P 3fceccefba274b1dbbd8216af8b0bc29: MemRowSet was empty: no flush needed.
I20240502 01:54:16.997646 24963 raft_consensus.cc:348] T e61094fdf9324469afa7ecdc45fd55de P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:16.998374 25148 tablet_bootstrap.cc:492] T e61094fdf9324469afa7ecdc45fd55de P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:54:17.000715 25143 tablet_bootstrap.cc:492] T e61094fdf9324469afa7ecdc45fd55de P dfa1152987644bd781b46c7e42a7a685: No bootstrap required, opened a new log
I20240502 01:54:17.000924 25148 ts_tablet_manager.cc:1367] T e61094fdf9324469afa7ecdc45fd55de P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240502 01:54:17.000910 17326 catalog_manager.cc:5576] TS 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531) does not have the latest schema for tablet 5824db6154734b0f8f9094ae46633771 (table rebalancer_dynamic_table [id=bcaa76f028574d38944cbe8a4de2342b]). Expected version 1 got 0
I20240502 01:54:17.002058 25148 raft_consensus.cc:348] T e61094fdf9324469afa7ecdc45fd55de P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:17.002841 24963 raft_consensus.cc:374] T e61094fdf9324469afa7ecdc45fd55de P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:17.002985 24963 raft_consensus.cc:729] T e61094fdf9324469afa7ecdc45fd55de P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: FOLLOWER
I20240502 01:54:17.003271 24963 consensus_queue.cc:260] T e61094fdf9324469afa7ecdc45fd55de P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:17.003633 25143 ts_tablet_manager.cc:1367] T e61094fdf9324469afa7ecdc45fd55de P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20240502 01:54:17.004145 24963 ts_tablet_manager.cc:1398] T e61094fdf9324469afa7ecdc45fd55de P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 01:54:17.005414 25148 raft_consensus.cc:374] T e61094fdf9324469afa7ecdc45fd55de P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:17.005765 25148 raft_consensus.cc:729] T e61094fdf9324469afa7ecdc45fd55de P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:54:17.006384 25128 tablet.cc:1722] T 5824db6154734b0f8f9094ae46633771 P 6a36c952c51542c0bf7ea3ec2857a9f9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:17.006877 25128 tablet.cc:1620] T 5824db6154734b0f8f9094ae46633771 P 6a36c952c51542c0bf7ea3ec2857a9f9: MemRowSet was empty: no flush needed.
I20240502 01:54:17.006394 25148 consensus_queue.cc:260] T e61094fdf9324469afa7ecdc45fd55de P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:17.006811 25143 raft_consensus.cc:348] T e61094fdf9324469afa7ecdc45fd55de P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:17.010573 25144 tablet_bootstrap.cc:492] T 0615786e80314766892988f3cff6a74d P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:54:17.012499 25144 tablet_bootstrap.cc:654] T 0615786e80314766892988f3cff6a74d P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:17.015686 25143 raft_consensus.cc:374] T e61094fdf9324469afa7ecdc45fd55de P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:17.016049 25143 raft_consensus.cc:729] T e61094fdf9324469afa7ecdc45fd55de P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: FOLLOWER
I20240502 01:54:17.016832 25143 consensus_queue.cc:260] T e61094fdf9324469afa7ecdc45fd55de P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:17.015452 25148 ts_tablet_manager.cc:1398] T e61094fdf9324469afa7ecdc45fd55de P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20240502 01:54:17.018666 17333 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=bcaa76f028574d38944cbe8a4de2342b] alter complete (version 1)
I20240502 01:54:17.022759 25148 tablet_bootstrap.cc:492] T 0615786e80314766892988f3cff6a74d P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:54:17.025561 25143 ts_tablet_manager.cc:1398] T e61094fdf9324469afa7ecdc45fd55de P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20240502 01:54:17.027602 25148 tablet_bootstrap.cc:654] T 0615786e80314766892988f3cff6a74d P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:17.034535 25151 tablet_bootstrap.cc:492] T 1423ea912bfc4472acb2b6798b3a99e9 P 2f306549407b4bb281ccf2994035e4f1: No bootstrap required, opened a new log
I20240502 01:54:17.034605 25156 tablet.cc:1722] T 5824db6154734b0f8f9094ae46633771 P dfa1152987644bd781b46c7e42a7a685: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 01:54:17.035076 25156 tablet.cc:1620] T 5824db6154734b0f8f9094ae46633771 P dfa1152987644bd781b46c7e42a7a685: MemRowSet was empty: no flush needed.
I20240502 01:54:17.036509 25151 ts_tablet_manager.cc:1367] T 1423ea912bfc4472acb2b6798b3a99e9 P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.050s user 0.003s sys 0.004s
I20240502 01:54:17.040891 25151 raft_consensus.cc:348] T 1423ea912bfc4472acb2b6798b3a99e9 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:17.041811 25151 raft_consensus.cc:374] T 1423ea912bfc4472acb2b6798b3a99e9 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:17.042162 25151 raft_consensus.cc:729] T 1423ea912bfc4472acb2b6798b3a99e9 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: FOLLOWER
I20240502 01:54:17.042805 25151 consensus_queue.cc:260] T 1423ea912bfc4472acb2b6798b3a99e9 P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:17.043932 25148 tablet_bootstrap.cc:492] T 0615786e80314766892988f3cff6a74d P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:54:17.044798 25151 ts_tablet_manager.cc:1398] T 1423ea912bfc4472acb2b6798b3a99e9 P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 01:54:17.045092 25148 ts_tablet_manager.cc:1367] T 0615786e80314766892988f3cff6a74d P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20240502 01:54:17.045877 25148 raft_consensus.cc:348] T 0615786e80314766892988f3cff6a74d P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:17.046334 20496 raft_consensus.cc:1261] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:54:17.046861 25073 consensus_queue.cc:1035] T f3e4c8f98b5e48e3ba6affc2df08943a P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:17.047298 25148 raft_consensus.cc:374] T 0615786e80314766892988f3cff6a74d P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:17.047497 25148 raft_consensus.cc:729] T 0615786e80314766892988f3cff6a74d P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:54:17.047891 25148 consensus_queue.cc:260] T 0615786e80314766892988f3cff6a74d P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:17.052228 25148 ts_tablet_manager.cc:1398] T 0615786e80314766892988f3cff6a74d P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 01:54:17.052943 25148 tablet_bootstrap.cc:492] T 1423ea912bfc4472acb2b6798b3a99e9 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:54:17.054618 25148 tablet_bootstrap.cc:654] T 1423ea912bfc4472acb2b6798b3a99e9 P bb31bceba76d466db24b9341c04900c3: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:17.056774 17737 raft_consensus.cc:1261] T 61ab80ce377a44ac95a9c574e9a615f2 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:17.057971 24811 consensus_queue.cc:1035] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:17.057829 17867 raft_consensus.cc:1261] T 61ab80ce377a44ac95a9c574e9a615f2 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:17.060933 24744 raft_consensus.cc:482] T 0615786e80314766892988f3cff6a74d P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:17.063978 24744 raft_consensus.cc:504] T 0615786e80314766892988f3cff6a74d P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:17.064613 24744 leader_election.cc:288] T 0615786e80314766892988f3cff6a74d P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:54:17.067067 24811 consensus_queue.cc:1035] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:17.068189 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0615786e80314766892988f3cff6a74d" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:54:17.068928 20627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0615786e80314766892988f3cff6a74d" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091" is_pre_election: true
W20240502 01:54:17.068984 17802 leader_election.cc:341] T 0615786e80314766892988f3cff6a74d P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:17.068864 23908 raft_consensus.cc:2932] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
W20240502 01:54:17.070447 17802 leader_election.cc:341] T 0615786e80314766892988f3cff6a74d P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:17.070549 17802 leader_election.cc:302] T 0615786e80314766892988f3cff6a74d P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb31bceba76d466db24b9341c04900c3; no voters: 4da690c0fa13407a85c0689ce8de9ade, f19cc6cb1e76460aa6711e49a5775091
I20240502 01:54:17.070927 24744 raft_consensus.cc:2726] T 0615786e80314766892988f3cff6a74d P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:54:17.072883 20231 raft_consensus.cc:2932] T 8eed067f639542d487de971761e91931 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:17.079550 20627 raft_consensus.cc:2932] T 8eed067f639542d487de971761e91931 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:17.082696 20361 raft_consensus.cc:1261] T 61ab80ce377a44ac95a9c574e9a615f2 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:17.084827 17326 catalog_manager.cc:5484] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:17.085844 25047 consensus_queue.cc:1035] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:17.096082 25148 tablet_bootstrap.cc:492] T 1423ea912bfc4472acb2b6798b3a99e9 P bb31bceba76d466db24b9341c04900c3: No bootstrap required, opened a new log
I20240502 01:54:17.096290 25148 ts_tablet_manager.cc:1367] T 1423ea912bfc4472acb2b6798b3a99e9 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.044s user 0.006s sys 0.000s
I20240502 01:54:17.098531 24811 raft_consensus.cc:2932] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:17.099478 25147 tablet_bootstrap.cc:492] T 0615786e80314766892988f3cff6a74d P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
I20240502 01:54:17.103438 17737 raft_consensus.cc:2932] T 61ab80ce377a44ac95a9c574e9a615f2 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:17.097034 25148 raft_consensus.cc:348] T 1423ea912bfc4472acb2b6798b3a99e9 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:17.110610 25148 raft_consensus.cc:374] T 1423ea912bfc4472acb2b6798b3a99e9 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:17.110684 25148 raft_consensus.cc:729] T 1423ea912bfc4472acb2b6798b3a99e9 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: FOLLOWER
I20240502 01:54:17.110872 25148 consensus_queue.cc:260] T 1423ea912bfc4472acb2b6798b3a99e9 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:17.111433 25148 ts_tablet_manager.cc:1398] T 1423ea912bfc4472acb2b6798b3a99e9 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.015s user 0.001s sys 0.000s
I20240502 01:54:17.117172 25144 tablet_bootstrap.cc:492] T 0615786e80314766892988f3cff6a74d P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:54:17.117331 25144 ts_tablet_manager.cc:1367] T 0615786e80314766892988f3cff6a74d P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.107s user 0.024s sys 0.031s
I20240502 01:54:17.117880 17328 catalog_manager.cc:5484] T 61ab80ce377a44ac95a9c574e9a615f2 P dfa1152987644bd781b46c7e42a7a685 reported cstate change: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fdc2ac372492454c85387589a91f63af" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:17.118072 25144 raft_consensus.cc:348] T 0615786e80314766892988f3cff6a74d P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:17.118237 25144 raft_consensus.cc:374] T 0615786e80314766892988f3cff6a74d P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:17.118281 25144 raft_consensus.cc:729] T 0615786e80314766892988f3cff6a74d P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:54:17.119794 25144 consensus_queue.cc:260] T 0615786e80314766892988f3cff6a74d P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:17.122325 25144 ts_tablet_manager.cc:1398] T 0615786e80314766892988f3cff6a74d P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:54:17.124495 17326 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59784:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 01:54:17.125147 25144 tablet_bootstrap.cc:492] T 1423ea912bfc4472acb2b6798b3a99e9 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:54:17.128248 25144 tablet_bootstrap.cc:654] T 1423ea912bfc4472acb2b6798b3a99e9 P 4da690c0fa13407a85c0689ce8de9ade: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:17.131664 25140 ts_tablet_manager.cc:902] T 88e68acb7de346f0ad165cadabd077be P dfa1152987644bd781b46c7e42a7a685: Initiating tablet copy from peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:54:17.132393 17734 raft_consensus.cc:2932] T 8eed067f639542d487de971761e91931 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:17.146963 25147 tablet_bootstrap.cc:654] T 0615786e80314766892988f3cff6a74d P f19cc6cb1e76460aa6711e49a5775091: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:17.148757 24888 raft_consensus.cc:1050] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3: attempting to promote NON_VOTER f19cc6cb1e76460aa6711e49a5775091 to VOTER
I20240502 01:54:17.149626 24888 consensus_queue.cc:237] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } }
I20240502 01:54:17.149513 17867 raft_consensus.cc:2932] T 61ab80ce377a44ac95a9c574e9a615f2 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:17.155977 17326 catalog_manager.cc:5576] TS 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155) does not have the latest schema for tablet 5824db6154734b0f8f9094ae46633771 (table rebalancer_dynamic_table [id=bcaa76f028574d38944cbe8a4de2342b]). Expected version 2 got 1
I20240502 01:54:17.163870 25128 tablet.cc:1722] T 5824db6154734b0f8f9094ae46633771 P 6a36c952c51542c0bf7ea3ec2857a9f9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:17.166805 25147 tablet_bootstrap.cc:492] T 0615786e80314766892988f3cff6a74d P f19cc6cb1e76460aa6711e49a5775091: No bootstrap required, opened a new log
I20240502 01:54:17.166743 25156 tablet.cc:1722] T 5824db6154734b0f8f9094ae46633771 P dfa1152987644bd781b46c7e42a7a685: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:17.166971 17326 catalog_manager.cc:5576] TS 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531) does not have the latest schema for tablet 5824db6154734b0f8f9094ae46633771 (table rebalancer_dynamic_table [id=bcaa76f028574d38944cbe8a4de2342b]). Expected version 2 got 1
I20240502 01:54:17.167194 25156 tablet.cc:1620] T 5824db6154734b0f8f9094ae46633771 P dfa1152987644bd781b46c7e42a7a685: MemRowSet was empty: no flush needed.
I20240502 01:54:17.170619 20361 raft_consensus.cc:2932] T 61ab80ce377a44ac95a9c574e9a615f2 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:17.175504 25128 tablet.cc:1620] T 5824db6154734b0f8f9094ae46633771 P 6a36c952c51542c0bf7ea3ec2857a9f9: MemRowSet was empty: no flush needed.
I20240502 01:54:17.179348 25147 ts_tablet_manager.cc:1367] T 0615786e80314766892988f3cff6a74d P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.080s user 0.004s sys 0.000s
I20240502 01:54:17.181972 25147 raft_consensus.cc:348] T 0615786e80314766892988f3cff6a74d P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:17.182296 25147 raft_consensus.cc:374] T 0615786e80314766892988f3cff6a74d P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:17.182430 25147 raft_consensus.cc:729] T 0615786e80314766892988f3cff6a74d P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: FOLLOWER
I20240502 01:54:17.182693 25147 consensus_queue.cc:260] T 0615786e80314766892988f3cff6a74d P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:17.183385 25147 ts_tablet_manager.cc:1398] T 0615786e80314766892988f3cff6a74d P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:54:17.187152 17330 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b99aaf99a462480c9ce2bd089027d768: rebalancer_dynamic_table [id=bcaa76f028574d38944cbe8a4de2342b] alter complete (version 2)
I20240502 01:54:17.190892 25130 tablet.cc:1722] T 5824db6154734b0f8f9094ae46633771 P 3fceccefba274b1dbbd8216af8b0bc29: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 01:54:17.191401 25130 tablet.cc:1620] T 5824db6154734b0f8f9094ae46633771 P 3fceccefba274b1dbbd8216af8b0bc29: MemRowSet was empty: no flush needed.
I20240502 01:54:17.197928 25144 tablet_bootstrap.cc:492] T 1423ea912bfc4472acb2b6798b3a99e9 P 4da690c0fa13407a85c0689ce8de9ade: No bootstrap required, opened a new log
I20240502 01:54:17.198138 25144 ts_tablet_manager.cc:1367] T 1423ea912bfc4472acb2b6798b3a99e9 P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.073s user 0.004s sys 0.002s
I20240502 01:54:17.198966 25144 raft_consensus.cc:348] T 1423ea912bfc4472acb2b6798b3a99e9 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:17.199118 25140 tablet_copy_client.cc:308] T 88e68acb7de346f0ad165cadabd077be P dfa1152987644bd781b46c7e42a7a685: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.69:34339
I20240502 01:54:17.199869 20253 tablet_copy_service.cc:140] P fdc2ac372492454c85387589a91f63af: Received BeginTabletCopySession request for tablet 88e68acb7de346f0ad165cadabd077be from peer dfa1152987644bd781b46c7e42a7a685 ({username='slave'} at 127.11.42.67:38077)
I20240502 01:54:17.200057 20253 tablet_copy_service.cc:161] P fdc2ac372492454c85387589a91f63af: Beginning new tablet copy session on tablet 88e68acb7de346f0ad165cadabd077be from peer dfa1152987644bd781b46c7e42a7a685 at {username='slave'} at 127.11.42.67:38077: session id = dfa1152987644bd781b46c7e42a7a685-88e68acb7de346f0ad165cadabd077be
I20240502 01:54:17.201184 25144 raft_consensus.cc:374] T 1423ea912bfc4472acb2b6798b3a99e9 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:17.201256 25144 raft_consensus.cc:729] T 1423ea912bfc4472acb2b6798b3a99e9 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: FOLLOWER
I20240502 01:54:17.202877 20253 tablet_copy_source_session.cc:205] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:17.201460 25144 consensus_queue.cc:260] T 1423ea912bfc4472acb2b6798b3a99e9 P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:17.203768 25144 ts_tablet_manager.cc:1398] T 1423ea912bfc4472acb2b6798b3a99e9 P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 01:54:17.207219 24598 raft_consensus.cc:482] T e61094fdf9324469afa7ecdc45fd55de P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:17.207474 24598 raft_consensus.cc:504] T e61094fdf9324469afa7ecdc45fd55de P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:17.208082 25140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88e68acb7de346f0ad165cadabd077be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:17.208209 24598 leader_election.cc:288] T e61094fdf9324469afa7ecdc45fd55de P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:54:17.209193 20627 raft_consensus.cc:1261] T 0618761d30f34cfe9650d13c9275cdcb P f19cc6cb1e76460aa6711e49a5775091 [term 2 LEARNER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:54:17.209731 24744 consensus_queue.cc:1035] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:17.210736 17864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61094fdf9324469afa7ecdc45fd55de" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:54:17.210960 17864 raft_consensus.cc:2454] T e61094fdf9324469afa7ecdc45fd55de P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 0.
I20240502 01:54:17.211347 17543 leader_election.cc:302] T e61094fdf9324469afa7ecdc45fd55de P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:54:17.211642 24918 raft_consensus.cc:2781] T e61094fdf9324469afa7ecdc45fd55de P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:17.211735 24918 raft_consensus.cc:482] T e61094fdf9324469afa7ecdc45fd55de P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:17.211825 24918 raft_consensus.cc:3037] T e61094fdf9324469afa7ecdc45fd55de P 3fceccefba274b1dbbd8216af8b0bc29 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:17.212904 20231 raft_consensus.cc:1261] T 0618761d30f34cfe9650d13c9275cdcb P fdc2ac372492454c85387589a91f63af [term 2 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:54:17.213979 17735 raft_consensus.cc:1261] T 0618761d30f34cfe9650d13c9275cdcb P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:54:17.214510 24918 raft_consensus.cc:504] T e61094fdf9324469afa7ecdc45fd55de P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:17.215893 24918 leader_election.cc:288] T e61094fdf9324469afa7ecdc45fd55de P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Requested vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:54:17.216326 17864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61094fdf9324469afa7ecdc45fd55de" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:54:17.216491 17864 raft_consensus.cc:3037] T e61094fdf9324469afa7ecdc45fd55de P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:17.219134 17864 raft_consensus.cc:2454] T e61094fdf9324469afa7ecdc45fd55de P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:54:17.219446 24744 consensus_queue.cc:1035] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:17.219925 24744 consensus_queue.cc:1035] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:17.221210 17543 leader_election.cc:302] T e61094fdf9324469afa7ecdc45fd55de P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:54:17.222077 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61094fdf9324469afa7ecdc45fd55de" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:54:17.222330 17735 raft_consensus.cc:3037] T e61094fdf9324469afa7ecdc45fd55de P dfa1152987644bd781b46c7e42a7a685 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:17.225301 17735 raft_consensus.cc:2454] T e61094fdf9324469afa7ecdc45fd55de P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:54:17.225739 24918 raft_consensus.cc:2781] T e61094fdf9324469afa7ecdc45fd55de P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:17.225944 24918 raft_consensus.cc:686] T e61094fdf9324469afa7ecdc45fd55de P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Becoming Leader. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Running, Role: LEADER
I20240502 01:54:17.226240 24918 consensus_queue.cc:237] T e61094fdf9324469afa7ecdc45fd55de P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:17.232379 25140 tablet_copy_client.cc:783] T 88e68acb7de346f0ad165cadabd077be P dfa1152987644bd781b46c7e42a7a685: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:17.232730 25140 tablet_copy_client.cc:647] T 88e68acb7de346f0ad165cadabd077be P dfa1152987644bd781b46c7e42a7a685: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:17.237066 25140 tablet_copy_client.cc:523] T 88e68acb7de346f0ad165cadabd077be P dfa1152987644bd781b46c7e42a7a685: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:17.239605 25174 raft_consensus.cc:2932] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:17.240670 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61094fdf9324469afa7ecdc45fd55de" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa1152987644bd781b46c7e42a7a685" is_pre_election: true
I20240502 01:54:17.240862 17737 raft_consensus.cc:2362] T e61094fdf9324469afa7ecdc45fd55de P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1. Re-sending same reply.
I20240502 01:54:17.242125 25140 tablet_bootstrap.cc:492] T 88e68acb7de346f0ad165cadabd077be P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:54:17.244757 17327 catalog_manager.cc:5484] T e61094fdf9324469afa7ecdc45fd55de P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66). New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:17.266225 20361 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8eed067f639542d487de971761e91931"
dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56720
I20240502 01:54:17.266535 20361 raft_consensus.cc:595] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:54:17.270993 20627 consensus_queue.cc:237] T f3e4c8f98b5e48e3ba6affc2df08943a P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } }
I20240502 01:54:17.272720 20496 raft_consensus.cc:1261] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:54:17.275537 20625 raft_consensus.cc:2932] T 0618761d30f34cfe9650d13c9275cdcb P f19cc6cb1e76460aa6711e49a5775091 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:17.276073 20231 raft_consensus.cc:2932] T 0618761d30f34cfe9650d13c9275cdcb P fdc2ac372492454c85387589a91f63af [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:17.285598 24955 consensus_queue.cc:1035] T f3e4c8f98b5e48e3ba6affc2df08943a P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:17.287531 17330 catalog_manager.cc:5484] T 0618761d30f34cfe9650d13c9275cdcb P fdc2ac372492454c85387589a91f63af reported cstate change: config changed from index 8 to 9, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:17.288095 25073 raft_consensus.cc:2932] T f3e4c8f98b5e48e3ba6affc2df08943a P f19cc6cb1e76460aa6711e49a5775091 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:17.288820 20496 raft_consensus.cc:2932] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:17.296700 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f3e4c8f98b5e48e3ba6affc2df08943a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:17.300175 20361 raft_consensus.cc:1261] T f3e4c8f98b5e48e3ba6affc2df08943a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:54:17.300747 24962 consensus_queue.cc:1035] T f3e4c8f98b5e48e3ba6affc2df08943a P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:17.302351 20361 raft_consensus.cc:2932] T f3e4c8f98b5e48e3ba6affc2df08943a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:17.312425 17867 consensus_queue.cc:237] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } }
I20240502 01:54:17.315970 20231 raft_consensus.cc:1261] T 0618761d30f34cfe9650d13c9275cdcb P fdc2ac372492454c85387589a91f63af [term 2 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:54:17.316536 25174 consensus_queue.cc:1035] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:54:17.310113 17334 catalog_manager.cc:5484] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: config changed from index 3 to 5, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New cstate: current_term: 2 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:17.323043 25174 raft_consensus.cc:2932] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:17.324122 20231 raft_consensus.cc:2932] T 0618761d30f34cfe9650d13c9275cdcb P fdc2ac372492454c85387589a91f63af [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:17.325515 17736 raft_consensus.cc:2932] T 0618761d30f34cfe9650d13c9275cdcb P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:17.329107 25140 tablet_bootstrap.cc:492] T 88e68acb7de346f0ad165cadabd077be P dfa1152987644bd781b46c7e42a7a685: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:17.329524 25140 tablet_bootstrap.cc:492] T 88e68acb7de346f0ad165cadabd077be P dfa1152987644bd781b46c7e42a7a685: Bootstrap complete.
I20240502 01:54:17.329742 25140 ts_tablet_manager.cc:1367] T 88e68acb7de346f0ad165cadabd077be P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.088s user 0.007s sys 0.000s
I20240502 01:54:17.330363 20627 raft_consensus.cc:1261] T 0618761d30f34cfe9650d13c9275cdcb P f19cc6cb1e76460aa6711e49a5775091 [term 2 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:54:17.330415 25140 raft_consensus.cc:348] T 88e68acb7de346f0ad165cadabd077be P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:54:17.330603 25140 raft_consensus.cc:729] T 88e68acb7de346f0ad165cadabd077be P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: LEARNER
I20240502 01:54:17.330811 25140 consensus_queue.cc:260] T 88e68acb7de346f0ad165cadabd077be P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:54:17.335459 24212 raft_consensus.cc:482] T 0615786e80314766892988f3cff6a74d P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:17.335610 24212 raft_consensus.cc:504] T 0615786e80314766892988f3cff6a74d P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:17.336043 24888 consensus_queue.cc:1035] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:54:17.336375 24212 leader_election.cc:288] T 0615786e80314766892988f3cff6a74d P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:54:17.337945 17867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0615786e80314766892988f3cff6a74d" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:54:17.338027 17322 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0618761d30f34cfe9650d13c9275cdcb with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:17.338132 17867 raft_consensus.cc:2454] T 0615786e80314766892988f3cff6a74d P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 0.
I20240502 01:54:17.338517 17412 leader_election.cc:302] T 0615786e80314766892988f3cff6a74d P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:54:17.338950 24797 raft_consensus.cc:1050] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685: attempting to promote NON_VOTER 2f306549407b4bb281ccf2994035e4f1 to VOTER
I20240502 01:54:17.338949 24212 raft_consensus.cc:2781] T 0615786e80314766892988f3cff6a74d P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:17.339025 24212 raft_consensus.cc:482] T 0615786e80314766892988f3cff6a74d P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:17.339082 24212 raft_consensus.cc:3037] T 0615786e80314766892988f3cff6a74d P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:17.339720 24797 consensus_queue.cc:237] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } }
I20240502 01:54:17.340166 20625 raft_consensus.cc:2932] T 0618761d30f34cfe9650d13c9275cdcb P f19cc6cb1e76460aa6711e49a5775091 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } }
I20240502 01:54:17.342149 20627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0615786e80314766892988f3cff6a74d" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091" is_pre_election: true
I20240502 01:54:17.342387 20627 raft_consensus.cc:2454] T 0615786e80314766892988f3cff6a74d P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 0.
I20240502 01:54:17.343407 17326 catalog_manager.cc:5484] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 reported cstate change: config changed from index 9 to 10, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New cstate: current_term: 2 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:17.344137 24212 raft_consensus.cc:504] T 0615786e80314766892988f3cff6a74d P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:17.345234 20627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0615786e80314766892988f3cff6a74d" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19cc6cb1e76460aa6711e49a5775091"
I20240502 01:54:17.345448 20627 raft_consensus.cc:3037] T 0615786e80314766892988f3cff6a74d P f19cc6cb1e76460aa6711e49a5775091 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:17.348143 17867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0615786e80314766892988f3cff6a74d" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:54:17.348299 17715 tablet_service.cc:1514] Processing DeleteTablet for tablet 0618761d30f34cfe9650d13c9275cdcb with delete_type TABLET_DATA_TOMBSTONED (TS dfa1152987644bd781b46c7e42a7a685 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:52198
I20240502 01:54:17.348409 17867 raft_consensus.cc:3037] T 0615786e80314766892988f3cff6a74d P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:17.349833 25140 ts_tablet_manager.cc:1398] T 88e68acb7de346f0ad165cadabd077be P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20240502 01:54:17.350541 20253 tablet_copy_service.cc:342] P fdc2ac372492454c85387589a91f63af: Request end of tablet copy session dfa1152987644bd781b46c7e42a7a685-88e68acb7de346f0ad165cadabd077be received from {username='slave'} at 127.11.42.67:38077
I20240502 01:54:17.350663 20253 tablet_copy_service.cc:433] P fdc2ac372492454c85387589a91f63af: ending tablet copy session dfa1152987644bd781b46c7e42a7a685-88e68acb7de346f0ad165cadabd077be on tablet 88e68acb7de346f0ad165cadabd077be with peer dfa1152987644bd781b46c7e42a7a685
I20240502 01:54:17.351048 17867 raft_consensus.cc:2454] T 0615786e80314766892988f3cff6a74d P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
I20240502 01:54:17.351549 24212 leader_election.cc:288] T 0615786e80314766892988f3cff6a74d P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Requested vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:54:17.351990 17412 leader_election.cc:302] T 0615786e80314766892988f3cff6a74d P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:54:17.352329 24212 raft_consensus.cc:2781] T 0615786e80314766892988f3cff6a74d P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:17.352813 24212 raft_consensus.cc:686] T 0615786e80314766892988f3cff6a74d P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Becoming Leader. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Running, Role: LEADER
I20240502 01:54:17.353140 24212 consensus_queue.cc:237] T 0615786e80314766892988f3cff6a74d P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:17.358484 20364 raft_consensus.cc:1261] T 71cbc012efe14fdb937302796a9f84e5 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:17.359299 17475 raft_consensus.cc:1261] T 71cbc012efe14fdb937302796a9f84e5 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:17.359908 25000 consensus_queue.cc:1035] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:17.360401 25000 consensus_queue.cc:1035] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:17.361347 20627 raft_consensus.cc:2454] T 0615786e80314766892988f3cff6a74d P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
I20240502 01:54:17.363204 20496 raft_consensus.cc:1261] T 71cbc012efe14fdb937302796a9f84e5 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:17.363610 20211 tablet_service.cc:1514] Processing DeleteTablet for tablet f3e4c8f98b5e48e3ba6affc2df08943a with delete_type TABLET_DATA_TOMBSTONED (TS fdc2ac372492454c85387589a91f63af not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44624
I20240502 01:54:17.364876 24999 consensus_queue.cc:1035] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:17.368659 25193 tablet_replica.cc:330] T 0618761d30f34cfe9650d13c9275cdcb P dfa1152987644bd781b46c7e42a7a685: stopping tablet replica
I20240502 01:54:17.368968 25193 raft_consensus.cc:2229] T 0618761d30f34cfe9650d13c9275cdcb P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:17.369364 25193 raft_consensus.cc:2258] T 0618761d30f34cfe9650d13c9275cdcb P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:17.370711 25193 ts_tablet_manager.cc:1875] T 0618761d30f34cfe9650d13c9275cdcb P dfa1152987644bd781b46c7e42a7a685: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:17.371560 25198 tablet_replica.cc:330] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af: stopping tablet replica
I20240502 01:54:17.372388 25198 raft_consensus.cc:2229] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:17.372920 25198 raft_consensus.cc:2258] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:17.374579 25198 ts_tablet_manager.cc:1875] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:17.381656 25193 ts_tablet_manager.cc:1888] T 0618761d30f34cfe9650d13c9275cdcb P dfa1152987644bd781b46c7e42a7a685: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 01:54:17.381836 25193 log.cc:1196] T 0618761d30f34cfe9650d13c9275cdcb P dfa1152987644bd781b46c7e42a7a685: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/wals/0618761d30f34cfe9650d13c9275cdcb
I20240502 01:54:17.382645 17321 catalog_manager.cc:4830] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): tablet 0618761d30f34cfe9650d13c9275cdcb (table rebalance_test_table_15 [id=21fe1df2eeed4efd801d10edf592ecce]) successfully deleted
I20240502 01:54:17.385681 25198 ts_tablet_manager.cc:1888] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 01:54:17.385805 25198 log.cc:1196] T f3e4c8f98b5e48e3ba6affc2df08943a P fdc2ac372492454c85387589a91f63af: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal/wals/f3e4c8f98b5e48e3ba6affc2df08943a
I20240502 01:54:17.386687 24959 raft_consensus.cc:482] T 1423ea912bfc4472acb2b6798b3a99e9 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:17.386883 24959 raft_consensus.cc:504] T 1423ea912bfc4472acb2b6798b3a99e9 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:17.387677 24959 leader_election.cc:288] T 1423ea912bfc4472acb2b6798b3a99e9 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:54:17.388463 17476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1423ea912bfc4472acb2b6798b3a99e9" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:54:17.388705 17476 raft_consensus.cc:2454] T 1423ea912bfc4472acb2b6798b3a99e9 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 0.
I20240502 01:54:17.388885 17322 catalog_manager.cc:4830] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): tablet f3e4c8f98b5e48e3ba6affc2df08943a (table rebalancer_extra_table_24 [id=6077652d45454150b5c7d7072780d895]) successfully deleted
I20240502 01:54:17.389310 20433 leader_election.cc:302] T 1423ea912bfc4472acb2b6798b3a99e9 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 4da690c0fa13407a85c0689ce8de9ade; no voters:
I20240502 01:54:17.389284 17867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1423ea912bfc4472acb2b6798b3a99e9" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:54:17.389504 17867 raft_consensus.cc:2454] T 1423ea912bfc4472acb2b6798b3a99e9 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 0.
I20240502 01:54:17.389780 24959 raft_consensus.cc:2781] T 1423ea912bfc4472acb2b6798b3a99e9 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:17.389874 24959 raft_consensus.cc:482] T 1423ea912bfc4472acb2b6798b3a99e9 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:17.389977 24959 raft_consensus.cc:3037] T 1423ea912bfc4472acb2b6798b3a99e9 P 2f306549407b4bb281ccf2994035e4f1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:17.393406 24959 raft_consensus.cc:504] T 1423ea912bfc4472acb2b6798b3a99e9 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:17.394675 17476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1423ea912bfc4472acb2b6798b3a99e9" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:54:17.394958 17476 raft_consensus.cc:3037] T 1423ea912bfc4472acb2b6798b3a99e9 P 4da690c0fa13407a85c0689ce8de9ade [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:17.398402 17476 raft_consensus.cc:2454] T 1423ea912bfc4472acb2b6798b3a99e9 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 1.
I20240502 01:54:17.398993 20433 leader_election.cc:302] T 1423ea912bfc4472acb2b6798b3a99e9 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 4da690c0fa13407a85c0689ce8de9ade; no voters:
I20240502 01:54:17.399282 24957 raft_consensus.cc:2781] T 1423ea912bfc4472acb2b6798b3a99e9 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:17.401321 24957 raft_consensus.cc:686] T 1423ea912bfc4472acb2b6798b3a99e9 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Running, Role: LEADER
I20240502 01:54:17.401656 24957 consensus_queue.cc:237] T 1423ea912bfc4472acb2b6798b3a99e9 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:17.410678 24959 leader_election.cc:288] T 1423ea912bfc4472acb2b6798b3a99e9 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 1 election: Requested vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:54:17.411922 17867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1423ea912bfc4472acb2b6798b3a99e9" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:54:17.412108 17867 raft_consensus.cc:3037] T 1423ea912bfc4472acb2b6798b3a99e9 P bb31bceba76d466db24b9341c04900c3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:17.413825 17327 catalog_manager.cc:5484] T 1423ea912bfc4472acb2b6798b3a99e9 P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71). New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:17.414759 17867 raft_consensus.cc:2454] T 1423ea912bfc4472acb2b6798b3a99e9 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 1.
I20240502 01:54:17.433828 20364 raft_consensus.cc:2932] T 71cbc012efe14fdb937302796a9f84e5 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:17.432961 24999 raft_consensus.cc:2932] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:17.438591 17736 raft_consensus.cc:1203] T 88e68acb7de346f0ad165cadabd077be P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:17.446894 17333 catalog_manager.cc:5484] T 71cbc012efe14fdb937302796a9f84e5 P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: config changed from index 7 to 8, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:17.453470 17475 raft_consensus.cc:2932] T 71cbc012efe14fdb937302796a9f84e5 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:17.456097 17327 catalog_manager.cc:5484] T 0615786e80314766892988f3cff6a74d P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: term changed from 0 to 1, leader changed from <none> to 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65). New cstate: current_term: 1 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:17.460386 20496 raft_consensus.cc:2932] T 71cbc012efe14fdb937302796a9f84e5 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:17.496099 24817 raft_consensus.cc:979] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9: : Instructing follower fdc2ac372492454c85387589a91f63af to start an election
I20240502 01:54:17.496270 24817 raft_consensus.cc:1067] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Signalling peer fdc2ac372492454c85387589a91f63af to start an election
I20240502 01:54:17.497152 20231 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8eed067f639542d487de971761e91931"
dest_uuid: "fdc2ac372492454c85387589a91f63af"
from {username='slave'} at 127.11.42.70:59563
I20240502 01:54:17.497507 20231 raft_consensus.cc:482] T 8eed067f639542d487de971761e91931 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:17.497742 20231 raft_consensus.cc:3037] T 8eed067f639542d487de971761e91931 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:17.500488 20231 raft_consensus.cc:504] T 8eed067f639542d487de971761e91931 P fdc2ac372492454c85387589a91f63af [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:17.501672 20361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eed067f639542d487de971761e91931" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
I20240502 01:54:17.501783 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eed067f639542d487de971761e91931" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:54:17.501781 20627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eed067f639542d487de971761e91931" candidate_uuid: "fdc2ac372492454c85387589a91f63af" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "f19cc6cb1e76460aa6711e49a5775091"
I20240502 01:54:17.501904 20361 raft_consensus.cc:3032] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:17.501987 20361 raft_consensus.cc:729] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Running, Role: LEADER
I20240502 01:54:17.502009 17736 raft_consensus.cc:3037] T 8eed067f639542d487de971761e91931 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:17.502017 20627 raft_consensus.cc:3037] T 8eed067f639542d487de971761e91931 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:17.502265 20361 consensus_queue.cc:260] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:17.502651 20361 raft_consensus.cc:3037] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:17.503844 20627 raft_consensus.cc:2454] T 8eed067f639542d487de971761e91931 P f19cc6cb1e76460aa6711e49a5775091 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 2.
I20240502 01:54:17.504642 17736 raft_consensus.cc:2454] T 8eed067f639542d487de971761e91931 P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 2.
I20240502 01:54:17.501399 20231 leader_election.cc:288] T 8eed067f639542d487de971761e91931 P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 2 election: Requested vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:54:17.505154 20167 leader_election.cc:302] T 8eed067f639542d487de971761e91931 P fdc2ac372492454c85387589a91f63af [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: dfa1152987644bd781b46c7e42a7a685, f19cc6cb1e76460aa6711e49a5775091, fdc2ac372492454c85387589a91f63af; no voters:
I20240502 01:54:17.505465 25047 raft_consensus.cc:2781] T 8eed067f639542d487de971761e91931 P fdc2ac372492454c85387589a91f63af [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:17.505617 25047 raft_consensus.cc:686] T 8eed067f639542d487de971761e91931 P fdc2ac372492454c85387589a91f63af [term 2 LEADER]: Becoming Leader. State: Replica: fdc2ac372492454c85387589a91f63af, State: Running, Role: LEADER
I20240502 01:54:17.505952 20361 raft_consensus.cc:2454] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdc2ac372492454c85387589a91f63af in term 2.
I20240502 01:54:17.505959 25047 consensus_queue.cc:237] T 8eed067f639542d487de971761e91931 P fdc2ac372492454c85387589a91f63af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:17.511227 17333 catalog_manager.cc:5484] T 8eed067f639542d487de971761e91931 P fdc2ac372492454c85387589a91f63af reported cstate change: term changed from 1 to 2, leader changed from 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) to fdc2ac372492454c85387589a91f63af (127.11.42.69). New cstate: current_term: 2 leader_uuid: "fdc2ac372492454c85387589a91f63af" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:17.534436 20231 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "61ab80ce377a44ac95a9c574e9a615f2"
dest_uuid: "fdc2ac372492454c85387589a91f63af"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43048
I20240502 01:54:17.534662 20231 raft_consensus.cc:595] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Received request to transfer leadership
I20240502 01:54:17.552817 17737 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "71cbc012efe14fdb937302796a9f84e5"
dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57664
I20240502 01:54:17.552975 17737 raft_consensus.cc:595] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:54:17.621498 24598 consensus_queue.cc:1035] T e61094fdf9324469afa7ecdc45fd55de P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:17.627732 24918 consensus_queue.cc:1035] T e61094fdf9324469afa7ecdc45fd55de P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:17.644989 17867 consensus_queue.cc:237] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } }
I20240502 01:54:17.657935 20496 raft_consensus.cc:1261] T 03bd47783e884a70bfea1dfab4f2e563 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 01:54:17.659030 17802 consensus_peers.cc:485] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 -> Peer 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): Couldn't send request to peer 6a36c952c51542c0bf7ea3ec2857a9f9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 03bd47783e884a70bfea1dfab4f2e563. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:17.659201 25187 consensus_queue.cc:1035] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:17.661247 17737 raft_consensus.cc:1261] T 03bd47783e884a70bfea1dfab4f2e563 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:17.661798 25174 consensus_queue.cc:1035] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:17.662230 25173 raft_consensus.cc:2932] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } } }
I20240502 01:54:17.662864 20496 raft_consensus.cc:2932] T 03bd47783e884a70bfea1dfab4f2e563 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } } }
I20240502 01:54:17.670848 17737 raft_consensus.cc:2932] T 03bd47783e884a70bfea1dfab4f2e563 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } } }
I20240502 01:54:17.671998 17327 catalog_manager.cc:5484] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 reported cstate change: config changed from index 6 to 7, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:17.722306 25047 raft_consensus.cc:979] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af: : Instructing follower 6a36c952c51542c0bf7ea3ec2857a9f9 to start an election
I20240502 01:54:17.722476 25047 raft_consensus.cc:1067] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Signalling peer 6a36c952c51542c0bf7ea3ec2857a9f9 to start an election
I20240502 01:54:17.723361 20361 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "61ab80ce377a44ac95a9c574e9a615f2"
dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
from {username='slave'} at 127.11.42.69:50257
I20240502 01:54:17.723560 20361 raft_consensus.cc:482] T 61ab80ce377a44ac95a9c574e9a615f2 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:17.723680 20361 raft_consensus.cc:3037] T 61ab80ce377a44ac95a9c574e9a615f2 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:17.726461 20361 raft_consensus.cc:504] T 61ab80ce377a44ac95a9c574e9a615f2 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:17.730077 17867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61ab80ce377a44ac95a9c574e9a615f2" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:54:17.730279 17867 raft_consensus.cc:3037] T 61ab80ce377a44ac95a9c574e9a615f2 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:17.735819 17867 raft_consensus.cc:2454] T 61ab80ce377a44ac95a9c574e9a615f2 P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 2.
I20240502 01:54:17.736932 20232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61ab80ce377a44ac95a9c574e9a615f2" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "fdc2ac372492454c85387589a91f63af"
I20240502 01:54:17.737036 20361 leader_election.cc:288] T 61ab80ce377a44ac95a9c574e9a615f2 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 election: Requested vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:54:17.737156 20232 raft_consensus.cc:3032] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:17.737260 20232 raft_consensus.cc:729] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Running, Role: LEADER
I20240502 01:54:17.737545 20232 consensus_queue.cc:260] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:17.737727 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61ab80ce377a44ac95a9c574e9a615f2" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:54:17.737958 17737 raft_consensus.cc:3037] T 61ab80ce377a44ac95a9c574e9a615f2 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:17.737993 20232 raft_consensus.cc:3037] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:17.740696 20232 raft_consensus.cc:2454] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 2.
I20240502 01:54:17.740746 17737 raft_consensus.cc:2454] T 61ab80ce377a44ac95a9c574e9a615f2 P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 2.
I20240502 01:54:17.743237 20302 leader_election.cc:302] T 61ab80ce377a44ac95a9c574e9a615f2 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9, bb31bceba76d466db24b9341c04900c3, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:54:17.743551 23611 raft_consensus.cc:2781] T 61ab80ce377a44ac95a9c574e9a615f2 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:17.743727 23611 raft_consensus.cc:686] T 61ab80ce377a44ac95a9c574e9a615f2 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 LEADER]: Becoming Leader. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Running, Role: LEADER
I20240502 01:54:17.744019 23611 consensus_queue.cc:237] T 61ab80ce377a44ac95a9c574e9a615f2 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:17.750618 17327 catalog_manager.cc:5484] T 61ab80ce377a44ac95a9c574e9a615f2 P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: term changed from 1 to 2, leader changed from fdc2ac372492454c85387589a91f63af (127.11.42.69) to 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70). New cstate: current_term: 2 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:17.782807 24212 consensus_queue.cc:1035] T 0615786e80314766892988f3cff6a74d P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:17.813735 25004 consensus_queue.cc:1035] T 0615786e80314766892988f3cff6a74d P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:17.833694 25194 raft_consensus.cc:979] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685: : Instructing follower 6a36c952c51542c0bf7ea3ec2857a9f9 to start an election
I20240502 01:54:17.833865 25194 raft_consensus.cc:1067] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Signalling peer 6a36c952c51542c0bf7ea3ec2857a9f9 to start an election
I20240502 01:54:17.834789 20361 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "71cbc012efe14fdb937302796a9f84e5"
dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
from {username='slave'} at 127.11.42.67:35525
I20240502 01:54:17.834992 20361 raft_consensus.cc:482] T 71cbc012efe14fdb937302796a9f84e5 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:17.835111 20361 raft_consensus.cc:3037] T 71cbc012efe14fdb937302796a9f84e5 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:17.837457 20232 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "61ab80ce377a44ac95a9c574e9a615f2"
dest_uuid: "fdc2ac372492454c85387589a91f63af"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43096
I20240502 01:54:17.837663 20232 raft_consensus.cc:595] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af [term 2 FOLLOWER]: Received request to transfer leadership
I20240502 01:54:17.837808 20232 raft_consensus.cc:603] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240502 01:54:17.840101 20361 raft_consensus.cc:504] T 71cbc012efe14fdb937302796a9f84e5 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } }
I20240502 01:54:17.840835 20361 leader_election.cc:288] T 71cbc012efe14fdb937302796a9f84e5 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 election: Requested vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:54:17.841259 17475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71cbc012efe14fdb937302796a9f84e5" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:54:17.841352 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71cbc012efe14fdb937302796a9f84e5" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "dfa1152987644bd781b46c7e42a7a685"
I20240502 01:54:17.841452 17475 raft_consensus.cc:3037] T 71cbc012efe14fdb937302796a9f84e5 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:17.841572 17737 raft_consensus.cc:3032] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:17.841662 17737 raft_consensus.cc:729] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Running, Role: LEADER
I20240502 01:54:17.841943 17737 consensus_queue.cc:260] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } }
I20240502 01:54:17.842281 17737 raft_consensus.cc:3037] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:17.844043 17475 raft_consensus.cc:2454] T 71cbc012efe14fdb937302796a9f84e5 P 4da690c0fa13407a85c0689ce8de9ade [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 2.
I20240502 01:54:17.844177 17737 raft_consensus.cc:2454] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 2.
I20240502 01:54:17.844409 20496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71cbc012efe14fdb937302796a9f84e5" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
I20240502 01:54:17.844614 20496 raft_consensus.cc:3037] T 71cbc012efe14fdb937302796a9f84e5 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:17.845453 20302 leader_election.cc:302] T 71cbc012efe14fdb937302796a9f84e5 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, 6a36c952c51542c0bf7ea3ec2857a9f9, dfa1152987644bd781b46c7e42a7a685; no voters:
I20240502 01:54:17.845736 23611 raft_consensus.cc:2781] T 71cbc012efe14fdb937302796a9f84e5 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:17.845891 23611 raft_consensus.cc:686] T 71cbc012efe14fdb937302796a9f84e5 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 LEADER]: Becoming Leader. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Running, Role: LEADER
I20240502 01:54:17.846174 23611 consensus_queue.cc:237] T 71cbc012efe14fdb937302796a9f84e5 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } }
I20240502 01:54:17.847677 20496 raft_consensus.cc:2454] T 71cbc012efe14fdb937302796a9f84e5 P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 2.
I20240502 01:54:17.849892 17327 catalog_manager.cc:5484] T 71cbc012efe14fdb937302796a9f84e5 P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: term changed from 1 to 2, leader changed from dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) to 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70). New cstate: current_term: 2 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:17.971923 24957 consensus_queue.cc:1035] T 1423ea912bfc4472acb2b6798b3a99e9 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:17.977170 24957 consensus_queue.cc:1035] T 1423ea912bfc4472acb2b6798b3a99e9 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:17.989786 24814 raft_consensus.cc:1050] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af: attempting to promote NON_VOTER dfa1152987644bd781b46c7e42a7a685 to VOTER
I20240502 01:54:17.990767 24814 consensus_queue.cc:237] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:17.992832 17737 raft_consensus.cc:1261] T 88e68acb7de346f0ad165cadabd077be P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:17.992899 20496 raft_consensus.cc:1261] T 88e68acb7de346f0ad165cadabd077be P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:17.993579 25047 consensus_queue.cc:1035] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:17.994108 24811 consensus_queue.cc:1035] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:17.996510 20361 raft_consensus.cc:1261] T 88e68acb7de346f0ad165cadabd077be P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:17.996986 24814 consensus_queue.cc:1035] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:17.997673 25047 raft_consensus.cc:2932] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:18.000860 20496 raft_consensus.cc:2932] T 88e68acb7de346f0ad165cadabd077be P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:18.000928 20361 raft_consensus.cc:2932] T 88e68acb7de346f0ad165cadabd077be P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:18.004253 17737 raft_consensus.cc:2932] T 88e68acb7de346f0ad165cadabd077be P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:18.008441 17327 catalog_manager.cc:5484] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af reported cstate change: config changed from index 2 to 3, dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fdc2ac372492454c85387589a91f63af" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:18.014786 20232 consensus_queue.cc:237] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:18.016638 20496 raft_consensus.cc:1261] T 88e68acb7de346f0ad165cadabd077be P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:18.017342 17736 raft_consensus.cc:1261] T 88e68acb7de346f0ad165cadabd077be P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:18.018018 25298 consensus_queue.cc:1035] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:18.019237 24814 consensus_queue.cc:1035] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:18.022130 25047 raft_consensus.cc:2932] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:18.023257 20496 raft_consensus.cc:2932] T 88e68acb7de346f0ad165cadabd077be P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:18.024922 17322 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 88e68acb7de346f0ad165cadabd077be with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:18.026798 17736 raft_consensus.cc:2932] T 88e68acb7de346f0ad165cadabd077be P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:18.027537 17334 catalog_manager.cc:5484] T 88e68acb7de346f0ad165cadabd077be P fdc2ac372492454c85387589a91f63af reported cstate change: config changed from index 3 to 4, VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) evicted. New cstate: current_term: 1 leader_uuid: "fdc2ac372492454c85387589a91f63af" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:18.035879 20345 tablet_service.cc:1514] Processing DeleteTablet for tablet 88e68acb7de346f0ad165cadabd077be with delete_type TABLET_DATA_TOMBSTONED (TS 6a36c952c51542c0bf7ea3ec2857a9f9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40196
I20240502 01:54:18.037401 25299 tablet_replica.cc:330] T 88e68acb7de346f0ad165cadabd077be P 6a36c952c51542c0bf7ea3ec2857a9f9: stopping tablet replica
I20240502 01:54:18.037613 25299 raft_consensus.cc:2229] T 88e68acb7de346f0ad165cadabd077be P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:18.037879 25299 raft_consensus.cc:2258] T 88e68acb7de346f0ad165cadabd077be P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:18.038918 25299 ts_tablet_manager.cc:1875] T 88e68acb7de346f0ad165cadabd077be P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:18.045040 25299 ts_tablet_manager.cc:1888] T 88e68acb7de346f0ad165cadabd077be P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:54:18.045158 25299 log.cc:1196] T 88e68acb7de346f0ad165cadabd077be P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal/wals/88e68acb7de346f0ad165cadabd077be
I20240502 01:54:18.045943 17321 catalog_manager.cc:4830] TS 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): tablet 88e68acb7de346f0ad165cadabd077be (table rebalancer_extra_table_28 [id=736004bf70524edd905a6bb90bb25cc0]) successfully deleted
I20240502 01:54:18.084327 20625 raft_consensus.cc:1261] T 8eed067f639542d487de971761e91931 P f19cc6cb1e76460aa6711e49a5775091 [term 2 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:54:18.085215 25298 consensus_queue.cc:1035] T 8eed067f639542d487de971761e91931 P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:18.093654 20361 raft_consensus.cc:1261] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:54:18.094470 25298 consensus_queue.cc:1035] T 8eed067f639542d487de971761e91931 P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:18.103907 17736 raft_consensus.cc:1261] T 8eed067f639542d487de971761e91931 P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:54:18.106783 24811 consensus_queue.cc:1035] T 8eed067f639542d487de971761e91931 P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:18.131018 20232 raft_consensus.cc:1261] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af [term 2 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:54:18.131785 23611 consensus_queue.cc:1035] T 61ab80ce377a44ac95a9c574e9a615f2 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:18.144330 25324 ts_tablet_manager.cc:902] T 03bd47783e884a70bfea1dfab4f2e563 P 6a36c952c51542c0bf7ea3ec2857a9f9: Initiating tablet copy from peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:54:18.155530 25324 tablet_copy_client.cc:308] T 03bd47783e884a70bfea1dfab4f2e563 P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.68:44083
I20240502 01:54:18.156185 17887 tablet_copy_service.cc:140] P bb31bceba76d466db24b9341c04900c3: Received BeginTabletCopySession request for tablet 03bd47783e884a70bfea1dfab4f2e563 from peer 6a36c952c51542c0bf7ea3ec2857a9f9 ({username='slave'} at 127.11.42.70:41203)
I20240502 01:54:18.156344 17887 tablet_copy_service.cc:161] P bb31bceba76d466db24b9341c04900c3: Beginning new tablet copy session on tablet 03bd47783e884a70bfea1dfab4f2e563 from peer 6a36c952c51542c0bf7ea3ec2857a9f9 at {username='slave'} at 127.11.42.70:41203: session id = 6a36c952c51542c0bf7ea3ec2857a9f9-03bd47783e884a70bfea1dfab4f2e563
I20240502 01:54:18.159209 17887 tablet_copy_source_session.cc:205] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:18.161306 25324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03bd47783e884a70bfea1dfab4f2e563. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:18.166746 25324 tablet_copy_client.cc:783] T 03bd47783e884a70bfea1dfab4f2e563 P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:18.166990 25324 tablet_copy_client.cc:647] T 03bd47783e884a70bfea1dfab4f2e563 P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:18.169109 25324 tablet_copy_client.cc:523] T 03bd47783e884a70bfea1dfab4f2e563 P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:18.172369 25324 tablet_bootstrap.cc:492] T 03bd47783e884a70bfea1dfab4f2e563 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:54:18.220755 25324 tablet_bootstrap.cc:492] T 03bd47783e884a70bfea1dfab4f2e563 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:18.221256 25324 tablet_bootstrap.cc:492] T 03bd47783e884a70bfea1dfab4f2e563 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap complete.
I20240502 01:54:18.221547 25324 ts_tablet_manager.cc:1367] T 03bd47783e884a70bfea1dfab4f2e563 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.049s user 0.045s sys 0.001s
I20240502 01:54:18.222707 25324 raft_consensus.cc:348] T 03bd47783e884a70bfea1dfab4f2e563 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } }
I20240502 01:54:18.222990 25324 raft_consensus.cc:729] T 03bd47783e884a70bfea1dfab4f2e563 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: LEARNER
I20240502 01:54:18.223196 25324 consensus_queue.cc:260] T 03bd47783e884a70bfea1dfab4f2e563 P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } }
I20240502 01:54:18.225215 25324 ts_tablet_manager.cc:1398] T 03bd47783e884a70bfea1dfab4f2e563 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 01:54:18.225905 17887 tablet_copy_service.cc:342] P bb31bceba76d466db24b9341c04900c3: Request end of tablet copy session 6a36c952c51542c0bf7ea3ec2857a9f9-03bd47783e884a70bfea1dfab4f2e563 received from {username='slave'} at 127.11.42.70:41203
I20240502 01:54:18.226006 17887 tablet_copy_service.cc:433] P bb31bceba76d466db24b9341c04900c3: ending tablet copy session 6a36c952c51542c0bf7ea3ec2857a9f9-03bd47783e884a70bfea1dfab4f2e563 on tablet 03bd47783e884a70bfea1dfab4f2e563 with peer 6a36c952c51542c0bf7ea3ec2857a9f9
I20240502 01:54:18.274565 17867 raft_consensus.cc:1261] T 61ab80ce377a44ac95a9c574e9a615f2 P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:54:18.275050 24817 consensus_queue.cc:1035] T 61ab80ce377a44ac95a9c574e9a615f2 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:18.284767 17736 raft_consensus.cc:1261] T 61ab80ce377a44ac95a9c574e9a615f2 P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:54:18.286116 25357 consensus_queue.cc:1035] T 61ab80ce377a44ac95a9c574e9a615f2 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:18.314875 17736 raft_consensus.cc:1261] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:54:18.315325 23611 consensus_queue.cc:1035] T 71cbc012efe14fdb937302796a9f84e5 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:18.321641 20496 raft_consensus.cc:1261] T 71cbc012efe14fdb937302796a9f84e5 P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:54:18.322150 25358 consensus_queue.cc:1035] T 71cbc012efe14fdb937302796a9f84e5 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:18.327199 17475 raft_consensus.cc:1261] T 71cbc012efe14fdb937302796a9f84e5 P 4da690c0fa13407a85c0689ce8de9ade [term 2 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:54:18.327741 24817 consensus_queue.cc:1035] T 71cbc012efe14fdb937302796a9f84e5 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:18.522756 20364 raft_consensus.cc:1203] T 03bd47783e884a70bfea1dfab4f2e563 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 01:54:18.691563 20232 consensus_queue.cc:237] T 8eed067f639542d487de971761e91931 P fdc2ac372492454c85387589a91f63af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:18.693322 17736 raft_consensus.cc:1261] T 8eed067f639542d487de971761e91931 P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:54:18.693575 20625 raft_consensus.cc:1261] T 8eed067f639542d487de971761e91931 P f19cc6cb1e76460aa6711e49a5775091 [term 2 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:54:18.693832 25297 consensus_queue.cc:1035] T 8eed067f639542d487de971761e91931 P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:18.694134 25384 consensus_queue.cc:1035] T 8eed067f639542d487de971761e91931 P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:18.696141 25298 raft_consensus.cc:2932] T 8eed067f639542d487de971761e91931 P fdc2ac372492454c85387589a91f63af [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:18.696880 20625 raft_consensus.cc:2932] T 8eed067f639542d487de971761e91931 P f19cc6cb1e76460aa6711e49a5775091 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:18.699743 17736 raft_consensus.cc:2932] T 8eed067f639542d487de971761e91931 P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:18.700284 17322 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8eed067f639542d487de971761e91931 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:18.702865 17326 catalog_manager.cc:5484] T 8eed067f639542d487de971761e91931 P fdc2ac372492454c85387589a91f63af reported cstate change: config changed from index 3 to 5, VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) evicted. New cstate: current_term: 2 leader_uuid: "fdc2ac372492454c85387589a91f63af" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:18.741276 20345 tablet_service.cc:1514] Processing DeleteTablet for tablet 8eed067f639542d487de971761e91931 with delete_type TABLET_DATA_TOMBSTONED (TS 6a36c952c51542c0bf7ea3ec2857a9f9 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:40196
I20240502 01:54:18.757975 20364 consensus_queue.cc:237] T 61ab80ce377a44ac95a9c574e9a615f2 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:18.759909 17736 raft_consensus.cc:1261] T 61ab80ce377a44ac95a9c574e9a615f2 P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:54:18.761140 17864 raft_consensus.cc:1261] T 61ab80ce377a44ac95a9c574e9a615f2 P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:54:18.762835 23611 consensus_queue.cc:1035] T 61ab80ce377a44ac95a9c574e9a615f2 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:18.767500 24817 raft_consensus.cc:2932] T 61ab80ce377a44ac95a9c574e9a615f2 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:18.772328 25357 consensus_queue.cc:1035] T 61ab80ce377a44ac95a9c574e9a615f2 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:18.774253 17737 raft_consensus.cc:2932] T 61ab80ce377a44ac95a9c574e9a615f2 P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:18.775807 17867 raft_consensus.cc:2932] T 61ab80ce377a44ac95a9c574e9a615f2 P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } }
I20240502 01:54:18.780148 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 61ab80ce377a44ac95a9c574e9a615f2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:18.783006 17333 catalog_manager.cc:5484] T 61ab80ce377a44ac95a9c574e9a615f2 P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: config changed from index 3 to 5, VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) evicted. New cstate: current_term: 2 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:18.788082 20211 tablet_service.cc:1514] Processing DeleteTablet for tablet 61ab80ce377a44ac95a9c574e9a615f2 with delete_type TABLET_DATA_TOMBSTONED (TS fdc2ac372492454c85387589a91f63af not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44624
I20240502 01:54:18.799516 25394 tablet_replica.cc:330] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9: stopping tablet replica
I20240502 01:54:18.799728 25394 raft_consensus.cc:2229] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:18.799921 25394 raft_consensus.cc:2258] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:18.801569 25394 ts_tablet_manager.cc:1875] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:18.808477 25394 ts_tablet_manager.cc:1888] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 01:54:18.808593 25394 log.cc:1196] T 8eed067f639542d487de971761e91931 P 6a36c952c51542c0bf7ea3ec2857a9f9: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal/wals/8eed067f639542d487de971761e91931
I20240502 01:54:18.813733 25396 tablet_replica.cc:330] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af: stopping tablet replica
I20240502 01:54:18.813930 25396 raft_consensus.cc:2229] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:18.814121 25396 raft_consensus.cc:2258] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:18.815788 25396 ts_tablet_manager.cc:1875] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:18.821970 25396 ts_tablet_manager.cc:1888] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 01:54:18.822178 25396 log.cc:1196] T 61ab80ce377a44ac95a9c574e9a615f2 P fdc2ac372492454c85387589a91f63af: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal/wals/61ab80ce377a44ac95a9c574e9a615f2
I20240502 01:54:18.822190 17321 catalog_manager.cc:4830] TS 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): tablet 8eed067f639542d487de971761e91931 (table rebalancer_extra_table_12 [id=ecb981650f8c425798b67a5194ad793c]) successfully deleted
I20240502 01:54:18.823474 17322 catalog_manager.cc:4830] TS fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): tablet 61ab80ce377a44ac95a9c574e9a615f2 (table rebalancer_extra_table_24 [id=6077652d45454150b5c7d7072780d895]) successfully deleted
I20240502 01:54:18.862002 20364 consensus_queue.cc:237] T 71cbc012efe14fdb937302796a9f84e5 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } }
I20240502 01:54:18.864303 20496 raft_consensus.cc:1261] T 71cbc012efe14fdb937302796a9f84e5 P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:54:18.864921 17476 raft_consensus.cc:1261] T 71cbc012efe14fdb937302796a9f84e5 P 4da690c0fa13407a85c0689ce8de9ade [term 2 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:54:18.865140 23611 consensus_queue.cc:1035] T 71cbc012efe14fdb937302796a9f84e5 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:54:18.865669 23611 consensus_queue.cc:1035] T 71cbc012efe14fdb937302796a9f84e5 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:54:18.868012 23611 raft_consensus.cc:2932] T 71cbc012efe14fdb937302796a9f84e5 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:18.868711 17476 raft_consensus.cc:2932] T 71cbc012efe14fdb937302796a9f84e5 P 4da690c0fa13407a85c0689ce8de9ade [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:18.873135 20496 raft_consensus.cc:2932] T 71cbc012efe14fdb937302796a9f84e5 P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:18.875010 17326 catalog_manager.cc:5484] T 71cbc012efe14fdb937302796a9f84e5 P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: config changed from index 8 to 10, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New cstate: current_term: 2 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:18.876926 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 71cbc012efe14fdb937302796a9f84e5 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:18.881388 17715 tablet_service.cc:1514] Processing DeleteTablet for tablet 71cbc012efe14fdb937302796a9f84e5 with delete_type TABLET_DATA_TOMBSTONED (TS dfa1152987644bd781b46c7e42a7a685 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:52198
I20240502 01:54:18.882697 25419 tablet_replica.cc:330] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685: stopping tablet replica
I20240502 01:54:18.882968 25419 raft_consensus.cc:2229] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:18.883217 25419 raft_consensus.cc:2258] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:18.884634 25419 ts_tablet_manager.cc:1875] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:18.892027 25419 ts_tablet_manager.cc:1888] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 01:54:18.892199 25419 log.cc:1196] T 71cbc012efe14fdb937302796a9f84e5 P dfa1152987644bd781b46c7e42a7a685: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/wals/71cbc012efe14fdb937302796a9f84e5
I20240502 01:54:18.895393 17321 catalog_manager.cc:4830] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): tablet 71cbc012efe14fdb937302796a9f84e5 (table rebalance_test_table_19 [id=b69eb497d10340fea8df98ad03a1e465]) successfully deleted
I20240502 01:54:19.100351 25173 raft_consensus.cc:1050] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3: attempting to promote NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 to VOTER
I20240502 01:54:19.101303 25173 consensus_queue.cc:237] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } }
I20240502 01:54:19.104300 20496 raft_consensus.cc:1261] T 03bd47783e884a70bfea1dfab4f2e563 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:19.104622 17737 raft_consensus.cc:1261] T 03bd47783e884a70bfea1dfab4f2e563 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:19.104856 20361 raft_consensus.cc:1261] T 03bd47783e884a70bfea1dfab4f2e563 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:19.105499 25383 consensus_queue.cc:1035] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:19.105794 25173 consensus_queue.cc:1035] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:19.113687 25429 consensus_queue.cc:1035] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:19.115295 25383 raft_consensus.cc:2932] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:54:19.115669 17737 raft_consensus.cc:2932] T 03bd47783e884a70bfea1dfab4f2e563 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:54:19.122673 20496 raft_consensus.cc:2932] T 03bd47783e884a70bfea1dfab4f2e563 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:54:19.127234 17333 catalog_manager.cc:5484] T 03bd47783e884a70bfea1dfab4f2e563 P dfa1152987644bd781b46c7e42a7a685 reported cstate change: config changed from index 7 to 8, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:54:19.149091 20361 raft_consensus.cc:2932] T 03bd47783e884a70bfea1dfab4f2e563 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:54:19.149462 17867 consensus_queue.cc:237] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } }
I20240502 01:54:19.152236 20496 raft_consensus.cc:1261] T 03bd47783e884a70bfea1dfab4f2e563 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:19.152797 25173 consensus_queue.cc:1035] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:19.155239 25436 raft_consensus.cc:2932] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:54:19.156106 20496 raft_consensus.cc:2932] T 03bd47783e884a70bfea1dfab4f2e563 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:54:19.159526 17322 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 03bd47783e884a70bfea1dfab4f2e563 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:19.163514 17333 catalog_manager.cc:5484] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 reported cstate change: config changed from index 8 to 9, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New cstate: current_term: 1 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:19.166914 20364 raft_consensus.cc:1261] T 03bd47783e884a70bfea1dfab4f2e563 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:19.167647 25173 consensus_queue.cc:1035] T 03bd47783e884a70bfea1dfab4f2e563 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:19.168978 20364 raft_consensus.cc:2932] T 03bd47783e884a70bfea1dfab4f2e563 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:54:19.179147 17715 tablet_service.cc:1514] Processing DeleteTablet for tablet 03bd47783e884a70bfea1dfab4f2e563 with delete_type TABLET_DATA_TOMBSTONED (TS dfa1152987644bd781b46c7e42a7a685 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:52198
I20240502 01:54:19.179440 25419 tablet_replica.cc:330] T 03bd47783e884a70bfea1dfab4f2e563 P dfa1152987644bd781b46c7e42a7a685: stopping tablet replica
I20240502 01:54:19.179630 25419 raft_consensus.cc:2229] T 03bd47783e884a70bfea1dfab4f2e563 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:19.179828 25419 raft_consensus.cc:2258] T 03bd47783e884a70bfea1dfab4f2e563 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:19.181196 25419 ts_tablet_manager.cc:1875] T 03bd47783e884a70bfea1dfab4f2e563 P dfa1152987644bd781b46c7e42a7a685: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:19.188638 25419 ts_tablet_manager.cc:1888] T 03bd47783e884a70bfea1dfab4f2e563 P dfa1152987644bd781b46c7e42a7a685: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:54:19.188844 25419 log.cc:1196] T 03bd47783e884a70bfea1dfab4f2e563 P dfa1152987644bd781b46c7e42a7a685: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/wals/03bd47783e884a70bfea1dfab4f2e563
I20240502 01:54:19.190933 17321 catalog_manager.cc:4830] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): tablet 03bd47783e884a70bfea1dfab4f2e563 (table rebalance_test_table_3 [id=b996bd60793c4d3c83bf34f4989da909]) successfully deleted
I20240502 01:54:19.496178 20477 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:54:19.506907 17585 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:19.534500 17846 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 01:54:19.547711 17454 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 01:54:19.548079 17715 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 01:54:19.599615 20610 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 01:54:19.607457 20345 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 01:54:19.698606 20211 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:19.888128 20627 consensus_queue.cc:237] T ee28f4aa31504dbfac977c4178aefdb1 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } }
I20240502 01:54:19.890288 17737 raft_consensus.cc:1261] T ee28f4aa31504dbfac977c4178aefdb1 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:19.890553 20496 raft_consensus.cc:1261] T ee28f4aa31504dbfac977c4178aefdb1 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:54:19.890744 20565 consensus_peers.cc:485] T ee28f4aa31504dbfac977c4178aefdb1 P f19cc6cb1e76460aa6711e49a5775091 -> Peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): Couldn't send request to peer bb31bceba76d466db24b9341c04900c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ee28f4aa31504dbfac977c4178aefdb1. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:19.891114 25501 consensus_queue.cc:1035] T ee28f4aa31504dbfac977c4178aefdb1 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:19.891487 25188 consensus_queue.cc:1035] T ee28f4aa31504dbfac977c4178aefdb1 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:19.893973 25501 raft_consensus.cc:2932] T ee28f4aa31504dbfac977c4178aefdb1 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } } }
I20240502 01:54:19.894582 20496 raft_consensus.cc:2932] T ee28f4aa31504dbfac977c4178aefdb1 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } } }
I20240502 01:54:19.894731 17737 raft_consensus.cc:2932] T ee28f4aa31504dbfac977c4178aefdb1 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } } }
I20240502 01:54:19.903164 17334 catalog_manager.cc:5484] T ee28f4aa31504dbfac977c4178aefdb1 P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: config changed from index -1 to 2, NON_VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) added. New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:19.917253 17605 consensus_queue.cc:237] T fa4d38dc9b674a9b90d4793dad2687ec P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
W20240502 01:54:19.919231 17542 consensus_peers.cc:485] T fa4d38dc9b674a9b90d4793dad2687ec P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fa4d38dc9b674a9b90d4793dad2687ec. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:19.919345 17475 raft_consensus.cc:1261] T fa4d38dc9b674a9b90d4793dad2687ec P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:19.919433 20361 raft_consensus.cc:1261] T fa4d38dc9b674a9b90d4793dad2687ec P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:19.919823 25513 consensus_queue.cc:1035] T fa4d38dc9b674a9b90d4793dad2687ec P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:19.920115 24918 consensus_queue.cc:1035] T fa4d38dc9b674a9b90d4793dad2687ec P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:19.924463 24598 raft_consensus.cc:2932] T fa4d38dc9b674a9b90d4793dad2687ec P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:19.928484 17475 raft_consensus.cc:2932] T fa4d38dc9b674a9b90d4793dad2687ec P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:19.931916 20364 raft_consensus.cc:2932] T fa4d38dc9b674a9b90d4793dad2687ec P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:19.934412 17330 catalog_manager.cc:5484] T fa4d38dc9b674a9b90d4793dad2687ec P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:19.949748 20231 consensus_queue.cc:237] T f6cac9478db94a21a9799b80d24c8dc9 P fdc2ac372492454c85387589a91f63af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } }
I20240502 01:54:19.952438 20627 raft_consensus.cc:1261] T f6cac9478db94a21a9799b80d24c8dc9 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:19.953572 17604 raft_consensus.cc:1261] T f6cac9478db94a21a9799b80d24c8dc9 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:19.954157 25298 consensus_queue.cc:1035] T f6cac9478db94a21a9799b80d24c8dc9 P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:54:19.954633 20167 consensus_peers.cc:485] T f6cac9478db94a21a9799b80d24c8dc9 P fdc2ac372492454c85387589a91f63af -> Peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Couldn't send request to peer 4da690c0fa13407a85c0689ce8de9ade. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f6cac9478db94a21a9799b80d24c8dc9. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:19.955025 25384 consensus_queue.cc:1035] T f6cac9478db94a21a9799b80d24c8dc9 P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:19.965307 25384 raft_consensus.cc:2932] T f6cac9478db94a21a9799b80d24c8dc9 P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } } }
I20240502 01:54:19.966024 17604 raft_consensus.cc:2932] T f6cac9478db94a21a9799b80d24c8dc9 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } } }
I20240502 01:54:19.967612 20627 raft_consensus.cc:2932] T f6cac9478db94a21a9799b80d24c8dc9 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } } }
I20240502 01:54:19.972824 17330 catalog_manager.cc:5484] T f6cac9478db94a21a9799b80d24c8dc9 P fdc2ac372492454c85387589a91f63af reported cstate change: config changed from index -1 to 2, NON_VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) added. New cstate: current_term: 1 leader_uuid: "fdc2ac372492454c85387589a91f63af" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:20.001613 17475 consensus_queue.cc:237] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } }
W20240502 01:54:20.004539 17409 consensus_peers.cc:485] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 4da690c0fa13407a85c0689ce8de9ade -> Peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): Couldn't send request to peer 3fceccefba274b1dbbd8216af8b0bc29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 29fb2b31cd9c4c8fb5923bb849a097d9. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:20.004521 20627 raft_consensus.cc:1261] T 29fb2b31cd9c4c8fb5923bb849a097d9 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:20.004838 20496 raft_consensus.cc:1261] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:20.005396 25532 consensus_queue.cc:1035] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:20.006302 25004 consensus_queue.cc:1035] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:20.010385 25532 raft_consensus.cc:2932] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } } }
I20240502 01:54:20.011154 20496 raft_consensus.cc:2932] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } } }
I20240502 01:54:20.014519 20627 raft_consensus.cc:2932] T 29fb2b31cd9c4c8fb5923bb849a097d9 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } } }
I20240502 01:54:20.018940 17326 catalog_manager.cc:5484] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: config changed from index -1 to 2, NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) added. New cstate: current_term: 1 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:20.039449 20231 consensus_queue.cc:237] T bbf05e3b3ad44354be7600c3c07756fd P fdc2ac372492454c85387589a91f63af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
I20240502 01:54:20.041590 17604 raft_consensus.cc:1261] T bbf05e3b3ad44354be7600c3c07756fd P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:54:20.041774 20168 consensus_peers.cc:485] T bbf05e3b3ad44354be7600c3c07756fd P fdc2ac372492454c85387589a91f63af -> Peer 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): Couldn't send request to peer 2f306549407b4bb281ccf2994035e4f1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bbf05e3b3ad44354be7600c3c07756fd. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:20.042075 20364 raft_consensus.cc:1261] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:20.042093 25523 consensus_queue.cc:1035] T bbf05e3b3ad44354be7600c3c07756fd P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:20.042723 25523 consensus_queue.cc:1035] T bbf05e3b3ad44354be7600c3c07756fd P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:20.045080 25523 raft_consensus.cc:2932] T bbf05e3b3ad44354be7600c3c07756fd P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:54:20.045439 17604 raft_consensus.cc:2932] T bbf05e3b3ad44354be7600c3c07756fd P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:54:20.046487 20364 raft_consensus.cc:2932] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:54:20.052083 17327 catalog_manager.cc:5484] T bbf05e3b3ad44354be7600c3c07756fd P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: config changed from index -1 to 2, NON_VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) added. New cstate: current_term: 1 leader_uuid: "fdc2ac372492454c85387589a91f63af" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } } }
I20240502 01:54:20.079054 17475 consensus_queue.cc:237] T 0615786e80314766892988f3cff6a74d P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } }
I20240502 01:54:20.080989 20627 raft_consensus.cc:1261] T 0615786e80314766892988f3cff6a74d P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:20.081158 17867 raft_consensus.cc:1261] T 0615786e80314766892988f3cff6a74d P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:20.081614 25535 consensus_queue.cc:1035] T 0615786e80314766892988f3cff6a74d P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:20.081863 25532 consensus_queue.cc:1035] T 0615786e80314766892988f3cff6a74d P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:54:20.082134 17412 consensus_peers.cc:485] T 0615786e80314766892988f3cff6a74d P 4da690c0fa13407a85c0689ce8de9ade -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0615786e80314766892988f3cff6a74d. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:20.084604 25004 raft_consensus.cc:2932] T 0615786e80314766892988f3cff6a74d P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:54:20.085384 20627 raft_consensus.cc:2932] T 0615786e80314766892988f3cff6a74d P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:54:20.086462 17867 raft_consensus.cc:2932] T 0615786e80314766892988f3cff6a74d P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:54:20.091308 17330 catalog_manager.cc:5484] T 0615786e80314766892988f3cff6a74d P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: config changed from index -1 to 2, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New cstate: current_term: 1 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:54:20.105918 17604 consensus_queue.cc:237] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } }
I20240502 01:54:20.108067 17737 raft_consensus.cc:1261] T a86548104dca43bd9e1b3ec4d23ec0e4 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:20.108161 20496 raft_consensus.cc:1261] T a86548104dca43bd9e1b3ec4d23ec0e4 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:20.108654 25516 consensus_queue.cc:1035] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:54:20.108928 17542 consensus_peers.cc:485] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Couldn't send request to peer 4da690c0fa13407a85c0689ce8de9ade. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a86548104dca43bd9e1b3ec4d23ec0e4. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:20.109241 25513 consensus_queue.cc:1035] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:20.111812 25513 raft_consensus.cc:2932] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } } }
I20240502 01:54:20.113657 20496 raft_consensus.cc:2932] T a86548104dca43bd9e1b3ec4d23ec0e4 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } } }
I20240502 01:54:20.120103 17334 catalog_manager.cc:5484] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: config changed from index -1 to 2, NON_VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) added. New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:20.128388 17737 raft_consensus.cc:2932] T a86548104dca43bd9e1b3ec4d23ec0e4 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } } }
I20240502 01:54:20.147629 17476 consensus_queue.cc:237] T 495700f8ed5e41228f2e597465dad126 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } }
I20240502 01:54:20.150096 20364 raft_consensus.cc:1261] T 495700f8ed5e41228f2e597465dad126 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:20.150120 20627 raft_consensus.cc:1261] T 495700f8ed5e41228f2e597465dad126 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:54:20.150277 17412 consensus_peers.cc:485] T 495700f8ed5e41228f2e597465dad126 P 4da690c0fa13407a85c0689ce8de9ade -> Peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): Couldn't send request to peer bb31bceba76d466db24b9341c04900c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 495700f8ed5e41228f2e597465dad126. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:20.150599 25004 consensus_queue.cc:1035] T 495700f8ed5e41228f2e597465dad126 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:20.150857 25535 consensus_queue.cc:1035] T 495700f8ed5e41228f2e597465dad126 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:20.155297 25532 raft_consensus.cc:2932] T 495700f8ed5e41228f2e597465dad126 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } } }
I20240502 01:54:20.155966 20364 raft_consensus.cc:2932] T 495700f8ed5e41228f2e597465dad126 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } } }
I20240502 01:54:20.160120 20625 raft_consensus.cc:2932] T 495700f8ed5e41228f2e597465dad126 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } } }
I20240502 01:54:20.166563 17330 catalog_manager.cc:5484] T 495700f8ed5e41228f2e597465dad126 P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: config changed from index -1 to 2, NON_VOTER bb31bceba76d466db24b9341c04900c3 (127.11.42.68) added. New cstate: current_term: 1 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:20.192885 20496 consensus_queue.cc:237] T 6ded7434afab44999620dfa3213c7a82 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } }
I20240502 01:54:20.196177 17604 raft_consensus.cc:1261] T 6ded7434afab44999620dfa3213c7a82 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:20.196204 20625 raft_consensus.cc:1261] T 6ded7434afab44999620dfa3213c7a82 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:20.196825 25361 consensus_queue.cc:1035] T 6ded7434afab44999620dfa3213c7a82 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:20.197649 24959 consensus_queue.cc:1035] T 6ded7434afab44999620dfa3213c7a82 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:54:20.198730 20433 consensus_peers.cc:485] T 6ded7434afab44999620dfa3213c7a82 P 2f306549407b4bb281ccf2994035e4f1 -> Peer 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): Couldn't send request to peer 6a36c952c51542c0bf7ea3ec2857a9f9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6ded7434afab44999620dfa3213c7a82. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:20.201237 25361 raft_consensus.cc:2932] T 6ded7434afab44999620dfa3213c7a82 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } } }
I20240502 01:54:20.202555 20625 raft_consensus.cc:2932] T 6ded7434afab44999620dfa3213c7a82 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } } }
I20240502 01:54:20.206677 17604 raft_consensus.cc:2932] T 6ded7434afab44999620dfa3213c7a82 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } } }
I20240502 01:54:20.210510 17334 catalog_manager.cc:5484] T 6ded7434afab44999620dfa3213c7a82 P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: config changed from index -1 to 2, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:20.229743 20625 consensus_queue.cc:237] T 38bb8dd915f743f49b26ee2572871375 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } }
I20240502 01:54:20.232216 20496 raft_consensus.cc:1261] T 38bb8dd915f743f49b26ee2572871375 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:20.232282 17737 raft_consensus.cc:1261] T 38bb8dd915f743f49b26ee2572871375 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:20.232873 25501 consensus_queue.cc:1035] T 38bb8dd915f743f49b26ee2572871375 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:54:20.233194 20565 consensus_peers.cc:485] T 38bb8dd915f743f49b26ee2572871375 P f19cc6cb1e76460aa6711e49a5775091 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 38bb8dd915f743f49b26ee2572871375. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:20.233453 25501 consensus_queue.cc:1035] T 38bb8dd915f743f49b26ee2572871375 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:20.235810 24955 raft_consensus.cc:2932] T 38bb8dd915f743f49b26ee2572871375 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:54:20.236333 17737 raft_consensus.cc:2932] T 38bb8dd915f743f49b26ee2572871375 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:54:20.240109 20496 raft_consensus.cc:2932] T 38bb8dd915f743f49b26ee2572871375 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:54:20.242086 17334 catalog_manager.cc:5484] T 38bb8dd915f743f49b26ee2572871375 P dfa1152987644bd781b46c7e42a7a685 reported cstate change: config changed from index -1 to 2, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:54:20.276260 25594 ts_tablet_manager.cc:902] T ee28f4aa31504dbfac977c4178aefdb1 P bb31bceba76d466db24b9341c04900c3: Initiating tablet copy from peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:54:20.277639 25594 tablet_copy_client.cc:308] T ee28f4aa31504dbfac977c4178aefdb1 P bb31bceba76d466db24b9341c04900c3: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.72:42411
I20240502 01:54:20.278492 20649 tablet_copy_service.cc:140] P f19cc6cb1e76460aa6711e49a5775091: Received BeginTabletCopySession request for tablet ee28f4aa31504dbfac977c4178aefdb1 from peer bb31bceba76d466db24b9341c04900c3 ({username='slave'} at 127.11.42.68:36611)
I20240502 01:54:20.278635 20649 tablet_copy_service.cc:161] P f19cc6cb1e76460aa6711e49a5775091: Beginning new tablet copy session on tablet ee28f4aa31504dbfac977c4178aefdb1 from peer bb31bceba76d466db24b9341c04900c3 at {username='slave'} at 127.11.42.68:36611: session id = bb31bceba76d466db24b9341c04900c3-ee28f4aa31504dbfac977c4178aefdb1
I20240502 01:54:20.281255 20649 tablet_copy_source_session.cc:205] T ee28f4aa31504dbfac977c4178aefdb1 P f19cc6cb1e76460aa6711e49a5775091: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:20.284547 20364 consensus_queue.cc:237] T 0399c9d1d7b2408292dc78c185605ee8 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:54:20.285637 25594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee28f4aa31504dbfac977c4178aefdb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:20.288175 17604 raft_consensus.cc:1261] T 0399c9d1d7b2408292dc78c185605ee8 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:54:20.288379 20302 consensus_peers.cc:485] T 0399c9d1d7b2408292dc78c185605ee8 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0399c9d1d7b2408292dc78c185605ee8. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:20.288754 25358 consensus_queue.cc:1035] T 0399c9d1d7b2408292dc78c185605ee8 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:20.289692 17867 raft_consensus.cc:1261] T 0399c9d1d7b2408292dc78c185605ee8 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:20.290308 24817 consensus_queue.cc:1035] T 0399c9d1d7b2408292dc78c185605ee8 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:20.291759 25594 tablet_copy_client.cc:783] T ee28f4aa31504dbfac977c4178aefdb1 P bb31bceba76d466db24b9341c04900c3: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:20.291960 25594 tablet_copy_client.cc:647] T ee28f4aa31504dbfac977c4178aefdb1 P bb31bceba76d466db24b9341c04900c3: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:20.292817 24817 raft_consensus.cc:2932] T 0399c9d1d7b2408292dc78c185605ee8 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:20.293290 17604 raft_consensus.cc:2932] T 0399c9d1d7b2408292dc78c185605ee8 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:20.299906 17326 catalog_manager.cc:5484] T 0399c9d1d7b2408292dc78c185605ee8 P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:20.305378 17867 raft_consensus.cc:2932] T 0399c9d1d7b2408292dc78c185605ee8 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:20.315132 25594 tablet_copy_client.cc:523] T ee28f4aa31504dbfac977c4178aefdb1 P bb31bceba76d466db24b9341c04900c3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:20.319028 25594 tablet_bootstrap.cc:492] T ee28f4aa31504dbfac977c4178aefdb1 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:54:20.323980 25594 tablet_bootstrap.cc:492] T ee28f4aa31504dbfac977c4178aefdb1 P bb31bceba76d466db24b9341c04900c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:20.324265 25594 tablet_bootstrap.cc:492] T ee28f4aa31504dbfac977c4178aefdb1 P bb31bceba76d466db24b9341c04900c3: Bootstrap complete.
I20240502 01:54:20.324456 25594 ts_tablet_manager.cc:1367] T ee28f4aa31504dbfac977c4178aefdb1 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20240502 01:54:20.325066 25594 raft_consensus.cc:348] T ee28f4aa31504dbfac977c4178aefdb1 P bb31bceba76d466db24b9341c04900c3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } }
I20240502 01:54:20.325316 25594 raft_consensus.cc:729] T ee28f4aa31504dbfac977c4178aefdb1 P bb31bceba76d466db24b9341c04900c3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: LEARNER
I20240502 01:54:20.325552 25594 consensus_queue.cc:260] T ee28f4aa31504dbfac977c4178aefdb1 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } }
I20240502 01:54:20.326261 25594 ts_tablet_manager.cc:1398] T ee28f4aa31504dbfac977c4178aefdb1 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 01:54:20.327021 20649 tablet_copy_service.cc:342] P f19cc6cb1e76460aa6711e49a5775091: Request end of tablet copy session bb31bceba76d466db24b9341c04900c3-ee28f4aa31504dbfac977c4178aefdb1 received from {username='slave'} at 127.11.42.68:36611
I20240502 01:54:20.327191 20649 tablet_copy_service.cc:433] P f19cc6cb1e76460aa6711e49a5775091: ending tablet copy session bb31bceba76d466db24b9341c04900c3-ee28f4aa31504dbfac977c4178aefdb1 on tablet ee28f4aa31504dbfac977c4178aefdb1 with peer bb31bceba76d466db24b9341c04900c3
I20240502 01:54:20.337448 20231 consensus_queue.cc:237] T 25ad701a0fec4e80b37df5dec598796a P fdc2ac372492454c85387589a91f63af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:54:20.339529 20496 raft_consensus.cc:1261] T 25ad701a0fec4e80b37df5dec598796a P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:20.339718 17476 raft_consensus.cc:1261] T 25ad701a0fec4e80b37df5dec598796a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:54:20.340040 20167 consensus_peers.cc:485] T 25ad701a0fec4e80b37df5dec598796a P fdc2ac372492454c85387589a91f63af -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 25ad701a0fec4e80b37df5dec598796a. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:20.340092 25523 consensus_queue.cc:1035] T 25ad701a0fec4e80b37df5dec598796a P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:20.340345 25298 consensus_queue.cc:1035] T 25ad701a0fec4e80b37df5dec598796a P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:20.343122 25523 raft_consensus.cc:2932] T 25ad701a0fec4e80b37df5dec598796a P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:54:20.343606 20496 raft_consensus.cc:2932] T 25ad701a0fec4e80b37df5dec598796a P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:54:20.344439 17476 raft_consensus.cc:2932] T 25ad701a0fec4e80b37df5dec598796a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:54:20.350057 17327 catalog_manager.cc:5484] T 25ad701a0fec4e80b37df5dec598796a P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: config changed from index -1 to 2, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New cstate: current_term: 1 leader_uuid: "fdc2ac372492454c85387589a91f63af" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:54:20.359004 25616 ts_tablet_manager.cc:902] T f6cac9478db94a21a9799b80d24c8dc9 P 4da690c0fa13407a85c0689ce8de9ade: Initiating tablet copy from peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:54:20.367816 25616 tablet_copy_client.cc:308] T f6cac9478db94a21a9799b80d24c8dc9 P 4da690c0fa13407a85c0689ce8de9ade: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.69:34339
I20240502 01:54:20.368410 20253 tablet_copy_service.cc:140] P fdc2ac372492454c85387589a91f63af: Received BeginTabletCopySession request for tablet f6cac9478db94a21a9799b80d24c8dc9 from peer 4da690c0fa13407a85c0689ce8de9ade ({username='slave'} at 127.11.42.65:54337)
I20240502 01:54:20.368582 20253 tablet_copy_service.cc:161] P fdc2ac372492454c85387589a91f63af: Beginning new tablet copy session on tablet f6cac9478db94a21a9799b80d24c8dc9 from peer 4da690c0fa13407a85c0689ce8de9ade at {username='slave'} at 127.11.42.65:54337: session id = 4da690c0fa13407a85c0689ce8de9ade-f6cac9478db94a21a9799b80d24c8dc9
I20240502 01:54:20.371191 20253 tablet_copy_source_session.cc:205] T f6cac9478db94a21a9799b80d24c8dc9 P fdc2ac372492454c85387589a91f63af: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:20.372334 25616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6cac9478db94a21a9799b80d24c8dc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:20.378489 25616 tablet_copy_client.cc:783] T f6cac9478db94a21a9799b80d24c8dc9 P 4da690c0fa13407a85c0689ce8de9ade: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:20.378695 25616 tablet_copy_client.cc:647] T f6cac9478db94a21a9799b80d24c8dc9 P 4da690c0fa13407a85c0689ce8de9ade: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:20.380122 25616 tablet_copy_client.cc:523] T f6cac9478db94a21a9799b80d24c8dc9 P 4da690c0fa13407a85c0689ce8de9ade: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:20.380157 25623 ts_tablet_manager.cc:902] T fa4d38dc9b674a9b90d4793dad2687ec P dfa1152987644bd781b46c7e42a7a685: Initiating tablet copy from peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:54:20.380836 17604 consensus_queue.cc:237] T e61094fdf9324469afa7ecdc45fd55de P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } }
I20240502 01:54:20.383198 17867 raft_consensus.cc:1261] T e61094fdf9324469afa7ecdc45fd55de P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:20.383541 25616 tablet_bootstrap.cc:492] T f6cac9478db94a21a9799b80d24c8dc9 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:54:20.384114 17737 raft_consensus.cc:1261] T e61094fdf9324469afa7ecdc45fd55de P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:20.384580 25513 consensus_queue.cc:1035] T e61094fdf9324469afa7ecdc45fd55de P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:54:20.384995 17542 consensus_peers.cc:485] T e61094fdf9324469afa7ecdc45fd55de P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): Couldn't send request to peer 6a36c952c51542c0bf7ea3ec2857a9f9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e61094fdf9324469afa7ecdc45fd55de. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:20.385639 24918 consensus_queue.cc:1035] T e61094fdf9324469afa7ecdc45fd55de P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:20.394210 24918 raft_consensus.cc:2932] T e61094fdf9324469afa7ecdc45fd55de P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } } }
I20240502 01:54:20.395421 25616 tablet_bootstrap.cc:492] T f6cac9478db94a21a9799b80d24c8dc9 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:20.395741 25616 tablet_bootstrap.cc:492] T f6cac9478db94a21a9799b80d24c8dc9 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap complete.
I20240502 01:54:20.395584 17867 raft_consensus.cc:2932] T e61094fdf9324469afa7ecdc45fd55de P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } } }
I20240502 01:54:20.396036 25616 ts_tablet_manager.cc:1367] T f6cac9478db94a21a9799b80d24c8dc9 P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.008s
I20240502 01:54:20.396703 25616 raft_consensus.cc:348] T f6cac9478db94a21a9799b80d24c8dc9 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } }
I20240502 01:54:20.396911 25616 raft_consensus.cc:729] T f6cac9478db94a21a9799b80d24c8dc9 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: LEARNER
I20240502 01:54:20.397096 25616 consensus_queue.cc:260] T f6cac9478db94a21a9799b80d24c8dc9 P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } }
I20240502 01:54:20.397907 25616 ts_tablet_manager.cc:1398] T f6cac9478db94a21a9799b80d24c8dc9 P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 01:54:20.401649 20253 tablet_copy_service.cc:342] P fdc2ac372492454c85387589a91f63af: Request end of tablet copy session 4da690c0fa13407a85c0689ce8de9ade-f6cac9478db94a21a9799b80d24c8dc9 received from {username='slave'} at 127.11.42.65:54337
I20240502 01:54:20.401804 20253 tablet_copy_service.cc:433] P fdc2ac372492454c85387589a91f63af: ending tablet copy session 4da690c0fa13407a85c0689ce8de9ade-f6cac9478db94a21a9799b80d24c8dc9 on tablet f6cac9478db94a21a9799b80d24c8dc9 with peer 4da690c0fa13407a85c0689ce8de9ade
I20240502 01:54:20.406867 17737 raft_consensus.cc:2932] T e61094fdf9324469afa7ecdc45fd55de P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } } }
I20240502 01:54:20.408172 17326 catalog_manager.cc:5484] T e61094fdf9324469afa7ecdc45fd55de P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: config changed from index -1 to 2, NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) added. New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:20.424819 25623 tablet_copy_client.cc:308] T fa4d38dc9b674a9b90d4793dad2687ec P dfa1152987644bd781b46c7e42a7a685: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.66:38531
I20240502 01:54:20.425447 17627 tablet_copy_service.cc:140] P 3fceccefba274b1dbbd8216af8b0bc29: Received BeginTabletCopySession request for tablet fa4d38dc9b674a9b90d4793dad2687ec from peer dfa1152987644bd781b46c7e42a7a685 ({username='slave'} at 127.11.42.67:57179)
I20240502 01:54:20.425593 17627 tablet_copy_service.cc:161] P 3fceccefba274b1dbbd8216af8b0bc29: Beginning new tablet copy session on tablet fa4d38dc9b674a9b90d4793dad2687ec from peer dfa1152987644bd781b46c7e42a7a685 at {username='slave'} at 127.11.42.67:57179: session id = dfa1152987644bd781b46c7e42a7a685-fa4d38dc9b674a9b90d4793dad2687ec
I20240502 01:54:20.428453 17627 tablet_copy_source_session.cc:205] T fa4d38dc9b674a9b90d4793dad2687ec P 3fceccefba274b1dbbd8216af8b0bc29: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:20.429638 25623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa4d38dc9b674a9b90d4793dad2687ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:20.435770 17604 consensus_queue.cc:237] T 3dcd890bdd2d413e95401897e9f1bcab P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:54:20.436009 25623 tablet_copy_client.cc:783] T fa4d38dc9b674a9b90d4793dad2687ec P dfa1152987644bd781b46c7e42a7a685: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:20.436219 25623 tablet_copy_client.cc:647] T fa4d38dc9b674a9b90d4793dad2687ec P dfa1152987644bd781b46c7e42a7a685: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:20.438606 20497 raft_consensus.cc:1261] T 3dcd890bdd2d413e95401897e9f1bcab P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:54:20.439515 17542 consensus_peers.cc:485] T 3dcd890bdd2d413e95401897e9f1bcab P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3dcd890bdd2d413e95401897e9f1bcab. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:20.438652 20364 raft_consensus.cc:1261] T 3dcd890bdd2d413e95401897e9f1bcab P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:20.439770 24598 consensus_queue.cc:1035] T 3dcd890bdd2d413e95401897e9f1bcab P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:20.440021 25623 tablet_copy_client.cc:523] T fa4d38dc9b674a9b90d4793dad2687ec P dfa1152987644bd781b46c7e42a7a685: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:20.440104 25516 consensus_queue.cc:1035] T 3dcd890bdd2d413e95401897e9f1bcab P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:20.442993 25623 tablet_bootstrap.cc:492] T fa4d38dc9b674a9b90d4793dad2687ec P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:54:20.442749 24598 raft_consensus.cc:2932] T 3dcd890bdd2d413e95401897e9f1bcab P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:20.443513 20496 raft_consensus.cc:2932] T 3dcd890bdd2d413e95401897e9f1bcab P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:20.450196 17334 catalog_manager.cc:5484] T 3dcd890bdd2d413e95401897e9f1bcab P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:20.464247 25623 tablet_bootstrap.cc:492] T fa4d38dc9b674a9b90d4793dad2687ec P dfa1152987644bd781b46c7e42a7a685: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:20.464557 25623 tablet_bootstrap.cc:492] T fa4d38dc9b674a9b90d4793dad2687ec P dfa1152987644bd781b46c7e42a7a685: Bootstrap complete.
I20240502 01:54:20.464727 25623 ts_tablet_manager.cc:1367] T fa4d38dc9b674a9b90d4793dad2687ec P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.000s
I20240502 01:54:20.465332 25623 raft_consensus.cc:348] T fa4d38dc9b674a9b90d4793dad2687ec P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:54:20.465552 25623 raft_consensus.cc:729] T fa4d38dc9b674a9b90d4793dad2687ec P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: LEARNER
I20240502 01:54:20.465723 25623 consensus_queue.cc:260] T fa4d38dc9b674a9b90d4793dad2687ec P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:54:20.467550 25623 ts_tablet_manager.cc:1398] T fa4d38dc9b674a9b90d4793dad2687ec P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:20.468483 17627 tablet_copy_service.cc:342] P 3fceccefba274b1dbbd8216af8b0bc29: Request end of tablet copy session dfa1152987644bd781b46c7e42a7a685-fa4d38dc9b674a9b90d4793dad2687ec received from {username='slave'} at 127.11.42.67:57179
I20240502 01:54:20.468591 17627 tablet_copy_service.cc:433] P 3fceccefba274b1dbbd8216af8b0bc29: ending tablet copy session dfa1152987644bd781b46c7e42a7a685-fa4d38dc9b674a9b90d4793dad2687ec on tablet fa4d38dc9b674a9b90d4793dad2687ec with peer dfa1152987644bd781b46c7e42a7a685
I20240502 01:54:20.468916 20361 raft_consensus.cc:2932] T 3dcd890bdd2d413e95401897e9f1bcab P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:20.482493 20496 consensus_queue.cc:237] T 7b2eae408b9843acaa61e698a3e44cff P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } }
W20240502 01:54:20.485087 20434 consensus_peers.cc:485] T 7b2eae408b9843acaa61e698a3e44cff P 2f306549407b4bb281ccf2994035e4f1 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7b2eae408b9843acaa61e698a3e44cff. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:20.485343 17604 raft_consensus.cc:1261] T 7b2eae408b9843acaa61e698a3e44cff P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:20.485374 17476 raft_consensus.cc:1261] T 7b2eae408b9843acaa61e698a3e44cff P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:20.486310 25361 consensus_queue.cc:1035] T 7b2eae408b9843acaa61e698a3e44cff P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:20.486800 25361 consensus_queue.cc:1035] T 7b2eae408b9843acaa61e698a3e44cff P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:20.489080 25577 raft_consensus.cc:2932] T 7b2eae408b9843acaa61e698a3e44cff P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:54:20.489550 17604 raft_consensus.cc:2932] T 7b2eae408b9843acaa61e698a3e44cff P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:54:20.494314 17476 raft_consensus.cc:2932] T 7b2eae408b9843acaa61e698a3e44cff P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:54:20.497920 17334 catalog_manager.cc:5484] T 7b2eae408b9843acaa61e698a3e44cff P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: config changed from index -1 to 2, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:54:20.504716 25616 ts_tablet_manager.cc:902] T a86548104dca43bd9e1b3ec4d23ec0e4 P 4da690c0fa13407a85c0689ce8de9ade: Initiating tablet copy from peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:54:20.520761 25616 tablet_copy_client.cc:308] T a86548104dca43bd9e1b3ec4d23ec0e4 P 4da690c0fa13407a85c0689ce8de9ade: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.66:38531
I20240502 01:54:20.521311 17627 tablet_copy_service.cc:140] P 3fceccefba274b1dbbd8216af8b0bc29: Received BeginTabletCopySession request for tablet a86548104dca43bd9e1b3ec4d23ec0e4 from peer 4da690c0fa13407a85c0689ce8de9ade ({username='slave'} at 127.11.42.65:48075)
I20240502 01:54:20.521464 17627 tablet_copy_service.cc:161] P 3fceccefba274b1dbbd8216af8b0bc29: Beginning new tablet copy session on tablet a86548104dca43bd9e1b3ec4d23ec0e4 from peer 4da690c0fa13407a85c0689ce8de9ade at {username='slave'} at 127.11.42.65:48075: session id = 4da690c0fa13407a85c0689ce8de9ade-a86548104dca43bd9e1b3ec4d23ec0e4
I20240502 01:54:20.522646 17737 consensus_queue.cc:237] T a390a222562f425eabb11d7f52cbe9a6 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:54:20.524261 17627 tablet_copy_source_session.cc:205] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:20.525260 17476 raft_consensus.cc:1261] T a390a222562f425eabb11d7f52cbe9a6 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:54:20.525578 17672 consensus_peers.cc:485] T a390a222562f425eabb11d7f52cbe9a6 P dfa1152987644bd781b46c7e42a7a685 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a390a222562f425eabb11d7f52cbe9a6. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:20.525905 20361 raft_consensus.cc:1261] T a390a222562f425eabb11d7f52cbe9a6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer dfa1152987644bd781b46c7e42a7a685: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:20.526486 25616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a86548104dca43bd9e1b3ec4d23ec0e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:20.526407 25439 consensus_queue.cc:1035] T a390a222562f425eabb11d7f52cbe9a6 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:20.526733 25660 consensus_queue.cc:1035] T a390a222562f425eabb11d7f52cbe9a6 P dfa1152987644bd781b46c7e42a7a685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:20.529693 24999 raft_consensus.cc:2932] T a390a222562f425eabb11d7f52cbe9a6 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:54:20.531765 25616 tablet_copy_client.cc:783] T a86548104dca43bd9e1b3ec4d23ec0e4 P 4da690c0fa13407a85c0689ce8de9ade: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:20.531955 25616 tablet_copy_client.cc:647] T a86548104dca43bd9e1b3ec4d23ec0e4 P 4da690c0fa13407a85c0689ce8de9ade: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:20.532019 20361 raft_consensus.cc:2932] T a390a222562f425eabb11d7f52cbe9a6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:54:20.536578 17327 catalog_manager.cc:5484] T a390a222562f425eabb11d7f52cbe9a6 P dfa1152987644bd781b46c7e42a7a685 reported cstate change: config changed from index -1 to 2, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New cstate: current_term: 1 leader_uuid: "dfa1152987644bd781b46c7e42a7a685" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:20.537647 25616 tablet_copy_client.cc:523] T a86548104dca43bd9e1b3ec4d23ec0e4 P 4da690c0fa13407a85c0689ce8de9ade: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:20.542502 25616 tablet_bootstrap.cc:492] T a86548104dca43bd9e1b3ec4d23ec0e4 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
I20240502 01:54:20.549616 25666 ts_tablet_manager.cc:902] T bbf05e3b3ad44354be7600c3c07756fd P 2f306549407b4bb281ccf2994035e4f1: Initiating tablet copy from peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:54:20.558565 25665 ts_tablet_manager.cc:902] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 3fceccefba274b1dbbd8216af8b0bc29: Initiating tablet copy from peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:54:20.560209 25616 tablet_bootstrap.cc:492] T a86548104dca43bd9e1b3ec4d23ec0e4 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:20.560545 25616 tablet_bootstrap.cc:492] T a86548104dca43bd9e1b3ec4d23ec0e4 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap complete.
I20240502 01:54:20.562067 17475 raft_consensus.cc:2932] T a390a222562f425eabb11d7f52cbe9a6 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } } }
I20240502 01:54:20.560765 25616 ts_tablet_manager.cc:1367] T a86548104dca43bd9e1b3ec4d23ec0e4 P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.002s
I20240502 01:54:20.563305 25616 raft_consensus.cc:348] T a86548104dca43bd9e1b3ec4d23ec0e4 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } }
I20240502 01:54:20.563541 25616 raft_consensus.cc:729] T a86548104dca43bd9e1b3ec4d23ec0e4 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: LEARNER
I20240502 01:54:20.563716 25616 consensus_queue.cc:260] T a86548104dca43bd9e1b3ec4d23ec0e4 P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } }
I20240502 01:54:20.566085 25616 ts_tablet_manager.cc:1398] T a86548104dca43bd9e1b3ec4d23ec0e4 P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:54:20.566982 17627 tablet_copy_service.cc:342] P 3fceccefba274b1dbbd8216af8b0bc29: Request end of tablet copy session 4da690c0fa13407a85c0689ce8de9ade-a86548104dca43bd9e1b3ec4d23ec0e4 received from {username='slave'} at 127.11.42.65:48075
I20240502 01:54:20.567106 17627 tablet_copy_service.cc:433] P 3fceccefba274b1dbbd8216af8b0bc29: ending tablet copy session 4da690c0fa13407a85c0689ce8de9ade-a86548104dca43bd9e1b3ec4d23ec0e4 on tablet a86548104dca43bd9e1b3ec4d23ec0e4 with peer 4da690c0fa13407a85c0689ce8de9ade
I20240502 01:54:20.589587 25665 tablet_copy_client.cc:308] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 3fceccefba274b1dbbd8216af8b0bc29: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.65:38321
I20240502 01:54:20.590298 17496 tablet_copy_service.cc:140] P 4da690c0fa13407a85c0689ce8de9ade: Received BeginTabletCopySession request for tablet 29fb2b31cd9c4c8fb5923bb849a097d9 from peer 3fceccefba274b1dbbd8216af8b0bc29 ({username='slave'} at 127.11.42.66:43759)
I20240502 01:54:20.590456 17496 tablet_copy_service.cc:161] P 4da690c0fa13407a85c0689ce8de9ade: Beginning new tablet copy session on tablet 29fb2b31cd9c4c8fb5923bb849a097d9 from peer 3fceccefba274b1dbbd8216af8b0bc29 at {username='slave'} at 127.11.42.66:43759: session id = 3fceccefba274b1dbbd8216af8b0bc29-29fb2b31cd9c4c8fb5923bb849a097d9
I20240502 01:54:20.596971 17496 tablet_copy_source_session.cc:205] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 4da690c0fa13407a85c0689ce8de9ade: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:20.598583 25665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29fb2b31cd9c4c8fb5923bb849a097d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:20.604075 25666 tablet_copy_client.cc:308] T bbf05e3b3ad44354be7600c3c07756fd P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.69:34339
I20240502 01:54:20.604779 20253 tablet_copy_service.cc:140] P fdc2ac372492454c85387589a91f63af: Received BeginTabletCopySession request for tablet bbf05e3b3ad44354be7600c3c07756fd from peer 2f306549407b4bb281ccf2994035e4f1 ({username='slave'} at 127.11.42.71:58049)
I20240502 01:54:20.604965 20253 tablet_copy_service.cc:161] P fdc2ac372492454c85387589a91f63af: Beginning new tablet copy session on tablet bbf05e3b3ad44354be7600c3c07756fd from peer 2f306549407b4bb281ccf2994035e4f1 at {username='slave'} at 127.11.42.71:58049: session id = 2f306549407b4bb281ccf2994035e4f1-bbf05e3b3ad44354be7600c3c07756fd
I20240502 01:54:20.608242 20253 tablet_copy_source_session.cc:205] T bbf05e3b3ad44354be7600c3c07756fd P fdc2ac372492454c85387589a91f63af: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:20.611464 25666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbf05e3b3ad44354be7600c3c07756fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:20.620467 25665 tablet_copy_client.cc:783] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 3fceccefba274b1dbbd8216af8b0bc29: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:20.620934 25665 tablet_copy_client.cc:647] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 3fceccefba274b1dbbd8216af8b0bc29: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:20.622547 25666 tablet_copy_client.cc:783] T bbf05e3b3ad44354be7600c3c07756fd P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:20.622777 25666 tablet_copy_client.cc:647] T bbf05e3b3ad44354be7600c3c07756fd P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:20.625944 25666 tablet_copy_client.cc:523] T bbf05e3b3ad44354be7600c3c07756fd P 2f306549407b4bb281ccf2994035e4f1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:20.625948 25665 tablet_copy_client.cc:523] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 3fceccefba274b1dbbd8216af8b0bc29: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:20.629433 25666 tablet_bootstrap.cc:492] T bbf05e3b3ad44354be7600c3c07756fd P 2f306549407b4bb281ccf2994035e4f1: Bootstrap starting.
I20240502 01:54:20.632611 25679 ts_tablet_manager.cc:902] T 0615786e80314766892988f3cff6a74d P fdc2ac372492454c85387589a91f63af: Initiating tablet copy from peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:54:20.634207 20364 consensus_queue.cc:237] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } }
I20240502 01:54:20.638201 25665 tablet_bootstrap.cc:492] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
W20240502 01:54:20.641028 20303 consensus_peers.cc:485] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 28463d45bfde44429d56179c1eec56b0. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:20.642030 17476 raft_consensus.cc:1261] T 28463d45bfde44429d56179c1eec56b0 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:20.642499 17867 raft_consensus.cc:1261] T 28463d45bfde44429d56179c1eec56b0 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:20.643126 25358 consensus_queue.cc:1035] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:20.643615 25358 consensus_queue.cc:1035] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:20.647606 25666 tablet_bootstrap.cc:492] T bbf05e3b3ad44354be7600c3c07756fd P 2f306549407b4bb281ccf2994035e4f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:20.648082 25666 tablet_bootstrap.cc:492] T bbf05e3b3ad44354be7600c3c07756fd P 2f306549407b4bb281ccf2994035e4f1: Bootstrap complete.
I20240502 01:54:20.648308 25666 ts_tablet_manager.cc:1367] T bbf05e3b3ad44354be7600c3c07756fd P 2f306549407b4bb281ccf2994035e4f1: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.003s
I20240502 01:54:20.649075 25666 raft_consensus.cc:348] T bbf05e3b3ad44354be7600c3c07756fd P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
I20240502 01:54:20.649296 25666 raft_consensus.cc:729] T bbf05e3b3ad44354be7600c3c07756fd P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Initialized, Role: LEARNER
I20240502 01:54:20.649520 25666 consensus_queue.cc:260] T bbf05e3b3ad44354be7600c3c07756fd P 2f306549407b4bb281ccf2994035e4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
I20240502 01:54:20.652634 25603 raft_consensus.cc:2932] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:54:20.654408 25666 ts_tablet_manager.cc:1398] T bbf05e3b3ad44354be7600c3c07756fd P 2f306549407b4bb281ccf2994035e4f1: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:54:20.656922 17475 raft_consensus.cc:2932] T 28463d45bfde44429d56179c1eec56b0 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:54:20.662676 17327 catalog_manager.cc:5484] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: config changed from index 4 to 5, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New cstate: current_term: 1 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:20.664958 25665 tablet_bootstrap.cc:492] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:20.666276 25665 tablet_bootstrap.cc:492] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap complete.
I20240502 01:54:20.666504 25665 ts_tablet_manager.cc:1367] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.005s
I20240502 01:54:20.667119 25665 raft_consensus.cc:348] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } }
I20240502 01:54:20.667353 25665 raft_consensus.cc:729] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: LEARNER
I20240502 01:54:20.667536 25665 consensus_queue.cc:260] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } }
I20240502 01:54:20.667735 20253 tablet_copy_service.cc:342] P fdc2ac372492454c85387589a91f63af: Request end of tablet copy session 2f306549407b4bb281ccf2994035e4f1-bbf05e3b3ad44354be7600c3c07756fd received from {username='slave'} at 127.11.42.71:58049
I20240502 01:54:20.667858 20253 tablet_copy_service.cc:433] P fdc2ac372492454c85387589a91f63af: ending tablet copy session 2f306549407b4bb281ccf2994035e4f1-bbf05e3b3ad44354be7600c3c07756fd on tablet bbf05e3b3ad44354be7600c3c07756fd with peer 2f306549407b4bb281ccf2994035e4f1
I20240502 01:54:20.668066 25665 ts_tablet_manager.cc:1398] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:54:20.673135 17496 tablet_copy_service.cc:342] P 4da690c0fa13407a85c0689ce8de9ade: Request end of tablet copy session 3fceccefba274b1dbbd8216af8b0bc29-29fb2b31cd9c4c8fb5923bb849a097d9 received from {username='slave'} at 127.11.42.66:43759
I20240502 01:54:20.673244 17496 tablet_copy_service.cc:433] P 4da690c0fa13407a85c0689ce8de9ade: ending tablet copy session 3fceccefba274b1dbbd8216af8b0bc29-29fb2b31cd9c4c8fb5923bb849a097d9 on tablet 29fb2b31cd9c4c8fb5923bb849a097d9 with peer 3fceccefba274b1dbbd8216af8b0bc29
I20240502 01:54:20.678146 17867 raft_consensus.cc:2932] T 28463d45bfde44429d56179c1eec56b0 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER fdc2ac372492454c85387589a91f63af (127.11.42.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } } }
I20240502 01:54:20.686774 25688 ts_tablet_manager.cc:902] T 38bb8dd915f743f49b26ee2572871375 P fdc2ac372492454c85387589a91f63af: Initiating tablet copy from peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:54:20.699779 25679 tablet_copy_client.cc:308] T 0615786e80314766892988f3cff6a74d P fdc2ac372492454c85387589a91f63af: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.65:38321
I20240502 01:54:20.701475 17496 tablet_copy_service.cc:140] P 4da690c0fa13407a85c0689ce8de9ade: Received BeginTabletCopySession request for tablet 0615786e80314766892988f3cff6a74d from peer fdc2ac372492454c85387589a91f63af ({username='slave'} at 127.11.42.69:58249)
I20240502 01:54:20.701633 17496 tablet_copy_service.cc:161] P 4da690c0fa13407a85c0689ce8de9ade: Beginning new tablet copy session on tablet 0615786e80314766892988f3cff6a74d from peer fdc2ac372492454c85387589a91f63af at {username='slave'} at 127.11.42.69:58249: session id = fdc2ac372492454c85387589a91f63af-0615786e80314766892988f3cff6a74d
I20240502 01:54:20.703608 25688 tablet_copy_client.cc:308] T 38bb8dd915f743f49b26ee2572871375 P fdc2ac372492454c85387589a91f63af: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.72:42411
I20240502 01:54:20.704201 17496 tablet_copy_source_session.cc:205] T 0615786e80314766892988f3cff6a74d P 4da690c0fa13407a85c0689ce8de9ade: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:20.704809 20649 tablet_copy_service.cc:140] P f19cc6cb1e76460aa6711e49a5775091: Received BeginTabletCopySession request for tablet 38bb8dd915f743f49b26ee2572871375 from peer fdc2ac372492454c85387589a91f63af ({username='slave'} at 127.11.42.69:49895)
I20240502 01:54:20.704994 20649 tablet_copy_service.cc:161] P f19cc6cb1e76460aa6711e49a5775091: Beginning new tablet copy session on tablet 38bb8dd915f743f49b26ee2572871375 from peer fdc2ac372492454c85387589a91f63af at {username='slave'} at 127.11.42.69:49895: session id = fdc2ac372492454c85387589a91f63af-38bb8dd915f743f49b26ee2572871375
I20240502 01:54:20.707913 20649 tablet_copy_source_session.cc:205] T 38bb8dd915f743f49b26ee2572871375 P f19cc6cb1e76460aa6711e49a5775091: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:20.708236 25679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0615786e80314766892988f3cff6a74d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:20.713230 25688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38bb8dd915f743f49b26ee2572871375. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:20.714840 25679 tablet_copy_client.cc:783] T 0615786e80314766892988f3cff6a74d P fdc2ac372492454c85387589a91f63af: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:20.715106 25679 tablet_copy_client.cc:647] T 0615786e80314766892988f3cff6a74d P fdc2ac372492454c85387589a91f63af: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:20.716360 25594 ts_tablet_manager.cc:902] T 495700f8ed5e41228f2e597465dad126 P bb31bceba76d466db24b9341c04900c3: Initiating tablet copy from peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:54:20.716885 25679 tablet_copy_client.cc:523] T 0615786e80314766892988f3cff6a74d P fdc2ac372492454c85387589a91f63af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:20.718919 20627 consensus_queue.cc:237] T f3e4c8f98b5e48e3ba6affc2df08943a P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } }
I20240502 01:54:20.719823 25688 tablet_copy_client.cc:783] T 38bb8dd915f743f49b26ee2572871375 P fdc2ac372492454c85387589a91f63af: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:20.720063 25688 tablet_copy_client.cc:647] T 38bb8dd915f743f49b26ee2572871375 P fdc2ac372492454c85387589a91f63af: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:20.722252 25679 tablet_bootstrap.cc:492] T 0615786e80314766892988f3cff6a74d P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:54:20.726847 17864 raft_consensus.cc:1203] T ee28f4aa31504dbfac977c4178aefdb1 P bb31bceba76d466db24b9341c04900c3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:20.731688 20497 raft_consensus.cc:1261] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:54:20.732355 20364 raft_consensus.cc:1261] T f3e4c8f98b5e48e3ba6affc2df08943a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:54:20.732329 25501 consensus_queue.cc:1035] T f3e4c8f98b5e48e3ba6affc2df08943a P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:20.732993 25501 consensus_queue.cc:1035] T f3e4c8f98b5e48e3ba6affc2df08943a P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 01:54:20.740921 20562 consensus_peers.cc:485] T f3e4c8f98b5e48e3ba6affc2df08943a P f19cc6cb1e76460aa6711e49a5775091 -> Peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): Couldn't send request to peer 3fceccefba274b1dbbd8216af8b0bc29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3e4c8f98b5e48e3ba6affc2df08943a. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:20.749377 25688 tablet_copy_client.cc:523] T 38bb8dd915f743f49b26ee2572871375 P fdc2ac372492454c85387589a91f63af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:20.752657 25688 tablet_bootstrap.cc:492] T 38bb8dd915f743f49b26ee2572871375 P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
I20240502 01:54:20.762641 25679 tablet_bootstrap.cc:492] T 0615786e80314766892988f3cff6a74d P fdc2ac372492454c85387589a91f63af: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:20.763036 25679 tablet_bootstrap.cc:492] T 0615786e80314766892988f3cff6a74d P fdc2ac372492454c85387589a91f63af: Bootstrap complete.
I20240502 01:54:20.763273 25679 ts_tablet_manager.cc:1367] T 0615786e80314766892988f3cff6a74d P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.041s user 0.009s sys 0.008s
I20240502 01:54:20.766774 25679 raft_consensus.cc:348] T 0615786e80314766892988f3cff6a74d P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } }
I20240502 01:54:20.767001 25679 raft_consensus.cc:729] T 0615786e80314766892988f3cff6a74d P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: LEARNER
I20240502 01:54:20.767223 25679 consensus_queue.cc:260] T 0615786e80314766892988f3cff6a74d P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } }
I20240502 01:54:20.772979 25188 raft_consensus.cc:2932] T f3e4c8f98b5e48e3ba6affc2df08943a P f19cc6cb1e76460aa6711e49a5775091 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } } }
I20240502 01:54:20.774718 25679 ts_tablet_manager.cc:1398] T 0615786e80314766892988f3cff6a74d P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240502 01:54:20.775278 17496 tablet_copy_service.cc:342] P 4da690c0fa13407a85c0689ce8de9ade: Request end of tablet copy session fdc2ac372492454c85387589a91f63af-0615786e80314766892988f3cff6a74d received from {username='slave'} at 127.11.42.69:58249
I20240502 01:54:20.775403 17496 tablet_copy_service.cc:433] P 4da690c0fa13407a85c0689ce8de9ade: ending tablet copy session fdc2ac372492454c85387589a91f63af-0615786e80314766892988f3cff6a74d on tablet 0615786e80314766892988f3cff6a74d with peer fdc2ac372492454c85387589a91f63af
I20240502 01:54:20.777149 25708 ts_tablet_manager.cc:902] T 6ded7434afab44999620dfa3213c7a82 P 6a36c952c51542c0bf7ea3ec2857a9f9: Initiating tablet copy from peer 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:54:20.785329 20362 raft_consensus.cc:2932] T f3e4c8f98b5e48e3ba6affc2df08943a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } } }
I20240502 01:54:20.789438 25594 tablet_copy_client.cc:308] T 495700f8ed5e41228f2e597465dad126 P bb31bceba76d466db24b9341c04900c3: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.65:38321
I20240502 01:54:20.793205 17496 tablet_copy_service.cc:140] P 4da690c0fa13407a85c0689ce8de9ade: Received BeginTabletCopySession request for tablet 495700f8ed5e41228f2e597465dad126 from peer bb31bceba76d466db24b9341c04900c3 ({username='slave'} at 127.11.42.68:59453)
I20240502 01:54:20.793361 17496 tablet_copy_service.cc:161] P 4da690c0fa13407a85c0689ce8de9ade: Beginning new tablet copy session on tablet 495700f8ed5e41228f2e597465dad126 from peer bb31bceba76d466db24b9341c04900c3 at {username='slave'} at 127.11.42.68:59453: session id = bb31bceba76d466db24b9341c04900c3-495700f8ed5e41228f2e597465dad126
I20240502 01:54:20.796466 17496 tablet_copy_source_session.cc:205] T 495700f8ed5e41228f2e597465dad126 P 4da690c0fa13407a85c0689ce8de9ade: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:20.800029 25594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 495700f8ed5e41228f2e597465dad126. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:20.806941 25594 tablet_copy_client.cc:783] T 495700f8ed5e41228f2e597465dad126 P bb31bceba76d466db24b9341c04900c3: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:20.807134 25594 tablet_copy_client.cc:647] T 495700f8ed5e41228f2e597465dad126 P bb31bceba76d466db24b9341c04900c3: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:20.808799 25594 tablet_copy_client.cc:523] T 495700f8ed5e41228f2e597465dad126 P bb31bceba76d466db24b9341c04900c3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:20.812491 25594 tablet_bootstrap.cc:492] T 495700f8ed5e41228f2e597465dad126 P bb31bceba76d466db24b9341c04900c3: Bootstrap starting.
I20240502 01:54:20.812974 17327 catalog_manager.cc:5484] T f3e4c8f98b5e48e3ba6affc2df08943a P f19cc6cb1e76460aa6711e49a5775091 reported cstate change: config changed from index 5 to 6, NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) added. New cstate: current_term: 2 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:20.817755 25688 tablet_bootstrap.cc:492] T 38bb8dd915f743f49b26ee2572871375 P fdc2ac372492454c85387589a91f63af: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:20.823405 25688 tablet_bootstrap.cc:492] T 38bb8dd915f743f49b26ee2572871375 P fdc2ac372492454c85387589a91f63af: Bootstrap complete.
I20240502 01:54:20.823706 25688 ts_tablet_manager.cc:1367] T 38bb8dd915f743f49b26ee2572871375 P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.071s user 0.003s sys 0.005s
I20240502 01:54:20.826494 25688 raft_consensus.cc:348] T 38bb8dd915f743f49b26ee2572871375 P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } }
I20240502 01:54:20.826745 25688 raft_consensus.cc:729] T 38bb8dd915f743f49b26ee2572871375 P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: LEARNER
I20240502 01:54:20.828105 25688 consensus_queue.cc:260] T 38bb8dd915f743f49b26ee2572871375 P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } }
I20240502 01:54:20.828815 25688 ts_tablet_manager.cc:1398] T 38bb8dd915f743f49b26ee2572871375 P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 01:54:20.829753 20649 tablet_copy_service.cc:342] P f19cc6cb1e76460aa6711e49a5775091: Request end of tablet copy session fdc2ac372492454c85387589a91f63af-38bb8dd915f743f49b26ee2572871375 received from {username='slave'} at 127.11.42.69:49895
I20240502 01:54:20.829864 20649 tablet_copy_service.cc:433] P f19cc6cb1e76460aa6711e49a5775091: ending tablet copy session fdc2ac372492454c85387589a91f63af-38bb8dd915f743f49b26ee2572871375 on tablet 38bb8dd915f743f49b26ee2572871375 with peer fdc2ac372492454c85387589a91f63af
I20240502 01:54:20.844379 25708 tablet_copy_client.cc:308] T 6ded7434afab44999620dfa3213c7a82 P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.71:38623
I20240502 01:54:20.845335 20518 tablet_copy_service.cc:140] P 2f306549407b4bb281ccf2994035e4f1: Received BeginTabletCopySession request for tablet 6ded7434afab44999620dfa3213c7a82 from peer 6a36c952c51542c0bf7ea3ec2857a9f9 ({username='slave'} at 127.11.42.70:51471)
I20240502 01:54:20.845508 20518 tablet_copy_service.cc:161] P 2f306549407b4bb281ccf2994035e4f1: Beginning new tablet copy session on tablet 6ded7434afab44999620dfa3213c7a82 from peer 6a36c952c51542c0bf7ea3ec2857a9f9 at {username='slave'} at 127.11.42.70:51471: session id = 6a36c952c51542c0bf7ea3ec2857a9f9-6ded7434afab44999620dfa3213c7a82
I20240502 01:54:20.848459 20518 tablet_copy_source_session.cc:205] T 6ded7434afab44999620dfa3213c7a82 P 2f306549407b4bb281ccf2994035e4f1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:20.850538 20496 raft_consensus.cc:2932] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } } }
I20240502 01:54:20.851888 25623 ts_tablet_manager.cc:902] T 0399c9d1d7b2408292dc78c185605ee8 P dfa1152987644bd781b46c7e42a7a685: Initiating tablet copy from peer 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:54:20.865543 25594 tablet_bootstrap.cc:492] T 495700f8ed5e41228f2e597465dad126 P bb31bceba76d466db24b9341c04900c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:20.865890 25594 tablet_bootstrap.cc:492] T 495700f8ed5e41228f2e597465dad126 P bb31bceba76d466db24b9341c04900c3: Bootstrap complete.
I20240502 01:54:20.866114 25594 ts_tablet_manager.cc:1367] T 495700f8ed5e41228f2e597465dad126 P bb31bceba76d466db24b9341c04900c3: Time spent bootstrapping tablet: real 0.054s user 0.029s sys 0.008s
I20240502 01:54:20.866724 25594 raft_consensus.cc:348] T 495700f8ed5e41228f2e597465dad126 P bb31bceba76d466db24b9341c04900c3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } }
I20240502 01:54:20.866953 25594 raft_consensus.cc:729] T 495700f8ed5e41228f2e597465dad126 P bb31bceba76d466db24b9341c04900c3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Initialized, Role: LEARNER
I20240502 01:54:20.867113 25594 consensus_queue.cc:260] T 495700f8ed5e41228f2e597465dad126 P bb31bceba76d466db24b9341c04900c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: NON_VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: true } }
I20240502 01:54:20.867636 25594 ts_tablet_manager.cc:1398] T 495700f8ed5e41228f2e597465dad126 P bb31bceba76d466db24b9341c04900c3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:54:20.872058 17496 tablet_copy_service.cc:342] P 4da690c0fa13407a85c0689ce8de9ade: Request end of tablet copy session bb31bceba76d466db24b9341c04900c3-495700f8ed5e41228f2e597465dad126 received from {username='slave'} at 127.11.42.68:59453
I20240502 01:54:20.872167 17496 tablet_copy_service.cc:433] P 4da690c0fa13407a85c0689ce8de9ade: ending tablet copy session bb31bceba76d466db24b9341c04900c3-495700f8ed5e41228f2e597465dad126 on tablet 495700f8ed5e41228f2e597465dad126 with peer bb31bceba76d466db24b9341c04900c3
I20240502 01:54:20.872696 25708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ded7434afab44999620dfa3213c7a82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:20.885618 17476 raft_consensus.cc:1203] T a86548104dca43bd9e1b3ec4d23ec0e4 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:20.888379 25623 tablet_copy_client.cc:308] T 0399c9d1d7b2408292dc78c185605ee8 P dfa1152987644bd781b46c7e42a7a685: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.70:39155
I20240502 01:54:20.891155 20387 tablet_copy_service.cc:140] P 6a36c952c51542c0bf7ea3ec2857a9f9: Received BeginTabletCopySession request for tablet 0399c9d1d7b2408292dc78c185605ee8 from peer dfa1152987644bd781b46c7e42a7a685 ({username='slave'} at 127.11.42.67:35525)
I20240502 01:54:20.891331 20387 tablet_copy_service.cc:161] P 6a36c952c51542c0bf7ea3ec2857a9f9: Beginning new tablet copy session on tablet 0399c9d1d7b2408292dc78c185605ee8 from peer dfa1152987644bd781b46c7e42a7a685 at {username='slave'} at 127.11.42.67:35525: session id = dfa1152987644bd781b46c7e42a7a685-0399c9d1d7b2408292dc78c185605ee8
I20240502 01:54:20.893828 20387 tablet_copy_source_session.cc:205] T 0399c9d1d7b2408292dc78c185605ee8 P 6a36c952c51542c0bf7ea3ec2857a9f9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:20.894862 25623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0399c9d1d7b2408292dc78c185605ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:20.899467 25623 tablet_copy_client.cc:783] T 0399c9d1d7b2408292dc78c185605ee8 P dfa1152987644bd781b46c7e42a7a685: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:20.899664 25623 tablet_copy_client.cc:647] T 0399c9d1d7b2408292dc78c185605ee8 P dfa1152987644bd781b46c7e42a7a685: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:20.901201 25623 tablet_copy_client.cc:523] T 0399c9d1d7b2408292dc78c185605ee8 P dfa1152987644bd781b46c7e42a7a685: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:20.903036 25708 tablet_copy_client.cc:783] T 6ded7434afab44999620dfa3213c7a82 P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:20.903256 25708 tablet_copy_client.cc:647] T 6ded7434afab44999620dfa3213c7a82 P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:20.905584 25623 tablet_bootstrap.cc:492] T 0399c9d1d7b2408292dc78c185605ee8 P dfa1152987644bd781b46c7e42a7a685: Bootstrap starting.
I20240502 01:54:20.906267 25708 tablet_copy_client.cc:523] T 6ded7434afab44999620dfa3213c7a82 P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:20.908824 25708 tablet_bootstrap.cc:492] T 6ded7434afab44999620dfa3213c7a82 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:54:20.909101 25722 ts_tablet_manager.cc:902] T 25ad701a0fec4e80b37df5dec598796a P f19cc6cb1e76460aa6711e49a5775091: Initiating tablet copy from peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:54:20.911442 25720 ts_tablet_manager.cc:902] T e61094fdf9324469afa7ecdc45fd55de P 6a36c952c51542c0bf7ea3ec2857a9f9: Initiating tablet copy from peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:54:20.913514 25623 tablet_bootstrap.cc:492] T 0399c9d1d7b2408292dc78c185605ee8 P dfa1152987644bd781b46c7e42a7a685: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:20.913904 25623 tablet_bootstrap.cc:492] T 0399c9d1d7b2408292dc78c185605ee8 P dfa1152987644bd781b46c7e42a7a685: Bootstrap complete.
I20240502 01:54:20.914177 25623 ts_tablet_manager.cc:1367] T 0399c9d1d7b2408292dc78c185605ee8 P dfa1152987644bd781b46c7e42a7a685: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.001s
I20240502 01:54:20.914865 25623 raft_consensus.cc:348] T 0399c9d1d7b2408292dc78c185605ee8 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:54:20.915166 25623 raft_consensus.cc:729] T 0399c9d1d7b2408292dc78c185605ee8 P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dfa1152987644bd781b46c7e42a7a685, State: Initialized, Role: LEARNER
I20240502 01:54:20.915416 25623 consensus_queue.cc:260] T 0399c9d1d7b2408292dc78c185605ee8 P dfa1152987644bd781b46c7e42a7a685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:54:20.917629 25623 ts_tablet_manager.cc:1398] T 0399c9d1d7b2408292dc78c185605ee8 P dfa1152987644bd781b46c7e42a7a685: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:54:20.918272 20387 tablet_copy_service.cc:342] P 6a36c952c51542c0bf7ea3ec2857a9f9: Request end of tablet copy session dfa1152987644bd781b46c7e42a7a685-0399c9d1d7b2408292dc78c185605ee8 received from {username='slave'} at 127.11.42.67:35525
I20240502 01:54:20.918390 20387 tablet_copy_service.cc:433] P 6a36c952c51542c0bf7ea3ec2857a9f9: ending tablet copy session dfa1152987644bd781b46c7e42a7a685-0399c9d1d7b2408292dc78c185605ee8 on tablet 0399c9d1d7b2408292dc78c185605ee8 with peer dfa1152987644bd781b46c7e42a7a685
I20240502 01:54:20.928797 25688 ts_tablet_manager.cc:902] T 7b2eae408b9843acaa61e698a3e44cff P fdc2ac372492454c85387589a91f63af: Initiating tablet copy from peer 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:54:20.933669 25708 tablet_bootstrap.cc:492] T 6ded7434afab44999620dfa3213c7a82 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:20.934165 25708 tablet_bootstrap.cc:492] T 6ded7434afab44999620dfa3213c7a82 P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap complete.
I20240502 01:54:20.935431 25688 tablet_copy_client.cc:308] T 7b2eae408b9843acaa61e698a3e44cff P fdc2ac372492454c85387589a91f63af: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.71:38623
I20240502 01:54:20.935674 25708 ts_tablet_manager.cc:1367] T 6ded7434afab44999620dfa3213c7a82 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.002s
I20240502 01:54:20.936308 20518 tablet_copy_service.cc:140] P 2f306549407b4bb281ccf2994035e4f1: Received BeginTabletCopySession request for tablet 7b2eae408b9843acaa61e698a3e44cff from peer fdc2ac372492454c85387589a91f63af ({username='slave'} at 127.11.42.69:57643)
I20240502 01:54:20.936436 20518 tablet_copy_service.cc:161] P 2f306549407b4bb281ccf2994035e4f1: Beginning new tablet copy session on tablet 7b2eae408b9843acaa61e698a3e44cff from peer fdc2ac372492454c85387589a91f63af at {username='slave'} at 127.11.42.69:57643: session id = fdc2ac372492454c85387589a91f63af-7b2eae408b9843acaa61e698a3e44cff
I20240502 01:54:20.937914 20233 consensus_queue.cc:237] T 2544ba74e9d146ef963fd92a1a6f8a98 P fdc2ac372492454c85387589a91f63af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:54:20.939033 20518 tablet_copy_source_session.cc:205] T 7b2eae408b9843acaa61e698a3e44cff P 2f306549407b4bb281ccf2994035e4f1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:20.940372 20497 raft_consensus.cc:1261] T 2544ba74e9d146ef963fd92a1a6f8a98 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:20.940964 25523 consensus_queue.cc:1035] T 2544ba74e9d146ef963fd92a1a6f8a98 P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:20.941216 25708 raft_consensus.cc:348] T 6ded7434afab44999620dfa3213c7a82 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } }
I20240502 01:54:20.941469 25708 raft_consensus.cc:729] T 6ded7434afab44999620dfa3213c7a82 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: LEARNER
I20240502 01:54:20.941466 17864 raft_consensus.cc:1261] T 2544ba74e9d146ef963fd92a1a6f8a98 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer fdc2ac372492454c85387589a91f63af: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:20.941692 25708 consensus_queue.cc:260] T 6ded7434afab44999620dfa3213c7a82 P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } }
I20240502 01:54:20.942239 25708 ts_tablet_manager.cc:1398] T 6ded7434afab44999620dfa3213c7a82 P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 01:54:20.943876 20518 tablet_copy_service.cc:342] P 2f306549407b4bb281ccf2994035e4f1: Request end of tablet copy session 6a36c952c51542c0bf7ea3ec2857a9f9-6ded7434afab44999620dfa3213c7a82 received from {username='slave'} at 127.11.42.70:51471
I20240502 01:54:20.944011 20518 tablet_copy_service.cc:433] P 2f306549407b4bb281ccf2994035e4f1: ending tablet copy session 6a36c952c51542c0bf7ea3ec2857a9f9-6ded7434afab44999620dfa3213c7a82 on tablet 6ded7434afab44999620dfa3213c7a82 with peer 6a36c952c51542c0bf7ea3ec2857a9f9
I20240502 01:54:20.944353 25720 tablet_copy_client.cc:308] T e61094fdf9324469afa7ecdc45fd55de P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.66:38531
I20240502 01:54:20.945295 17627 tablet_copy_service.cc:140] P 3fceccefba274b1dbbd8216af8b0bc29: Received BeginTabletCopySession request for tablet e61094fdf9324469afa7ecdc45fd55de from peer 6a36c952c51542c0bf7ea3ec2857a9f9 ({username='slave'} at 127.11.42.70:53247)
I20240502 01:54:20.945456 17627 tablet_copy_service.cc:161] P 3fceccefba274b1dbbd8216af8b0bc29: Beginning new tablet copy session on tablet e61094fdf9324469afa7ecdc45fd55de from peer 6a36c952c51542c0bf7ea3ec2857a9f9 at {username='slave'} at 127.11.42.70:53247: session id = 6a36c952c51542c0bf7ea3ec2857a9f9-e61094fdf9324469afa7ecdc45fd55de
I20240502 01:54:20.946522 25688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b2eae408b9843acaa61e698a3e44cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:20.947605 25722 tablet_copy_client.cc:308] T 25ad701a0fec4e80b37df5dec598796a P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.69:34339
I20240502 01:54:20.948197 20253 tablet_copy_service.cc:140] P fdc2ac372492454c85387589a91f63af: Received BeginTabletCopySession request for tablet 25ad701a0fec4e80b37df5dec598796a from peer f19cc6cb1e76460aa6711e49a5775091 ({username='slave'} at 127.11.42.72:41729)
I20240502 01:54:20.948220 17627 tablet_copy_source_session.cc:205] T e61094fdf9324469afa7ecdc45fd55de P 3fceccefba274b1dbbd8216af8b0bc29: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:20.948318 20253 tablet_copy_service.cc:161] P fdc2ac372492454c85387589a91f63af: Beginning new tablet copy session on tablet 25ad701a0fec4e80b37df5dec598796a from peer f19cc6cb1e76460aa6711e49a5775091 at {username='slave'} at 127.11.42.72:41729: session id = f19cc6cb1e76460aa6711e49a5775091-25ad701a0fec4e80b37df5dec598796a
I20240502 01:54:20.950577 20253 tablet_copy_source_session.cc:205] T 25ad701a0fec4e80b37df5dec598796a P fdc2ac372492454c85387589a91f63af: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:20.951465 25523 consensus_queue.cc:1035] T 2544ba74e9d146ef963fd92a1a6f8a98 P fdc2ac372492454c85387589a91f63af [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:54:20.952955 20167 consensus_peers.cc:485] T 2544ba74e9d146ef963fd92a1a6f8a98 P fdc2ac372492454c85387589a91f63af -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2544ba74e9d146ef963fd92a1a6f8a98. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:20.953756 25722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25ad701a0fec4e80b37df5dec598796a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:20.954617 25720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e61094fdf9324469afa7ecdc45fd55de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:20.956606 25623 ts_tablet_manager.cc:902] T 3dcd890bdd2d413e95401897e9f1bcab P dfa1152987644bd781b46c7e42a7a685: Initiating tablet copy from peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
F20240502 01:54:20.956920 25623 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0399c9d1d7b2408292dc78c185605ee8.queue_time_us, tablet-download-pool-0399c9d1d7b2408292dc78c185605ee8.queue_time_us) (tablet-download-pool-0399c9d1d7b2408292dc78c185605ee8.run_cpu_time_us, tablet-download-pool-0399c9d1d7b2408292dc78c185605ee8.run_cpu_time_us) (tablet-download-pool-0399c9d1d7b2408292dc78c185605ee8.run_wall_time_us, tablet-download-pool-0399c9d1d7b2408292dc78c185605ee8.run_wall_time_us) (tablet-download-pool-1b14ac3413004127b72904b631ec1b94.queue_time_us, tablet-download-pool-1b14ac3413004127b72904b631ec1b94.queue_time_us) (tablet-download-pool-1b14ac3413004127b72904b631ec1b94.run_cpu_time_us, tablet-download-pool-1b14ac3413004127b72904b631ec1b94.run_cpu_time_us) (tablet-download-pool-1b14ac3413004127b72904b631ec1b94.run_wall_time_us, tablet-download-pool-1b14ac3413004127b72904b631ec1b94.run_wall_time_us) (tablet-download-pool-3dcd890bdd2d413e95401897e9f1bcab.queue_time_us, tablet-download-pool-3dcd890bdd2d413e95401897e9f1bcab.queue_time_us) (tablet-download-pool-4aefc61cb95f4e949e7a23a1c677749e.queue_time_us, tablet-download-pool-4aefc61cb95f4e949e7a23a1c677749e.queue_time_us) (tablet-download-pool-4aefc61cb95f4e949e7a23a1c677749e.run_cpu_time_us, tablet-download-pool-4aefc61cb95f4e949e7a23a1c677749e.run_cpu_time_us) (tablet-download-pool-4aefc61cb95f4e949e7a23a1c677749e.run_wall_time_us, tablet-download-pool-4aefc61cb95f4e949e7a23a1c677749e.run_wall_time_us) (tablet-download-pool-61ab80ce377a44ac95a9c574e9a615f2.queue_time_us, tablet-download-pool-61ab80ce377a44ac95a9c574e9a615f2.queue_time_us) (tablet-download-pool-61ab80ce377a44ac95a9c574e9a615f2.run_cpu_time_us, tablet-download-pool-61ab80ce377a44ac95a9c574e9a615f2.run_cpu_time_us) (tablet-download-pool-61ab80ce377a44ac95a9c574e9a615f2.run_wall_time_us, tablet-download-pool-61ab80ce377a44ac95a9c574e9a615f2.run_wall_time_us) (tablet-download-pool-648b1cc4d8b54c5eb0948ca8c1984a1f.queue_time_us, tablet-download-pool-648b1cc4d8b54c5eb0948ca8c1984a1f.queue_time_us) (tablet-download-pool-648b1cc4d8b54c5eb0948ca8c1984a1f.run_cpu_time_us, tablet-download-pool-648b1cc4d8b54c5eb0948ca8c1984a1f.run_cpu_time_us) (tablet-download-pool-648b1cc4d8b54c5eb0948ca8c1984a1f.run_wall_time_us, tablet-download-pool-648b1cc4d8b54c5eb0948ca8c1984a1f.run_wall_time_us) (tablet-download-pool-88e68acb7de346f0ad165cadabd077be.queue_time_us, tablet-download-pool-88e68acb7de346f0ad165cadabd077be.queue_time_us) (tablet-download-pool-88e68acb7de346f0ad165cadabd077be.run_cpu_time_us, tablet-download-pool-88e68acb7de346f0ad165cadabd077be.run_cpu_time_us) (tablet-download-pool-88e68acb7de346f0ad165cadabd077be.run_wall_time_us, tablet-download-pool-88e68acb7de346f0ad165cadabd077be.run_wall_time_us) (tablet-download-pool-8eed067f639542d487de971761e91931.queue_time_us, tablet-download-pool-8eed067f639542d487de971761e91931.queue_time_us) (tablet-download-pool-8eed067f639542d487de971761e91931.run_cpu_time_us, tablet-download-pool-8eed067f639542d487de971761e91931.run_cpu_time_us) (tablet-download-pool-8eed067f639542d487de971761e91931.run_wall_time_us, tablet-download-pool-8eed067f639542d487de971761e91931.run_wall_time_us) (tablet-download-pool-a212c17ba64842c99b3d6a78a3b7cc8a.queue_time_us, tablet-download-pool-a212c17ba64842c99b3d6a78a3b7cc8a.queue_time_us) (tablet-download-pool-a212c17ba64842c99b3d6a78a3b7cc8a.run_cpu_time_us, tablet-download-pool-a212c17ba64842c99b3d6a78a3b7cc8a.run_cpu_time_us) (tablet-download-pool-a212c17ba64842c99b3d6a78a3b7cc8a.run_wall_time_us, tablet-download-pool-a212c17ba64842c99b3d6a78a3b7cc8a.run_wall_time_us) (tablet-download-pool-aebde87a7f114537abd9c1d8b7e97004.queue_time_us, tablet-download-pool-aebde87a7f114537abd9c1d8b7e97004.queue_time_us) (tablet-download-pool-aebde87a7f114537abd9c1d8b7e97004.run_cpu_time_us, tablet-download-pool-aebde87a7f114537abd9c1d8b7e97004.run_cpu_time_us) (tablet-download-pool-aebde87a7f114537abd9c1d8b7e97004.run_wall_time_us, tablet-download-pool-aebde87a7f114537abd9c1d8b7e97004.run_wall_time_us) (tablet-download-pool-bfd7b1d1002c4b57a14d43ed9c6d9f50.queue_time_us, tablet-download-pool-bfd7b1d1002c4b57a14d43ed9c6d9f50.queue_time_us) (tablet-download-pool-bfd7b1d1002c4b57a14d43ed9c6d9f50.run_cpu_time_us, tablet-download-pool-bfd7b1d1002c4b57a14d43ed9c6d9f50.run_cpu_time_us) (tablet-download-pool-bfd7b1d1002c4b57a14d43ed9c6d9f50.run_wall_time_us, tablet-download-pool-bfd7b1d1002c4b57a14d43ed9c6d9f50.run_wall_time_us) (tablet-download-pool-cf0237a4c65b495ba892871dd850bed9.queue_time_us, tablet-download-pool-cf0237a4c65b495ba892871dd850bed9.queue_time_us) (tablet-download-pool-cf0237a4c65b495ba892871dd850bed9.run_cpu_time_us, tablet-download-pool-cf0237a4c65b495ba892871dd850bed9.run_cpu_time_us) (tablet-download-pool-cf0237a4c65b495ba892871dd850bed9.run_wall_time_us, tablet-download-pool-cf0237a4c65b495ba892871dd850bed9.run_wall_time_us) (tablet-download-pool-d10d76ac7b094fe9b230b1afcbed9ca6.queue_time_us, tablet-download-pool-d10d76ac7b094fe9b230b1afcbed9ca6.queue_time_us) (tablet-download-pool-d10d76ac7b094fe9b230b1afcbed9ca6.run_cpu_time_us, tablet-download-pool-d10d76ac7b094fe9b230b1afcbed9ca6.run_cpu_time_us) (tablet-download-pool-d10d76ac7b094fe9b230b1afcbed9ca6.run_wall_time_us, tablet-download-pool-d10d76ac7b094fe9b230b1afcbed9ca6.run_wall_time_us) (tablet-download-pool-fa4d38dc9b674a9b90d4793dad2687ec.queue_time_us, tablet-download-pool-fa4d38dc9b674a9b90d4793dad2687ec.queue_time_us) (tablet-download-pool-fa4d38dc9b674a9b90d4793dad2687ec.run_cpu_time_us, tablet-download-pool-fa4d38dc9b674a9b90d4793dad2687ec.run_cpu_time_us) (tablet-download-pool-fa4d38dc9b674a9b90d4793dad2687ec.run_wall_time_us, tablet-download-pool-fa4d38dc9b674a9b90d4793dad2687ec.run_wall_time_us) (tablet-download-pool-fbd94103cae04473ba81709da39d7ed8.queue_time_us, tablet-download-pool-fbd94103cae04473ba81709da39d7ed8.queue_time_us) (tablet-download-pool-fbd94103cae04473ba81709da39d7ed8.run_cpu_time_us, tablet-download-pool-fbd94103cae04473ba81709da39d7ed8.run_cpu_time_us) (tablet-download-pool-fbd94103cae04473ba81709da39d7ed8.run_wall_time_us, tablet-download-pool-fbd94103cae04473ba81709da39d7ed8.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7fadc5546dcd google::LogMessage::Fail() at ??:0
@ 0x7fadc554ab93 google::LogMessage::SendToLog() at ??:0
@ 0x7fadc55467cc google::LogMessage::Flush() at ??:0
I20240502 01:54:20.964385 25722 tablet_copy_client.cc:783] T 25ad701a0fec4e80b37df5dec598796a P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:20.964620 25722 tablet_copy_client.cc:647] T 25ad701a0fec4e80b37df5dec598796a P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:20.965588 25688 tablet_copy_client.cc:783] T 7b2eae408b9843acaa61e698a3e44cff P fdc2ac372492454c85387589a91f63af: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:20.966681 25722 tablet_copy_client.cc:523] T 25ad701a0fec4e80b37df5dec598796a P f19cc6cb1e76460aa6711e49a5775091: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:20.967512 25720 tablet_copy_client.cc:783] T e61094fdf9324469afa7ecdc45fd55de P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:20.967725 25720 tablet_copy_client.cc:647] T e61094fdf9324469afa7ecdc45fd55de P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:20.969630 25720 tablet_copy_client.cc:523] T e61094fdf9324469afa7ecdc45fd55de P 6a36c952c51542c0bf7ea3ec2857a9f9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:20.970156 25722 tablet_bootstrap.cc:492] T 25ad701a0fec4e80b37df5dec598796a P f19cc6cb1e76460aa6711e49a5775091: Bootstrap starting.
@ 0x7fadc5547f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240502 01:54:20.973317 25688 tablet_copy_client.cc:647] T 7b2eae408b9843acaa61e698a3e44cff P fdc2ac372492454c85387589a91f63af: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:20.974442 25720 tablet_bootstrap.cc:492] T e61094fdf9324469afa7ecdc45fd55de P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap starting.
I20240502 01:54:20.976244 25523 raft_consensus.cc:2932] T 2544ba74e9d146ef963fd92a1a6f8a98 P fdc2ac372492454c85387589a91f63af [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:20.977908 17476 raft_consensus.cc:1203] T f6cac9478db94a21a9799b80d24c8dc9 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:20.979233 20497 raft_consensus.cc:2932] T 2544ba74e9d146ef963fd92a1a6f8a98 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:20.989652 17737 raft_consensus.cc:1203] T fa4d38dc9b674a9b90d4793dad2687ec P dfa1152987644bd781b46c7e42a7a685 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:20.996708 17606 raft_consensus.cc:1203] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:21.003842 17334 catalog_manager.cc:5484] T 2544ba74e9d146ef963fd92a1a6f8a98 P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New cstate: current_term: 1 leader_uuid: "fdc2ac372492454c85387589a91f63af" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:21.024225 25688 tablet_copy_client.cc:523] T 7b2eae408b9843acaa61e698a3e44cff P fdc2ac372492454c85387589a91f63af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:21.029834 25688 tablet_bootstrap.cc:492] T 7b2eae408b9843acaa61e698a3e44cff P fdc2ac372492454c85387589a91f63af: Bootstrap starting.
@ 0x7fadc8287742 kudu::TraceMetrics::InternName() at ??:0
I20240502 01:54:21.035338 17867 raft_consensus.cc:2932] T 2544ba74e9d146ef963fd92a1a6f8a98 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } } }
I20240502 01:54:21.053035 25720 tablet_bootstrap.cc:492] T e61094fdf9324469afa7ecdc45fd55de P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:21.053474 25720 tablet_bootstrap.cc:492] T e61094fdf9324469afa7ecdc45fd55de P 6a36c952c51542c0bf7ea3ec2857a9f9: Bootstrap complete.
I20240502 01:54:21.053547 25722 tablet_bootstrap.cc:492] T 25ad701a0fec4e80b37df5dec598796a P f19cc6cb1e76460aa6711e49a5775091: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:21.053819 25720 ts_tablet_manager.cc:1367] T e61094fdf9324469afa7ecdc45fd55de P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent bootstrapping tablet: real 0.080s user 0.010s sys 0.017s
I20240502 01:54:21.053961 25722 tablet_bootstrap.cc:492] T 25ad701a0fec4e80b37df5dec598796a P f19cc6cb1e76460aa6711e49a5775091: Bootstrap complete.
I20240502 01:54:21.054180 25722 ts_tablet_manager.cc:1367] T 25ad701a0fec4e80b37df5dec598796a P f19cc6cb1e76460aa6711e49a5775091: Time spent bootstrapping tablet: real 0.084s user 0.027s sys 0.010s
I20240502 01:54:21.054388 25720 raft_consensus.cc:348] T e61094fdf9324469afa7ecdc45fd55de P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } }
I20240502 01:54:21.054544 25720 raft_consensus.cc:729] T e61094fdf9324469afa7ecdc45fd55de P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Initialized, Role: LEARNER
I20240502 01:54:21.054692 25720 consensus_queue.cc:260] T e61094fdf9324469afa7ecdc45fd55de P 6a36c952c51542c0bf7ea3ec2857a9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: NON_VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: true } }
I20240502 01:54:21.054916 25722 raft_consensus.cc:348] T 25ad701a0fec4e80b37df5dec598796a P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:54:21.055126 25722 raft_consensus.cc:729] T 25ad701a0fec4e80b37df5dec598796a P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f19cc6cb1e76460aa6711e49a5775091, State: Initialized, Role: LEARNER
I20240502 01:54:21.055326 25722 consensus_queue.cc:260] T 25ad701a0fec4e80b37df5dec598796a P f19cc6cb1e76460aa6711e49a5775091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:54:21.058367 25720 ts_tablet_manager.cc:1398] T e61094fdf9324469afa7ecdc45fd55de P 6a36c952c51542c0bf7ea3ec2857a9f9: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:54:21.059055 17627 tablet_copy_service.cc:342] P 3fceccefba274b1dbbd8216af8b0bc29: Request end of tablet copy session 6a36c952c51542c0bf7ea3ec2857a9f9-e61094fdf9324469afa7ecdc45fd55de received from {username='slave'} at 127.11.42.70:53247
I20240502 01:54:21.059180 17627 tablet_copy_service.cc:433] P 3fceccefba274b1dbbd8216af8b0bc29: ending tablet copy session 6a36c952c51542c0bf7ea3ec2857a9f9-e61094fdf9324469afa7ecdc45fd55de on tablet e61094fdf9324469afa7ecdc45fd55de with peer 6a36c952c51542c0bf7ea3ec2857a9f9
I20240502 01:54:21.061383 25679 ts_tablet_manager.cc:902] T 28463d45bfde44429d56179c1eec56b0 P fdc2ac372492454c85387589a91f63af: Initiating tablet copy from peer 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
@ 0x7fadc8253f8b kudu::ThreadPool::ThreadPool() at ??:0
F20240502 01:54:21.061686 25679 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0615786e80314766892988f3cff6a74d.queue_time_us, tablet-download-pool-0615786e80314766892988f3cff6a74d.queue_time_us) (tablet-download-pool-0615786e80314766892988f3cff6a74d.run_cpu_time_us, tablet-download-pool-0615786e80314766892988f3cff6a74d.run_cpu_time_us) (tablet-download-pool-0615786e80314766892988f3cff6a74d.run_wall_time_us, tablet-download-pool-0615786e80314766892988f3cff6a74d.run_wall_time_us) (tablet-download-pool-0618761d30f34cfe9650d13c9275cdcb.queue_time_us, tablet-download-pool-0618761d30f34cfe9650d13c9275cdcb.queue_time_us) (tablet-download-pool-0618761d30f34cfe9650d13c9275cdcb.run_cpu_time_us, tablet-download-pool-0618761d30f34cfe9650d13c9275cdcb.run_cpu_time_us) (tablet-download-pool-0618761d30f34cfe9650d13c9275cdcb.run_wall_time_us, tablet-download-pool-0618761d30f34cfe9650d13c9275cdcb.run_wall_time_us) (tablet-download-pool-0d9f168de0d1410da2518abf936035ff.queue_time_us, tablet-download-pool-0d9f168de0d1410da2518abf936035ff.queue_time_us) (tablet-download-pool-0d9f168de0d1410da2518abf936035ff.run_cpu_time_us, tablet-download-pool-0d9f168de0d1410da2518abf936035ff.run_cpu_time_us) (tablet-download-pool-0d9f168de0d1410da2518abf936035ff.run_wall_time_us, tablet-download-pool-0d9f168de0d1410da2518abf936035ff.run_wall_time_us) (tablet-download-pool-28463d45bfde44429d56179c1eec56b0.queue_time_us, tablet-download-pool-28463d45bfde44429d56179c1eec56b0.queue_time_us) (tablet-download-pool-29d8cb76dc514e7b9002bd2291d8ee8d.queue_time_us, tablet-download-pool-29d8cb76dc514e7b9002bd2291d8ee8d.queue_time_us) (tablet-download-pool-29d8cb76dc514e7b9002bd2291d8ee8d.run_cpu_time_us, tablet-download-pool-29d8cb76dc514e7b9002bd2291d8ee8d.run_cpu_time_us) (tablet-download-pool-29d8cb76dc514e7b9002bd2291d8ee8d.run_wall_time_us, tablet-download-pool-29d8cb76dc514e7b9002bd2291d8ee8d.run_wall_time_us) (tablet-download-pool-30f43088bd0f48ee86907df16194b9be.queue_time_us, tablet-download-pool-30f43088bd0f48ee86907df16194b9be.queue_time_us) (tablet-download-pool-30f43088bd0f48ee86907df16194b9be.run_cpu_time_us, tablet-download-pool-30f43088bd0f48ee86907df16194b9be.run_cpu_time_us) (tablet-download-pool-30f43088bd0f48ee86907df16194b9be.run_wall_time_us, tablet-download-pool-30f43088bd0f48ee86907df16194b9be.run_wall_time_us) (tablet-download-pool-38bb8dd915f743f49b26ee2572871375.queue_time_us, tablet-download-pool-38bb8dd915f743f49b26ee2572871375.queue_time_us) (tablet-download-pool-38bb8dd915f743f49b26ee2572871375.run_cpu_time_us, tablet-download-pool-38bb8dd915f743f49b26ee2572871375.run_cpu_time_us) (tablet-download-pool-38bb8dd915f743f49b26ee2572871375.run_wall_time_us, tablet-download-pool-38bb8dd915f743f49b26ee2572871375.run_wall_time_us) (tablet-download-pool-5152c002eadd4d54ab10a093bce45f17.queue_time_us, tablet-download-pool-5152c002eadd4d54ab10a093bce45f17.queue_time_us) (tablet-download-pool-5152c002eadd4d54ab10a093bce45f17.run_cpu_time_us, tablet-download-pool-5152c002eadd4d54ab10a093bce45f17.run_cpu_time_us) (tablet-download-pool-5152c002eadd4d54ab10a093bce45f17.run_wall_time_us, tablet-download-pool-5152c002eadd4d54ab10a093bce45f17.run_wall_time_us) (tablet-download-pool-52430bbf347a4c5191edb23e1aa5d385.queue_time_us, tablet-download-pool-52430bbf347a4c5191edb23e1aa5d385.queue_time_us) (tablet-download-pool-52430bbf347a4c5191edb23e1aa5d385.run_cpu_time_us, tablet-download-pool-52430bbf347a4c5191edb23e1aa5d385.run_cpu_time_us) (tablet-download-pool-52430bbf347a4c5191edb23e1aa5d385.run_wall_time_us, tablet-download-pool-52430bbf347a4c5191edb23e1aa5d385.run_wall_time_us) (tablet-download-pool-59c288e7279f4119afb6c3bf860e6dc3.queue_time_us, tablet-download-pool-59c288e7279f4119afb6c3bf860e6dc3.queue_time_us) (tablet-download-pool-59c288e7279f4119afb6c3bf860e6dc3.run_cpu_time_us, tablet-download-pool-59c288e7279f4119afb6c3bf860e6dc3.run_cpu_time_us) (tablet-download-pool-59c288e7279f4119afb6c3bf860e6dc3.run_wall_time_us, tablet-download-pool-59c288e7279f4119afb6c3bf860e6dc3.run_wall_time_us) (tablet-download-pool-5df83f83d2674155afba93c2d992af0a.queue_time_us, tablet-download-pool-5df83f83d2674155afba93c2d992af0a.queue_time_us) (tablet-download-pool-5df83f83d2674155afba93c2d992af0a.run_cpu_time_us, tablet-download-pool-5df83f83d2674155afba93c2d992af0a.run_cpu_time_us) (tablet-download-pool-5df83f83d2674155afba93c2d992af0a.run_wall_time_us, tablet-download-pool-5df83f83d2674155afba93c2d992af0a.run_wall_time_us) (tablet-download-pool-7b2eae408b9843acaa61e698a3e44cff.queue_time_us, tablet-download-pool-7b2eae408b9843acaa61e698a3e44cff.queue_time_us) (tablet-download-pool-7b2eae408b9843acaa61e698a3e44cff.run_cpu_time_us, tablet-download-pool-7b2eae408b9843acaa61e698a3e44cff.run_cpu_time_us) (tablet-download-pool-7b2eae408b9843acaa61e698a3e44cff.run_wall_time_us, tablet-download-pool-7b2eae408b9843acaa61e698a3e44cff.run_wall_time_us) (tablet-download-pool-8e7f88b6c509477fb0900024e7de8f00.queue_time_us, tablet-download-pool-8e7f88b6c509477fb0900024e7de8f00.queue_time_us) (tablet-download-pool-8e7f88b6c509477fb0900024e7de8f00.run_cpu_time_us, tablet-download-pool-8e7f88b6c509477fb0900024e7de8f00.run_cpu_time_us) (tablet-download-pool-8e7f88b6c509477fb0900024e7de8f00.run_wall_time_us, tablet-download-pool-8e7f88b6c509477fb0900024e7de8f00.run_wall_time_us) (tablet-download-pool-ed1522ec150a46f28fd553f9b563651f.queue_time_us, tablet-download-pool-ed1522ec150a46f28fd553f9b563651f.queue_time_us) (tablet-download-pool-ed1522ec150a46f28fd553f9b563651f.run_cpu_time_us, tablet-download-pool-ed1522ec150a46f28fd553f9b563651f.run_cpu_time_us) (tablet-download-pool-ed1522ec150a46f28fd553f9b563651f.run_wall_time_us, tablet-download-pool-ed1522ec150a46f28fd553f9b563651f.run_wall_time_us) (tablet-download-pool-f0b3862e504d4b43a60477e5efef29b1.queue_time_us, tablet-download-pool-f0b3862e504d4b43a60477e5efef29b1.queue_time_us) (tablet-download-pool-f0b3862e504d4b43a60477e5efef29b1.run_cpu_time_us, tablet-download-pool-f0b3862e504d4b43a60477e5efef29b1.run_cpu_time_us) (tablet-download-pool-f0b3862e504d4b43a60477e5efef29b1.run_wall_time_us, tablet-download-pool-f0b3862e504d4b43a60477e5efef29b1.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240502 01:54:21.063652 25688 tablet_bootstrap.cc:492] T 7b2eae408b9843acaa61e698a3e44cff P fdc2ac372492454c85387589a91f63af: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:21.064062 25688 tablet_bootstrap.cc:492] T 7b2eae408b9843acaa61e698a3e44cff P fdc2ac372492454c85387589a91f63af: Bootstrap complete.
I20240502 01:54:21.064327 25688 ts_tablet_manager.cc:1367] T 7b2eae408b9843acaa61e698a3e44cff P fdc2ac372492454c85387589a91f63af: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.007s
@ 0x7fce37107dcd google::LogMessage::Fail() at ??:0
@ 0x7fadc824dcd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7fce3710bb93 google::LogMessage::SendToLog() at ??:0
I20240502 01:54:21.069047 25722 ts_tablet_manager.cc:1398] T 25ad701a0fec4e80b37df5dec598796a P f19cc6cb1e76460aa6711e49a5775091: Time spent starting tablet: real 0.014s user 0.003s sys 0.000s
@ 0x7fce371077cc google::LogMessage::Flush() at ??:0
I20240502 01:54:21.070556 20253 tablet_copy_service.cc:342] P fdc2ac372492454c85387589a91f63af: Request end of tablet copy session f19cc6cb1e76460aa6711e49a5775091-25ad701a0fec4e80b37df5dec598796a received from {username='slave'} at 127.11.42.72:41729
I20240502 01:54:21.070694 20253 tablet_copy_service.cc:433] P fdc2ac372492454c85387589a91f63af: ending tablet copy session f19cc6cb1e76460aa6711e49a5775091-25ad701a0fec4e80b37df5dec598796a on tablet 25ad701a0fec4e80b37df5dec598796a with peer f19cc6cb1e76460aa6711e49a5775091
I20240502 01:54:21.072039 25688 raft_consensus.cc:348] T 7b2eae408b9843acaa61e698a3e44cff P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } }
I20240502 01:54:21.072278 25688 raft_consensus.cc:729] T 7b2eae408b9843acaa61e698a3e44cff P fdc2ac372492454c85387589a91f63af [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fdc2ac372492454c85387589a91f63af, State: Initialized, Role: LEARNER
@ 0x7fce37108f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240502 01:54:21.072508 25688 consensus_queue.cc:260] T 7b2eae408b9843acaa61e698a3e44cff P fdc2ac372492454c85387589a91f63af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } }
@ 0x7fadd76a08de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240502 01:54:21.076180 25688 ts_tablet_manager.cc:1398] T 7b2eae408b9843acaa61e698a3e44cff P fdc2ac372492454c85387589a91f63af: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20240502 01:54:21.077605 20518 tablet_copy_service.cc:342] P 2f306549407b4bb281ccf2994035e4f1: Request end of tablet copy session fdc2ac372492454c85387589a91f63af-7b2eae408b9843acaa61e698a3e44cff received from {username='slave'} at 127.11.42.69:57643
@ 0x7fadd76b8daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240502 01:54:21.077740 20518 tablet_copy_service.cc:433] P 2f306549407b4bb281ccf2994035e4f1: ending tablet copy session fdc2ac372492454c85387589a91f63af-7b2eae408b9843acaa61e698a3e44cff on tablet 7b2eae408b9843acaa61e698a3e44cff with peer fdc2ac372492454c85387589a91f63af
@ 0x7fadd77e3480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240502 01:54:21.088789 20497 consensus_queue.cc:237] T adc2b57af23c41aebd1258ff8b60d074 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } }
@ 0x7fadd781ba8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7fadd77fe981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7fce39e48742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7fce39e14f8b kudu::ThreadPool::ThreadPool() at ??:0
I20240502 01:54:21.102432 20364 raft_consensus.cc:1261] T adc2b57af23c41aebd1258ff8b60d074 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:21.103041 25577 consensus_queue.cc:1035] T adc2b57af23c41aebd1258ff8b60d074 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:21.103928 20627 raft_consensus.cc:1261] T adc2b57af23c41aebd1258ff8b60d074 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 01:54:21.104699 20433 consensus_peers.cc:485] T adc2b57af23c41aebd1258ff8b60d074 P 2f306549407b4bb281ccf2994035e4f1 -> Peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Couldn't send request to peer 4da690c0fa13407a85c0689ce8de9ade. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: adc2b57af23c41aebd1258ff8b60d074. This is attempt 1: this message will repeat every 5th retry.
@ 0x7fce39e0ecd6 kudu::ThreadPoolBuilder::Build() at ??:0
I20240502 01:54:21.106067 25577 consensus_queue.cc:1035] T adc2b57af23c41aebd1258ff8b60d074 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:21.110749 25722 ts_tablet_manager.cc:902] T a390a222562f425eabb11d7f52cbe9a6 P f19cc6cb1e76460aa6711e49a5775091: Initiating tablet copy from peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
F20240502 01:54:21.111119 25722 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0618761d30f34cfe9650d13c9275cdcb.queue_time_us, tablet-download-pool-0618761d30f34cfe9650d13c9275cdcb.queue_time_us) (tablet-download-pool-0618761d30f34cfe9650d13c9275cdcb.run_cpu_time_us, tablet-download-pool-0618761d30f34cfe9650d13c9275cdcb.run_cpu_time_us) (tablet-download-pool-0618761d30f34cfe9650d13c9275cdcb.run_wall_time_us, tablet-download-pool-0618761d30f34cfe9650d13c9275cdcb.run_wall_time_us) (tablet-download-pool-0d9f168de0d1410da2518abf936035ff.queue_time_us, tablet-download-pool-0d9f168de0d1410da2518abf936035ff.queue_time_us) (tablet-download-pool-0d9f168de0d1410da2518abf936035ff.run_cpu_time_us, tablet-download-pool-0d9f168de0d1410da2518abf936035ff.run_cpu_time_us) (tablet-download-pool-0d9f168de0d1410da2518abf936035ff.run_wall_time_us, tablet-download-pool-0d9f168de0d1410da2518abf936035ff.run_wall_time_us) (tablet-download-pool-10ec78a08ab04678897a6892988e515f.queue_time_us, tablet-download-pool-10ec78a08ab04678897a6892988e515f.queue_time_us) (tablet-download-pool-10ec78a08ab04678897a6892988e515f.run_cpu_time_us, tablet-download-pool-10ec78a08ab04678897a6892988e515f.run_cpu_time_us) (tablet-download-pool-10ec78a08ab04678897a6892988e515f.run_wall_time_us, tablet-download-pool-10ec78a08ab04678897a6892988e515f.run_wall_time_us) (tablet-download-pool-18549019e31b47d899d011cef0637221.queue_time_us, tablet-download-pool-18549019e31b47d899d011cef0637221.queue_time_us) (tablet-download-pool-18549019e31b47d899d011cef0637221.run_cpu_time_us, tablet-download-pool-18549019e31b47d899d011cef0637221.run_cpu_time_us) (tablet-download-pool-18549019e31b47d899d011cef0637221.run_wall_time_us, tablet-download-pool-18549019e31b47d899d011cef0637221.run_wall_time_us) (tablet-download-pool-1f01949cf70440b38e59a5020fd48c44.queue_time_us, tablet-download-pool-1f01949cf70440b38e59a5020fd48c44.queue_time_us) (tablet-download-pool-1f01949cf70440b38e59a5020fd48c44.run_cpu_time_us, tablet-download-pool-1f01949cf70440b38e59a5020fd48c44.run_cpu_time_us) (tablet-download-pool-1f01949cf70440b38e59a5020fd48c44.run_wall_time_us, tablet-download-pool-1f01949cf70440b38e59a5020fd48c44.run_wall_time_us) (tablet-download-pool-25ad701a0fec4e80b37df5dec598796a.queue_time_us, tablet-download-pool-25ad701a0fec4e80b37df5dec598796a.queue_time_us) (tablet-download-pool-25ad701a0fec4e80b37df5dec598796a.run_cpu_time_us, tablet-download-pool-25ad701a0fec4e80b37df5dec598796a.run_cpu_time_us) (tablet-download-pool-25ad701a0fec4e80b37df5dec598796a.run_wall_time_us, tablet-download-pool-25ad701a0fec4e80b37df5dec598796a.run_wall_time_us) (tablet-download-pool-4099f4b2ae164020839faab53957750f.queue_time_us, tablet-download-pool-4099f4b2ae164020839faab53957750f.queue_time_us) (tablet-download-pool-4099f4b2ae164020839faab53957750f.run_cpu_time_us, tablet-download-pool-4099f4b2ae164020839faab53957750f.run_cpu_time_us) (tablet-download-pool-4099f4b2ae164020839faab53957750f.run_wall_time_us, tablet-download-pool-4099f4b2ae164020839faab53957750f.run_wall_time_us) (tablet-download-pool-4aefc61cb95f4e949e7a23a1c677749e.queue_time_us, tablet-download-pool-4aefc61cb95f4e949e7a23a1c677749e.queue_time_us) (tablet-download-pool-4aefc61cb95f4e949e7a23a1c677749e.run_cpu_time_us, tablet-download-pool-4aefc61cb95f4e949e7a23a1c677749e.run_cpu_time_us) (tablet-download-pool-4aefc61cb95f4e949e7a23a1c677749e.run_wall_time_us, tablet-download-pool-4aefc61cb95f4e949e7a23a1c677749e.run_wall_time_us) (tablet-download-pool-4bb42883b1a0475b863d008bd01fa70a.queue_time_us, tablet-download-pool-4bb42883b1a0475b863d008bd01fa70a.queue_time_us) (tablet-download-pool-4bb42883b1a0475b863d008bd01fa70a.run_cpu_time_us, tablet-download-pool-4bb42883b1a0475b863d008bd01fa70a.run_cpu_time_us) (tablet-download-pool-4bb42883b1a0475b863d008bd01fa70a.run_wall_time_us, tablet-download-pool-4bb42883b1a0475b863d008bd01fa70a.run_wall_time_us) (tablet-download-pool-5df83f83d2674155afba93c2d992af0a.queue_time_us, tablet-download-pool-5df83f83d2674155afba93c2d992af0a.queue_time_us) (tablet-download-pool-5df83f83d2674155afba93c2d992af0a.run_cpu_time_us, tablet-download-pool-5df83f83d2674155afba93c2d992af0a.run_cpu_time_us) (tablet-download-pool-5df83f83d2674155afba93c2d992af0a.run_wall_time_us, tablet-download-pool-5df83f83d2674155afba93c2d992af0a.run_wall_time_us) (tablet-download-pool-7c697210a14d4982a939ea7965cb769d.queue_time_us, tablet-download-pool-7c697210a14d4982a939ea7965cb769d.queue_time_us) (tablet-download-pool-7c697210a14d4982a939ea7965cb769d.run_cpu_time_us, tablet-download-pool-7c697210a14d4982a939ea7965cb769d.run_cpu_time_us) (tablet-download-pool-7c697210a14d4982a939ea7965cb769d.run_wall_time_us, tablet-download-pool-7c697210a14d4982a939ea7965cb769d.run_wall_time_us) (tablet-download-pool-9642b9b37a784cf881bfc4971d8c9cb6.queue_time_us, tablet-download-pool-9642b9b37a784cf881bfc4971d8c9cb6.queue_time_us) (tablet-download-pool-9642b9b37a784cf881bfc4971d8c9cb6.run_cpu_time_us, tablet-download-pool-9642b9b37a784cf881bfc4971d8c9cb6.run_cpu_time_us) (tablet-download-pool-9642b9b37a784cf881bfc4971d8c9cb6.run_wall_time_us, tablet-download-pool-9642b9b37a784cf881bfc4971d8c9cb6.run_wall_time_us) (tablet-download-pool-a212c17ba64842c99b3d6a78a3b7cc8a.queue_time_us, tablet-download-pool-a212c17ba64842c99b3d6a78a3b7cc8a.queue_time_us) (tablet-download-pool-a212c17ba64842c99b3d6a78a3b7cc8a.run_cpu_time_us, tablet-download-pool-a212c17ba64842c99b3d6a78a3b7cc8a.run_cpu_time_us) (tablet-download-pool-a212c17ba64842c99b3d6a78a3b7cc8a.run_wall_time_us, tablet-download-pool-a212c17ba64842c99b3d6a78a3b7cc8a.run_wall_time_us) (tablet-download-pool-a390a222562f425eabb11d7f52cbe9a6.queue_time_us, tablet-download-pool-a390a222562f425eabb11d7f52cbe9a6.queue_time_us) (tablet-download-pool-f3e4c8f98b5e48e3ba6affc2df08943a.queue_time_us, tablet-download-pool-f3e4c8f98b5e48e3ba6affc2df08943a.queue_time_us) (tablet-download-pool-f3e4c8f98b5e48e3ba6affc2df08943a.run_cpu_time_us, tablet-download-pool-f3e4c8f98b5e48e3ba6affc2df08943a.run_cpu_time_us) (tablet-download-pool-f3e4c8f98b5e48e3ba6affc2df08943a.run_wall_time_us, tablet-download-pool-f3e4c8f98b5e48e3ba6affc2df08943a.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
@ 0x7fce492618de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
*** Check failure stack trace: ***
@ 0x7fadd77fe70e std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f597d6d2dcd google::LogMessage::Fail() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fce49279daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f597d6d6b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fce493a4480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f597d6d27cc google::LogMessage::Flush() at ??:0
@ 0x7fce493dca8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240502 01:54:21.124809 25577 raft_consensus.cc:2932] T adc2b57af23c41aebd1258ff8b60d074 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } } }
@ 0x7f597d6d3f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fce493bf981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7fce493bf70e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f5980413742 kudu::TraceMetrics::InternName() at ??:0
I20240502 01:54:21.138878 20497 raft_consensus.cc:1203] T bbf05e3b3ad44354be7600c3c07756fd P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
@ 0x7fce39e18b75 kudu::ThreadPool::DispatchThread() at ??:0
I20240502 01:54:21.142163 17326 catalog_manager.cc:5484] T adc2b57af23c41aebd1258ff8b60d074 P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: config changed from index -1 to 2, NON_VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) added. New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
@ 0x7f59803dff8b kudu::ThreadPool::ThreadPool() at ??:0
I20240502 01:54:21.145512 20361 raft_consensus.cc:2932] T adc2b57af23c41aebd1258ff8b60d074 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } } }
@ 0x7fce39e1c19c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fce39e1c019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f59803d9cd6 kudu::ThreadPoolBuilder::Build() at ??:0
I20240502 01:54:21.156167 24955 raft_consensus.cc:1050] T ee28f4aa31504dbfac977c4178aefdb1 P f19cc6cb1e76460aa6711e49a5775091: attempting to promote NON_VOTER bb31bceba76d466db24b9341c04900c3 to VOTER
I20240502 01:54:21.156997 24955 consensus_queue.cc:237] T ee28f4aa31504dbfac977c4178aefdb1 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } }
@ 0x7fce39df701e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f598f82c8de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240502 01:54:21.159562 20627 raft_consensus.cc:2932] T adc2b57af23c41aebd1258ff8b60d074 P f19cc6cb1e76460aa6711e49a5775091 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } } }
@ 0x7fce3d17c6db start_thread at ??:0
@ 0x7fce35fba71f clone at ??:0
I20240502 01:54:21.166164 17737 raft_consensus.cc:1261] T ee28f4aa31504dbfac977c4178aefdb1 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:21.167090 17867 raft_consensus.cc:1261] T ee28f4aa31504dbfac977c4178aefdb1 P bb31bceba76d466db24b9341c04900c3 [term 1 LEARNER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:21.167214 20497 raft_consensus.cc:1261] T ee28f4aa31504dbfac977c4178aefdb1 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:21.168233 25501 consensus_queue.cc:1035] T ee28f4aa31504dbfac977c4178aefdb1 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:21.168776 25501 consensus_queue.cc:1035] T ee28f4aa31504dbfac977c4178aefdb1 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:21.169324 25501 consensus_queue.cc:1035] T ee28f4aa31504dbfac977c4178aefdb1 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
@ 0x7f598f844daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f598f96f480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240502 01:54:21.208729 24955 raft_consensus.cc:2932] T ee28f4aa31504dbfac977c4178aefdb1 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:21.209211 17867 raft_consensus.cc:2932] T ee28f4aa31504dbfac977c4178aefdb1 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:21.210724 25770 ts_tablet_manager.cc:902] T f3e4c8f98b5e48e3ba6affc2df08943a P 3fceccefba274b1dbbd8216af8b0bc29: Initiating tablet copy from peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:54:21.217219 17736 raft_consensus.cc:2932] T ee28f4aa31504dbfac977c4178aefdb1 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
@ 0x7fadc8257b75 kudu::ThreadPool::DispatchThread() at ??:0
I20240502 01:54:21.225371 17334 catalog_manager.cc:5484] T ee28f4aa31504dbfac977c4178aefdb1 P bb31bceba76d466db24b9341c04900c3 reported cstate change: config changed from index 2 to 3, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:21.225917 20496 raft_consensus.cc:2932] T ee28f4aa31504dbfac977c4178aefdb1 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:21.236323 20364 raft_consensus.cc:1203] T e61094fdf9324469afa7ecdc45fd55de P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:21.238207 20627 consensus_queue.cc:237] T ee28f4aa31504dbfac977c4178aefdb1 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } }
I20240502 01:54:21.240280 17864 raft_consensus.cc:1261] T ee28f4aa31504dbfac977c4178aefdb1 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:21.243842 25188 consensus_queue.cc:1035] T ee28f4aa31504dbfac977c4178aefdb1 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:21.246305 17737 raft_consensus.cc:1261] T ee28f4aa31504dbfac977c4178aefdb1 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Refusing update from remote peer f19cc6cb1e76460aa6711e49a5775091: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:21.247165 25188 consensus_queue.cc:1035] T ee28f4aa31504dbfac977c4178aefdb1 P f19cc6cb1e76460aa6711e49a5775091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:21.253293 17867 raft_consensus.cc:1203] T 495700f8ed5e41228f2e597465dad126 P bb31bceba76d466db24b9341c04900c3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:21.258699 17864 raft_consensus.cc:2932] T ee28f4aa31504dbfac977c4178aefdb1 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:21.264847 25188 raft_consensus.cc:2932] T ee28f4aa31504dbfac977c4178aefdb1 P f19cc6cb1e76460aa6711e49a5775091 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
@ 0x7f598f9a7a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240502 01:54:21.266474 17334 catalog_manager.cc:5484] T ee28f4aa31504dbfac977c4178aefdb1 P bb31bceba76d466db24b9341c04900c3 reported cstate change: config changed from index 3 to 4, VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) evicted. New cstate: current_term: 1 leader_uuid: "f19cc6cb1e76460aa6711e49a5775091" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:21.268776 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ee28f4aa31504dbfac977c4178aefdb1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:21.270658 17737 raft_consensus.cc:2932] T ee28f4aa31504dbfac977c4178aefdb1 P dfa1152987644bd781b46c7e42a7a685 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:21.274185 20477 tablet_service.cc:1514] Processing DeleteTablet for tablet ee28f4aa31504dbfac977c4178aefdb1 with delete_type TABLET_DATA_TOMBSTONED (TS 2f306549407b4bb281ccf2994035e4f1 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35018
@ 0x7fadc825b19c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fadc825b019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f598f98a981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f598f98a70e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fadc823601e kudu::Thread::SuperviseThread() at ??:0
@ 0x7fadcb5bb6db start_thread at ??:0
@ 0x7fadc43f971f clone at ??:0
I20240502 01:54:21.326436 25776 tablet_replica.cc:330] T ee28f4aa31504dbfac977c4178aefdb1 P 2f306549407b4bb281ccf2994035e4f1: stopping tablet replica
I20240502 01:54:21.326663 25776 raft_consensus.cc:2229] T ee28f4aa31504dbfac977c4178aefdb1 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:21.326893 25776 raft_consensus.cc:2258] T ee28f4aa31504dbfac977c4178aefdb1 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:21.329830 25776 ts_tablet_manager.cc:1875] T ee28f4aa31504dbfac977c4178aefdb1 P 2f306549407b4bb281ccf2994035e4f1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:21.335623 25770 tablet_copy_client.cc:308] T f3e4c8f98b5e48e3ba6affc2df08943a P 3fceccefba274b1dbbd8216af8b0bc29: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.72:42411
I20240502 01:54:21.336283 20649 tablet_copy_service.cc:140] P f19cc6cb1e76460aa6711e49a5775091: Received BeginTabletCopySession request for tablet f3e4c8f98b5e48e3ba6affc2df08943a from peer 3fceccefba274b1dbbd8216af8b0bc29 ({username='slave'} at 127.11.42.66:54577)
I20240502 01:54:21.336431 20649 tablet_copy_service.cc:161] P f19cc6cb1e76460aa6711e49a5775091: Beginning new tablet copy session on tablet f3e4c8f98b5e48e3ba6affc2df08943a from peer 3fceccefba274b1dbbd8216af8b0bc29 at {username='slave'} at 127.11.42.66:54577: session id = 3fceccefba274b1dbbd8216af8b0bc29-f3e4c8f98b5e48e3ba6affc2df08943a
I20240502 01:54:21.338408 20649 tablet_copy_source_session.cc:205] T f3e4c8f98b5e48e3ba6affc2df08943a P f19cc6cb1e76460aa6711e49a5775091: Tablet Copy: opened 0 blocks and 1 log segments
@ 0x7f59803e3b75 kudu::ThreadPool::DispatchThread() at ??:0
I20240502 01:54:21.339653 25770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3e4c8f98b5e48e3ba6affc2df08943a. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f59803e719c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
I20240502 01:54:21.357285 20364 raft_consensus.cc:1203] T 6ded7434afab44999620dfa3213c7a82 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:21.363849 25516 raft_consensus.cc:1050] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29: attempting to promote NON_VOTER 4da690c0fa13407a85c0689ce8de9ade to VOTER
I20240502 01:54:21.364687 25516 consensus_queue.cc:237] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } }
@ 0x7f59803e7019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
I20240502 01:54:21.370260 20496 raft_consensus.cc:1261] T a86548104dca43bd9e1b3ec4d23ec0e4 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:21.370920 24918 consensus_queue.cc:1035] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:21.373533 17476 raft_consensus.cc:1261] T a86548104dca43bd9e1b3ec4d23ec0e4 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEARNER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:21.374157 24598 consensus_queue.cc:1035] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:21.374557 25770 tablet_copy_client.cc:783] T f3e4c8f98b5e48e3ba6affc2df08943a P 3fceccefba274b1dbbd8216af8b0bc29: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:21.374787 25770 tablet_copy_client.cc:647] T f3e4c8f98b5e48e3ba6affc2df08943a P 3fceccefba274b1dbbd8216af8b0bc29: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:21.378492 25770 tablet_copy_client.cc:523] T f3e4c8f98b5e48e3ba6affc2df08943a P 3fceccefba274b1dbbd8216af8b0bc29: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:21.381381 25776 ts_tablet_manager.cc:1888] T ee28f4aa31504dbfac977c4178aefdb1 P 2f306549407b4bb281ccf2994035e4f1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:54:21.381654 25776 log.cc:1196] T ee28f4aa31504dbfac977c4178aefdb1 P 2f306549407b4bb281ccf2994035e4f1: Deleting WAL directory at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal/wals/ee28f4aa31504dbfac977c4178aefdb1
I20240502 01:54:21.382459 17322 catalog_manager.cc:4830] TS 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623): tablet ee28f4aa31504dbfac977c4178aefdb1 (table rebalancer_extra_table_36 [id=a95d02d60f11435a8aafb0c271c0aff0]) successfully deleted
I20240502 01:54:21.385704 25770 tablet_bootstrap.cc:492] T f3e4c8f98b5e48e3ba6affc2df08943a P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap starting.
I20240502 01:54:21.387362 17475 raft_consensus.cc:2932] T a86548104dca43bd9e1b3ec4d23ec0e4 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:21.389899 25790 raft_consensus.cc:2932] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
@ 0x7f59803c201e kudu::Thread::SuperviseThread() at ??:0
I20240502 01:54:21.397701 17326 catalog_manager.cc:5484] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
@ 0x7f59837476db start_thread at ??:0
@ 0x7f597c58571f clone at ??:0
I20240502 01:54:21.405405 20496 raft_consensus.cc:2932] T a86548104dca43bd9e1b3ec4d23ec0e4 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:21.410838 17604 consensus_queue.cc:237] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } }
I20240502 01:54:21.413483 25770 tablet_bootstrap.cc:492] T f3e4c8f98b5e48e3ba6affc2df08943a P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:21.413858 25770 tablet_bootstrap.cc:492] T f3e4c8f98b5e48e3ba6affc2df08943a P 3fceccefba274b1dbbd8216af8b0bc29: Bootstrap complete.
I20240502 01:54:21.414085 25770 ts_tablet_manager.cc:1367] T f3e4c8f98b5e48e3ba6affc2df08943a P 3fceccefba274b1dbbd8216af8b0bc29: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.006s
I20240502 01:54:21.414743 25770 raft_consensus.cc:348] T f3e4c8f98b5e48e3ba6affc2df08943a P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } }
I20240502 01:54:21.414971 25770 raft_consensus.cc:729] T f3e4c8f98b5e48e3ba6affc2df08943a P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Initialized, Role: LEARNER
I20240502 01:54:21.415160 25770 consensus_queue.cc:260] T f3e4c8f98b5e48e3ba6affc2df08943a P 3fceccefba274b1dbbd8216af8b0bc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } }
I20240502 01:54:21.415681 25770 ts_tablet_manager.cc:1398] T f3e4c8f98b5e48e3ba6affc2df08943a P 3fceccefba274b1dbbd8216af8b0bc29: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:54:21.417613 17475 raft_consensus.cc:1261] T a86548104dca43bd9e1b3ec4d23ec0e4 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:21.418100 25516 consensus_queue.cc:1035] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:21.428901 20497 raft_consensus.cc:1261] T a86548104dca43bd9e1b3ec4d23ec0e4 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:21.429502 24598 consensus_queue.cc:1035] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:21.431748 24598 raft_consensus.cc:2932] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:21.438140 20497 raft_consensus.cc:2932] T a86548104dca43bd9e1b3ec4d23ec0e4 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:21.439041 17319 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a86548104dca43bd9e1b3ec4d23ec0e4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:21.442215 17473 raft_consensus.cc:2932] T a86548104dca43bd9e1b3ec4d23ec0e4 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:21.442539 17326 catalog_manager.cc:5484] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: config changed from index 3 to 4, VOTER dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) evicted. New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:21.523967 25535 raft_consensus.cc:1050] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 4da690c0fa13407a85c0689ce8de9ade: attempting to promote NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 to VOTER
I20240502 01:54:21.524864 25535 consensus_queue.cc:237] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } }
I20240502 01:54:21.530505 20497 raft_consensus.cc:1261] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:21.531440 25004 consensus_queue.cc:1035] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:21.539278 17604 raft_consensus.cc:1261] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEARNER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:21.539827 25535 consensus_queue.cc:1035] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:21.559624 25535 raft_consensus.cc:2932] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:21.565111 17604 raft_consensus.cc:2932] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:21.569897 20496 raft_consensus.cc:2932] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
W20240502 01:54:21.572474 17543 connection.cc:535] client connection to 127.11.42.69:34339 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:54:21.572721 17543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:54:21.573370 17412 connection.cc:535] client connection to 127.11.42.69:34339 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:54:21.573614 17412 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:54:21.575922 17327 catalog_manager.cc:5484] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: config changed from index 2 to 3, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 01:54:21.581753 20303 connection.cc:535] client connection to 127.11.42.69:34339 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:54:21.581980 20303 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:54:21.582561 17803 connection.cc:535] client connection to 127.11.42.69:34339 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:54:21.582836 17803 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:54:21.583950 20434 connection.cc:535] client connection to 127.11.42.69:34339 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:54:21.584134 20434 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:54:21.595921 25823 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 01:54:21.590668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.42.69:34339 (local address 127.11.42.70:39309)
0502 01:54:21.591052 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0502 01:54:21.591069 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0502 01:54:21.591184 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 01:54:21.595139 (+ 3955us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":298,"threads_started":1}
I20240502 01:54:21.595832 17476 consensus_queue.cc:237] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } }
W20240502 01:54:21.596057 20303 consensus_peers.cc:462] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:54:21.596154 20303 consensus_peers.cc:485] T 52b83e43cae646a29bbf3b826fbbcfb6 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.596267 20303 consensus_peers.cc:485] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.596355 20303 consensus_peers.cc:485] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.596446 20303 consensus_peers.cc:485] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.596521 20303 consensus_peers.cc:485] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.599243 25816 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 01:54:21.582028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.42.69:34339 (local address 127.11.42.66:55449)
0502 01:54:21.589432 (+ 7404us) negotiation.cc:107] Waiting for socket to connect
0502 01:54:21.589456 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0502 01:54:21.589624 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 01:54:21.596842 (+ 7218us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":7275,"thread_start_us":7125,"threads_started":1}
W20240502 01:54:21.599460 17543 consensus_peers.cc:485] T 5152c002eadd4d54ab10a093bce45f17 P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.599557 17543 consensus_peers.cc:485] T 049f5bd8759c488789b7c396985be3f9 P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.599623 17543 consensus_peers.cc:485] T 30f43088bd0f48ee86907df16194b9be P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.600725 25820 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 01:54:21.585346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.42.69:34339 (local address 127.11.42.68:39419)
0502 01:54:21.591297 (+ 5951us) negotiation.cc:107] Waiting for socket to connect
0502 01:54:21.591314 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0502 01:54:21.591428 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 01:54:21.600114 (+ 8686us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":5846,"thread_start_us":7777,"threads_started":1}
W20240502 01:54:21.600874 17803 consensus_peers.cc:485] T 59c288e7279f4119afb6c3bf860e6dc3 P bb31bceba76d466db24b9341c04900c3 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.601002 17803 consensus_peers.cc:485] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.601094 17803 consensus_peers.cc:485] T f0b3862e504d4b43a60477e5efef29b1 P bb31bceba76d466db24b9341c04900c3 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.601753 25822 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 01:54:21.586664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.42.69:34339 (local address 127.11.42.71:49185)
0502 01:54:21.591842 (+ 5178us) negotiation.cc:107] Waiting for socket to connect
0502 01:54:21.591859 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0502 01:54:21.591962 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 01:54:21.601069 (+ 9107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":5085,"thread_start_us":5152,"threads_started":1}
W20240502 01:54:21.603395 25821 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 01:54:21.586015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.42.69:34339 (local address 127.11.42.65:60243)
0502 01:54:21.590813 (+ 4798us) negotiation.cc:107] Waiting for socket to connect
0502 01:54:21.590831 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0502 01:54:21.590945 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 01:54:21.602740 (+ 11795us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":4687,"thread_start_us":4590,"threads_started":1}
W20240502 01:54:21.603607 20434 consensus_peers.cc:485] T cb7d51b2b9f84d608d67e771cd958231 P 2f306549407b4bb281ccf2994035e4f1 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.603717 20434 consensus_peers.cc:485] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.603942 17412 consensus_peers.cc:485] T 5df83f83d2674155afba93c2d992af0a P 4da690c0fa13407a85c0689ce8de9ade -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.604096 17412 consensus_peers.cc:485] T 277c6ae5c8dc4f8eae261bf06890d793 P 4da690c0fa13407a85c0689ce8de9ade -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.695628 17321 connection.cc:535] client connection to 127.11.42.67:36115 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:54:21.622606 20303 consensus_peers.cc:485] T f537e32b31264def980094906bab350c P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.658047 17803 consensus_peers.cc:485] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.696748 17802 connection.cc:535] client connection to 127.11.42.67:36115 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:54:21.765918 20303 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.42.69:34339, user_credentials={real_user=slave}} blocked reactor thread for 143330us
W20240502 01:54:21.695950 17321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:54:21.765990 17803 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.42.69:34339, user_credentials={real_user=slave}} blocked reactor thread for 107953us
W20240502 01:54:21.696316 20302 connection.cc:535] client connection to 127.11.42.67:36115 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:54:21.773319 17802 consensus_peers.cc:485] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.773448 17802 consensus_peers.cc:485] T 65bdab2c9fba48a6828a864278319cc0 P bb31bceba76d466db24b9341c04900c3 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.773533 17802 consensus_peers.cc:485] T f0b3862e504d4b43a60477e5efef29b1 P bb31bceba76d466db24b9341c04900c3 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.773613 17802 consensus_peers.cc:485] T 5124abbb09db4be0955c6bc831af08fa P bb31bceba76d466db24b9341c04900c3 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.775933 20302 consensus_peers.cc:485] T 61ab80ce377a44ac95a9c574e9a615f2 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.776087 20302 consensus_peers.cc:485] T ebb9bca9f1bd41b0857939909b1110e4 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.776204 20302 consensus_peers.cc:485] T 5824db6154734b0f8f9094ae46633771 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.779829 17321 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.11.42.67:36115, user_credentials={real_user=slave}} blocked reactor thread for 83943.4us
W20240502 01:54:21.608832 17412 consensus_peers.cc:485] T aefcd4cee0724eeb866d76ae95bd0080 P 4da690c0fa13407a85c0689ce8de9ade -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.793701 17802 connection.cc:535] client connection to 127.11.42.72:42411 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:54:21.821341 17412 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.42.69:34339, user_credentials={real_user=slave}} blocked reactor thread for 212503us
I20240502 01:54:21.679612 25837 ts_tablet_manager.cc:902] T adc2b57af23c41aebd1258ff8b60d074 P 4da690c0fa13407a85c0689ce8de9ade: Initiating tablet copy from peer 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
W20240502 01:54:21.697208 17411 connection.cc:535] client connection to 127.11.42.67:36115 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:54:21.823702 17411 connection.cc:698] client connection to 127.11.42.72:42411 send error: Network error: sendmsg error: Connection reset by peer (error 104)
W20240502 01:54:21.824265 17411 consensus_peers.cc:485] T b39b88755cd74b0294d264215aea99cf P 4da690c0fa13407a85c0689ce8de9ade -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.824383 17802 consensus_peers.cc:485] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.824481 17802 consensus_peers.cc:485] T 2152c07e448b4582abff22b034363c70 P bb31bceba76d466db24b9341c04900c3 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.824554 17802 consensus_peers.cc:485] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:21.827448 25004 raft_consensus.cc:1050] T 495700f8ed5e41228f2e597465dad126 P 4da690c0fa13407a85c0689ce8de9ade: attempting to promote NON_VOTER bb31bceba76d466db24b9341c04900c3 to VOTER
I20240502 01:54:21.828088 25004 consensus_queue.cc:237] T 495700f8ed5e41228f2e597465dad126 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } }
W20240502 01:54:21.792555 17321 catalog_manager.cc:4573] TS dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a86548104dca43bd9e1b3ec4d23ec0e4: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
W20240502 01:54:21.880697 17321 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.11.42.67:36115, user_credentials={real_user=slave}} blocked reactor thread for 88254.7us
I20240502 01:54:21.605535 17604 raft_consensus.cc:1261] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:54:21.696602 17542 connection.cc:535] client connection to 127.11.42.67:36115 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:54:21.882012 17542 connection.cc:698] client connection to 127.11.42.72:42411 send error: Network error: sendmsg error: Connection reset by peer (error 104)
W20240502 01:54:21.883442 25770 tablet_copy_client.cc:1108] T f3e4c8f98b5e48e3ba6affc2df08943a P 3fceccefba274b1dbbd8216af8b0bc29: tablet copy: Unable to close tablet copy session: Network error: failure ending tablet copy session: sendmsg error: Connection reset by peer (error 104)
W20240502 01:54:21.884892 17542 consensus_peers.cc:462] T 3dcd890bdd2d413e95401897e9f1bcab P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
W20240502 01:54:21.884991 17542 consensus_peers.cc:485] T fa4d38dc9b674a9b90d4793dad2687ec P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.886314 17542 consensus_peers.cc:485] T 43572b8f8fba433ab00b046c769d886d P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.886461 17542 consensus_peers.cc:485] T e61094fdf9324469afa7ecdc45fd55de P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.886627 17542 consensus_peers.cc:485] T 049f5bd8759c488789b7c396985be3f9 P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.886759 17542 consensus_peers.cc:485] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.900377 17542 consensus_peers.cc:485] T 6aacf7c4f19546f581c9a54ad171916b P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.919544 17542 consensus_peers.cc:485] T 1f01949cf70440b38e59a5020fd48c44 P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.828279 17411 consensus_peers.cc:485] T 9642b9b37a784cf881bfc4971d8c9cb6 P 4da690c0fa13407a85c0689ce8de9ade -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.919750 17411 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.42.72:42411, user_credentials={real_user=slave}} blocked reactor thread for 91472.1us
W20240502 01:54:21.919917 17411 consensus_peers.cc:485] T aefcd4cee0724eeb866d76ae95bd0080 P 4da690c0fa13407a85c0689ce8de9ade -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:21.606287 20497 raft_consensus.cc:1261] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:54:21.828739 17412 consensus_peers.cc:485] T 7efd29396c7c4fb597be2251cdcdd48f P 4da690c0fa13407a85c0689ce8de9ade -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.921090 17412 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.42.69:34339, user_credentials={real_user=slave}} blocked reactor thread for 92345.3us
I20240502 01:54:21.922029 17867 raft_consensus.cc:1261] T 495700f8ed5e41228f2e597465dad126 P bb31bceba76d466db24b9341c04900c3 [term 1 LEARNER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:21.838497 25837 tablet_copy_client.cc:308] T adc2b57af23c41aebd1258ff8b60d074 P 4da690c0fa13407a85c0689ce8de9ade: tablet copy: Beginning tablet copy session from remote peer at address 127.11.42.71:38623
W20240502 01:54:21.697480 20433 connection.cc:535] client connection to 127.11.42.67:36115 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:54:21.881889 25751 consensus_queue.cc:1035] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:54:21.922863 20433 connection.cc:698] client connection to 127.11.42.72:42411 send error: Network error: sendmsg error: Connection reset by peer (error 104)
I20240502 01:54:21.922937 25751 consensus_queue.cc:1035] T 495700f8ed5e41228f2e597465dad126 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:21.923599 20518 tablet_copy_service.cc:140] P 2f306549407b4bb281ccf2994035e4f1: Received BeginTabletCopySession request for tablet adc2b57af23c41aebd1258ff8b60d074 from peer 4da690c0fa13407a85c0689ce8de9ade ({username='slave'} at 127.11.42.65:53097)
I20240502 01:54:21.923774 20518 tablet_copy_service.cc:161] P 2f306549407b4bb281ccf2994035e4f1: Beginning new tablet copy session on tablet adc2b57af23c41aebd1258ff8b60d074 from peer 4da690c0fa13407a85c0689ce8de9ade at {username='slave'} at 127.11.42.65:53097: session id = 4da690c0fa13407a85c0689ce8de9ade-adc2b57af23c41aebd1258ff8b60d074
I20240502 01:54:21.926627 20518 tablet_copy_source_session.cc:205] T adc2b57af23c41aebd1258ff8b60d074 P 2f306549407b4bb281ccf2994035e4f1: Tablet Copy: opened 0 blocks and 1 log segments
W20240502 01:54:21.776319 20302 consensus_peers.cc:485] T aebde87a7f114537abd9c1d8b7e97004 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.926937 20302 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.42.67:36115, user_credentials={real_user=slave}} blocked reactor thread for 150615us
W20240502 01:54:21.927364 20302 connection.cc:698] client connection to 127.11.42.72:42411 send error: Network error: sendmsg error: Connection reset by peer (error 104)
W20240502 01:54:21.929085 20302 consensus_peers.cc:485] T 59be89ca2c0b4275844f1e73cd20595b P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.929227 20302 consensus_peers.cc:485] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.929342 20302 consensus_peers.cc:485] T c491a39ca87847d9ace688346c0baf57 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:21.930438 24918 raft_consensus.cc:1050] T e61094fdf9324469afa7ecdc45fd55de P 3fceccefba274b1dbbd8216af8b0bc29: attempting to promote NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 to VOTER
I20240502 01:54:21.931200 24918 consensus_queue.cc:237] T e61094fdf9324469afa7ecdc45fd55de P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } }
I20240502 01:54:21.931950 25535 consensus_queue.cc:1035] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:21.936462 25535 raft_consensus.cc:2932] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:21.937525 25837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adc2b57af23c41aebd1258ff8b60d074. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 01:54:21.937546 17411 consensus_peers.cc:485] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 4da690c0fa13407a85c0689ce8de9ade -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.937808 17411 consensus_peers.cc:485] T 0615786e80314766892988f3cff6a74d P 4da690c0fa13407a85c0689ce8de9ade -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:21.939642 20364 raft_consensus.cc:1261] T 495700f8ed5e41228f2e597465dad126 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:21.941177 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 29fb2b31cd9c4c8fb5923bb849a097d9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:21.941154 17604 raft_consensus.cc:2932] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
W20240502 01:54:21.942274 20433 consensus_peers.cc:485] T 8e21dfeaf732445098f5b17a0615bb8b P 2f306549407b4bb281ccf2994035e4f1 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.942397 20433 consensus_peers.cc:485] T 4bb42883b1a0475b863d008bd01fa70a P 2f306549407b4bb281ccf2994035e4f1 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.942495 20433 consensus_peers.cc:485] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.942572 20433 consensus_peers.cc:485] T 139304da92ab4f48b8a08fa24ed60c95 P 2f306549407b4bb281ccf2994035e4f1 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.942744 20433 consensus_peers.cc:485] T adc2b57af23c41aebd1258ff8b60d074 P 2f306549407b4bb281ccf2994035e4f1 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:21.945211 25577 raft_consensus.cc:1050] T 6ded7434afab44999620dfa3213c7a82 P 2f306549407b4bb281ccf2994035e4f1: attempting to promote NON_VOTER 6a36c952c51542c0bf7ea3ec2857a9f9 to VOTER
I20240502 01:54:21.945966 25577 consensus_queue.cc:237] T 6ded7434afab44999620dfa3213c7a82 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } }
I20240502 01:54:21.947214 17326 catalog_manager.cc:5484] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: config changed from index 3 to 4, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New cstate: current_term: 1 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:21.948017 25751 consensus_queue.cc:1035] T 495700f8ed5e41228f2e597465dad126 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 01:54:21.949473 17411 consensus_peers.cc:485] T 495700f8ed5e41228f2e597465dad126 P 4da690c0fa13407a85c0689ce8de9ade -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:21.949990 17864 raft_consensus.cc:1261] T e61094fdf9324469afa7ecdc45fd55de P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20240502 01:54:21.950795 17411 consensus_peers.cc:485] T 5df83f83d2674155afba93c2d992af0a P 4da690c0fa13407a85c0689ce8de9ade -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.952073 17542 consensus_peers.cc:485] T e61094fdf9324469afa7ecdc45fd55de P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:21.954375 24598 consensus_queue.cc:1035] T e61094fdf9324469afa7ecdc45fd55de P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:21.956475 20497 raft_consensus.cc:2932] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:21.962628 20367 raft_consensus.cc:1261] T e61094fdf9324469afa7ecdc45fd55de P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:21.964242 25837 tablet_copy_client.cc:783] T adc2b57af23c41aebd1258ff8b60d074 P 4da690c0fa13407a85c0689ce8de9ade: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:21.964443 25837 tablet_copy_client.cc:647] T adc2b57af23c41aebd1258ff8b60d074 P 4da690c0fa13407a85c0689ce8de9ade: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:21.964356 25516 consensus_queue.cc:1035] T e61094fdf9324469afa7ecdc45fd55de P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 01:54:21.966249 20433 consensus_peers.cc:485] T 246cec994d274786aad7672f2a30e5d8 P 2f306549407b4bb281ccf2994035e4f1 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.966398 20433 consensus_peers.cc:485] T acbd240a2783424e96bd589f0232daf6 P 2f306549407b4bb281ccf2994035e4f1 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:21.966499 20433 consensus_peers.cc:485] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:21.967953 25837 tablet_copy_client.cc:523] T adc2b57af23c41aebd1258ff8b60d074 P 4da690c0fa13407a85c0689ce8de9ade: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:21.970829 17604 raft_consensus.cc:1261] T 6ded7434afab44999620dfa3213c7a82 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:21.971621 25813 raft_consensus.cc:2932] T 495700f8ed5e41228f2e597465dad126 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:21.972052 20364 raft_consensus.cc:1261] T 6ded7434afab44999620dfa3213c7a82 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 LEARNER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:21.972450 25917 consensus_queue.cc:1035] T 6ded7434afab44999620dfa3213c7a82 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:21.974527 25917 consensus_queue.cc:1035] T 6ded7434afab44999620dfa3213c7a82 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:21.982580 17330 catalog_manager.cc:5484] T 495700f8ed5e41228f2e597465dad126 P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: config changed from index 2 to 3, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:21.983407 20364 raft_consensus.cc:2932] T 495700f8ed5e41228f2e597465dad126 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:21.984051 25837 tablet_bootstrap.cc:492] T adc2b57af23c41aebd1258ff8b60d074 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap starting.
W20240502 01:54:21.987020 20303 consensus_peers.cc:462] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
I20240502 01:54:21.988903 17473 consensus_queue.cc:237] T 495700f8ed5e41228f2e597465dad126 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } }
I20240502 01:54:21.991039 20364 raft_consensus.cc:1261] T 495700f8ed5e41228f2e597465dad126 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:21.991508 25535 consensus_queue.cc:1035] T 495700f8ed5e41228f2e597465dad126 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:21.981123 17867 raft_consensus.cc:2932] T 495700f8ed5e41228f2e597465dad126 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bb31bceba76d466db24b9341c04900c3 (127.11.42.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:21.995129 25535 raft_consensus.cc:2932] T 495700f8ed5e41228f2e597465dad126 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:21.998217 20362 raft_consensus.cc:2932] T 495700f8ed5e41228f2e597465dad126 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
W20240502 01:54:22.001274 20433 consensus_peers.cc:485] T 6ded7434afab44999620dfa3213c7a82 P 2f306549407b4bb281ccf2994035e4f1 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:22.003618 17321 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 495700f8ed5e41228f2e597465dad126 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:22.008611 17326 catalog_manager.cc:5484] T 495700f8ed5e41228f2e597465dad126 P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: config changed from index 3 to 4, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New cstate: current_term: 1 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:22.013108 17864 raft_consensus.cc:1261] T 495700f8ed5e41228f2e597465dad126 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:22.015286 25813 consensus_queue.cc:1035] T 495700f8ed5e41228f2e597465dad126 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:22.016995 17864 raft_consensus.cc:2932] T 495700f8ed5e41228f2e597465dad126 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } } }
I20240502 01:54:22.026021 25516 raft_consensus.cc:2932] T e61094fdf9324469afa7ecdc45fd55de P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:54:22.031342 20362 raft_consensus.cc:2932] T e61094fdf9324469afa7ecdc45fd55de P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:54:22.039495 17864 raft_consensus.cc:2932] T e61094fdf9324469afa7ecdc45fd55de P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:54:22.039614 17334 catalog_manager.cc:5484] T e61094fdf9324469afa7ecdc45fd55de P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: config changed from index 2 to 3, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:22.064751 25917 raft_consensus.cc:2932] T 6ded7434afab44999620dfa3213c7a82 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:54:22.065330 20361 raft_consensus.cc:2932] T 6ded7434afab44999620dfa3213c7a82 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:54:22.069927 17604 raft_consensus.cc:2932] T 6ded7434afab44999620dfa3213c7a82 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } } }
I20240502 01:54:22.071311 25837 tablet_bootstrap.cc:492] T adc2b57af23c41aebd1258ff8b60d074 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:22.071691 25837 tablet_bootstrap.cc:492] T adc2b57af23c41aebd1258ff8b60d074 P 4da690c0fa13407a85c0689ce8de9ade: Bootstrap complete.
I20240502 01:54:22.071952 25837 ts_tablet_manager.cc:1367] T adc2b57af23c41aebd1258ff8b60d074 P 4da690c0fa13407a85c0689ce8de9ade: Time spent bootstrapping tablet: real 0.088s user 0.021s sys 0.032s
I20240502 01:54:22.072844 25837 raft_consensus.cc:348] T adc2b57af23c41aebd1258ff8b60d074 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } }
I20240502 01:54:22.073104 25837 raft_consensus.cc:729] T adc2b57af23c41aebd1258ff8b60d074 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Initialized, Role: LEARNER
I20240502 01:54:22.073313 25837 consensus_queue.cc:260] T adc2b57af23c41aebd1258ff8b60d074 P 4da690c0fa13407a85c0689ce8de9ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } }
I20240502 01:54:22.075665 25837 ts_tablet_manager.cc:1398] T adc2b57af23c41aebd1258ff8b60d074 P 4da690c0fa13407a85c0689ce8de9ade: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:54:22.076311 17334 catalog_manager.cc:5484] T 6ded7434afab44999620dfa3213c7a82 P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: config changed from index 2 to 3, 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:22.080180 20518 tablet_copy_service.cc:342] P 2f306549407b4bb281ccf2994035e4f1: Request end of tablet copy session 4da690c0fa13407a85c0689ce8de9ade-adc2b57af23c41aebd1258ff8b60d074 received from {username='slave'} at 127.11.42.65:53097
I20240502 01:54:22.080313 20518 tablet_copy_service.cc:433] P 2f306549407b4bb281ccf2994035e4f1: ending tablet copy session 4da690c0fa13407a85c0689ce8de9ade-adc2b57af23c41aebd1258ff8b60d074 on tablet adc2b57af23c41aebd1258ff8b60d074 with peer 4da690c0fa13407a85c0689ce8de9ade
I20240502 01:54:22.118324 17473 raft_consensus.cc:1203] T adc2b57af23c41aebd1258ff8b60d074 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 01:54:22.130499 25603 raft_consensus.cc:482] T b6d89235867f42418f8205167bb29755 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdc2ac372492454c85387589a91f63af)
I20240502 01:54:22.130699 25603 raft_consensus.cc:504] T b6d89235867f42418f8205167bb29755 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:22.131310 25603 leader_election.cc:288] T b6d89235867f42418f8205167bb29755 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:54:22.132020 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d89235867f42418f8205167bb29755" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:54:22.137060 25516 raft_consensus.cc:482] T bbf05e3b3ad44354be7600c3c07756fd P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdc2ac372492454c85387589a91f63af)
I20240502 01:54:22.137249 25516 raft_consensus.cc:504] T bbf05e3b3ad44354be7600c3c07756fd P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
I20240502 01:54:22.137872 25516 leader_election.cc:288] T bbf05e3b3ad44354be7600c3c07756fd P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
W20240502 01:54:22.138303 20303 leader_election.cc:334] T b6d89235867f42418f8205167bb29755 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
I20240502 01:54:22.138442 20303 leader_election.cc:302] T b6d89235867f42418f8205167bb29755 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9; no voters: 3fceccefba274b1dbbd8216af8b0bc29, fdc2ac372492454c85387589a91f63af
I20240502 01:54:22.138720 25603 raft_consensus.cc:2726] T b6d89235867f42418f8205167bb29755 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:22.139317 20361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbf05e3b3ad44354be7600c3c07756fd" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
W20240502 01:54:22.146512 17543 leader_election.cc:334] T bbf05e3b3ad44354be7600c3c07756fd P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
I20240502 01:54:22.146634 17543 leader_election.cc:302] T bbf05e3b3ad44354be7600c3c07756fd P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29; no voters: 6a36c952c51542c0bf7ea3ec2857a9f9, fdc2ac372492454c85387589a91f63af
I20240502 01:54:22.146912 25516 raft_consensus.cc:2726] T bbf05e3b3ad44354be7600c3c07756fd P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:22.218221 25790 raft_consensus.cc:482] T 164e04f7c9454c189f6907b319023d99 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdc2ac372492454c85387589a91f63af)
I20240502 01:54:22.218386 25790 raft_consensus.cc:504] T 164e04f7c9454c189f6907b319023d99 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:22.219034 25790 leader_election.cc:288] T 164e04f7c9454c189f6907b319023d99 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:54:22.219352 17864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "164e04f7c9454c189f6907b319023d99" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
W20240502 01:54:22.221164 17543 leader_election.cc:334] T 164e04f7c9454c189f6907b319023d99 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
I20240502 01:54:22.221304 17543 leader_election.cc:302] T 164e04f7c9454c189f6907b319023d99 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29; no voters: bb31bceba76d466db24b9341c04900c3, fdc2ac372492454c85387589a91f63af
I20240502 01:54:22.221544 25790 raft_consensus.cc:2726] T 164e04f7c9454c189f6907b319023d99 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:22.267608 25848 raft_consensus.cc:482] T b950e18825ac434ba44586d544667918 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dfa1152987644bd781b46c7e42a7a685)
I20240502 01:54:22.267778 25848 raft_consensus.cc:504] T b950e18825ac434ba44586d544667918 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:22.268637 25848 leader_election.cc:288] T b950e18825ac434ba44586d544667918 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:54:22.270948 20494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b950e18825ac434ba44586d544667918" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
W20240502 01:54:22.279219 20302 leader_election.cc:334] T b950e18825ac434ba44586d544667918 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
I20240502 01:54:22.279369 20302 leader_election.cc:302] T b950e18825ac434ba44586d544667918 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9; no voters: 2f306549407b4bb281ccf2994035e4f1, dfa1152987644bd781b46c7e42a7a685
I20240502 01:54:22.279901 24817 raft_consensus.cc:2726] T b950e18825ac434ba44586d544667918 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:22.293154 25604 raft_consensus.cc:482] T 62887e332443457f8a5b41a982354dc4 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdc2ac372492454c85387589a91f63af)
I20240502 01:54:22.293309 25604 raft_consensus.cc:504] T 62887e332443457f8a5b41a982354dc4 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } }
I20240502 01:54:22.293902 25604 leader_election.cc:288] T 62887e332443457f8a5b41a982354dc4 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
W20240502 01:54:22.295337 17803 leader_election.cc:334] T 62887e332443457f8a5b41a982354dc4 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
W20240502 01:54:22.295611 17802 leader_election.cc:334] T 62887e332443457f8a5b41a982354dc4 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
I20240502 01:54:22.295714 17802 leader_election.cc:302] T 62887e332443457f8a5b41a982354dc4 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb31bceba76d466db24b9341c04900c3; no voters: f19cc6cb1e76460aa6711e49a5775091, fdc2ac372492454c85387589a91f63af
I20240502 01:54:22.296128 25604 raft_consensus.cc:2726] T 62887e332443457f8a5b41a982354dc4 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:22.299105 25004 raft_consensus.cc:482] T 25ad701a0fec4e80b37df5dec598796a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdc2ac372492454c85387589a91f63af)
I20240502 01:54:22.299286 25004 raft_consensus.cc:504] T 25ad701a0fec4e80b37df5dec598796a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:54:22.300025 25004 leader_election.cc:288] T 25ad701a0fec4e80b37df5dec598796a P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:54:22.300385 20494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25ad701a0fec4e80b37df5dec598796a" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
W20240502 01:54:22.301820 17412 leader_election.cc:334] T 25ad701a0fec4e80b37df5dec598796a P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
I20240502 01:54:22.301939 17412 leader_election.cc:302] T 25ad701a0fec4e80b37df5dec598796a P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade; no voters: 2f306549407b4bb281ccf2994035e4f1, fdc2ac372492454c85387589a91f63af
I20240502 01:54:22.302186 25004 raft_consensus.cc:2726] T 25ad701a0fec4e80b37df5dec598796a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:22.302791 25004 raft_consensus.cc:482] T b29218a879e74895bfe00021ad358aa4 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dfa1152987644bd781b46c7e42a7a685)
I20240502 01:54:22.302888 25004 raft_consensus.cc:504] T b29218a879e74895bfe00021ad358aa4 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } }
I20240502 01:54:22.303364 25004 leader_election.cc:288] T b29218a879e74895bfe00021ad358aa4 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:54:22.303900 20494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b29218a879e74895bfe00021ad358aa4" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
W20240502 01:54:22.304530 17411 leader_election.cc:334] T b29218a879e74895bfe00021ad358aa4 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
I20240502 01:54:22.304622 17411 leader_election.cc:302] T b29218a879e74895bfe00021ad358aa4 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade; no voters: 2f306549407b4bb281ccf2994035e4f1, dfa1152987644bd781b46c7e42a7a685
I20240502 01:54:22.305188 25004 raft_consensus.cc:2726] T b29218a879e74895bfe00021ad358aa4 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:22.305593 25004 raft_consensus.cc:482] T bffb870715a243d992107583d7c17f1b P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdc2ac372492454c85387589a91f63af)
I20240502 01:54:22.305730 25004 raft_consensus.cc:504] T bffb870715a243d992107583d7c17f1b P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } }
I20240502 01:54:22.306250 25004 leader_election.cc:288] T bffb870715a243d992107583d7c17f1b P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:54:22.306622 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bffb870715a243d992107583d7c17f1b" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
W20240502 01:54:22.307467 17412 leader_election.cc:334] T bffb870715a243d992107583d7c17f1b P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
I20240502 01:54:22.307596 17412 leader_election.cc:302] T bffb870715a243d992107583d7c17f1b P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade; no voters: 3fceccefba274b1dbbd8216af8b0bc29, fdc2ac372492454c85387589a91f63af
I20240502 01:54:22.307816 25004 raft_consensus.cc:2726] T bffb870715a243d992107583d7c17f1b P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:22.354916 25604 raft_consensus.cc:482] T 75bb4e19cf57405fabcb532b55a29ced P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f19cc6cb1e76460aa6711e49a5775091)
I20240502 01:54:22.355077 25604 raft_consensus.cc:504] T 75bb4e19cf57405fabcb532b55a29ced P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } }
I20240502 01:54:22.355672 25604 leader_election.cc:288] T 75bb4e19cf57405fabcb532b55a29ced P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:54:22.356668 17473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75bb4e19cf57405fabcb532b55a29ced" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
W20240502 01:54:22.357889 17802 leader_election.cc:334] T 75bb4e19cf57405fabcb532b55a29ced P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
I20240502 01:54:22.357990 17802 leader_election.cc:302] T 75bb4e19cf57405fabcb532b55a29ced P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb31bceba76d466db24b9341c04900c3; no voters: 4da690c0fa13407a85c0689ce8de9ade, f19cc6cb1e76460aa6711e49a5775091
I20240502 01:54:22.358387 25604 raft_consensus.cc:2726] T 75bb4e19cf57405fabcb532b55a29ced P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:22.383657 24598 raft_consensus.cc:482] T 57ab576948e346cbbe6715bf53f7c665 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dfa1152987644bd781b46c7e42a7a685)
I20240502 01:54:22.383831 24598 raft_consensus.cc:504] T 57ab576948e346cbbe6715bf53f7c665 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:22.384491 24598 leader_election.cc:288] T 57ab576948e346cbbe6715bf53f7c665 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:54:22.386619 25917 raft_consensus.cc:482] T 29d8cb76dc514e7b9002bd2291d8ee8d P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dfa1152987644bd781b46c7e42a7a685)
I20240502 01:54:22.386831 25917 raft_consensus.cc:504] T 29d8cb76dc514e7b9002bd2291d8ee8d P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } }
I20240502 01:54:22.387492 25917 leader_election.cc:288] T 29d8cb76dc514e7b9002bd2291d8ee8d P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
W20240502 01:54:22.390586 20433 leader_election.cc:334] T 29d8cb76dc514e7b9002bd2291d8ee8d P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
W20240502 01:54:22.390818 20434 leader_election.cc:334] T 29d8cb76dc514e7b9002bd2291d8ee8d P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
I20240502 01:54:22.390926 20434 leader_election.cc:302] T 29d8cb76dc514e7b9002bd2291d8ee8d P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1; no voters: dfa1152987644bd781b46c7e42a7a685, fdc2ac372492454c85387589a91f63af
I20240502 01:54:22.391535 25920 raft_consensus.cc:2726] T 29d8cb76dc514e7b9002bd2291d8ee8d P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:22.391865 25004 raft_consensus.cc:482] T e070da07e2214744b3dacd1b839310b9 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f19cc6cb1e76460aa6711e49a5775091)
I20240502 01:54:22.392015 25004 raft_consensus.cc:504] T e070da07e2214744b3dacd1b839310b9 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:22.392637 25004 leader_election.cc:288] T e070da07e2214744b3dacd1b839310b9 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:54:22.395104 25516 raft_consensus.cc:482] T bffb870715a243d992107583d7c17f1b P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdc2ac372492454c85387589a91f63af)
I20240502 01:54:22.395234 25516 raft_consensus.cc:504] T bffb870715a243d992107583d7c17f1b P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } }
I20240502 01:54:22.395838 25516 leader_election.cc:288] T bffb870715a243d992107583d7c17f1b P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
W20240502 01:54:22.396052 17412 leader_election.cc:334] T e070da07e2214744b3dacd1b839310b9 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
W20240502 01:54:22.397717 17542 leader_election.cc:334] T 57ab576948e346cbbe6715bf53f7c665 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
I20240502 01:54:22.398824 17476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bffb870715a243d992107583d7c17f1b" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:54:22.399035 17476 raft_consensus.cc:2454] T bffb870715a243d992107583d7c17f1b P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:54:22.402392 17542 leader_election.cc:302] T bffb870715a243d992107583d7c17f1b P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, 4da690c0fa13407a85c0689ce8de9ade; no voters:
I20240502 01:54:22.402921 25790 raft_consensus.cc:2781] T bffb870715a243d992107583d7c17f1b P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:54:22.403023 25790 raft_consensus.cc:482] T bffb870715a243d992107583d7c17f1b P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting leader election (detected failure of leader fdc2ac372492454c85387589a91f63af)
I20240502 01:54:22.403126 25790 raft_consensus.cc:3037] T bffb870715a243d992107583d7c17f1b P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:54:22.407497 17411 leader_election.cc:334] T e070da07e2214744b3dacd1b839310b9 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
I20240502 01:54:22.407644 17411 leader_election.cc:302] T e070da07e2214744b3dacd1b839310b9 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade; no voters: f19cc6cb1e76460aa6711e49a5775091, fdc2ac372492454c85387589a91f63af
I20240502 01:54:22.408012 25751 raft_consensus.cc:2726] T e070da07e2214744b3dacd1b839310b9 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:54:22.408515 17543 leader_election.cc:334] T bffb870715a243d992107583d7c17f1b P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
I20240502 01:54:22.415652 25790 raft_consensus.cc:504] T bffb870715a243d992107583d7c17f1b P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } }
I20240502 01:54:22.416424 25790 leader_election.cc:288] T bffb870715a243d992107583d7c17f1b P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 election: Requested vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:54:22.418262 17476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bffb870715a243d992107583d7c17f1b" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:54:22.418479 17476 raft_consensus.cc:3037] T bffb870715a243d992107583d7c17f1b P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:22.421658 25917 raft_consensus.cc:482] T 88e68acb7de346f0ad165cadabd077be P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdc2ac372492454c85387589a91f63af)
I20240502 01:54:22.421823 25917 raft_consensus.cc:504] T 88e68acb7de346f0ad165cadabd077be P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
W20240502 01:54:22.422019 17542 consensus_peers.cc:462] T 3dcd890bdd2d413e95401897e9f1bcab P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
W20240502 01:54:22.422178 17543 leader_election.cc:334] T bffb870715a243d992107583d7c17f1b P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
I20240502 01:54:22.422461 25917 leader_election.cc:288] T 88e68acb7de346f0ad165cadabd077be P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:54:22.423628 17864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ab576948e346cbbe6715bf53f7c665" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
W20240502 01:54:22.424280 20433 leader_election.cc:334] T 88e68acb7de346f0ad165cadabd077be P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
I20240502 01:54:22.421476 17476 raft_consensus.cc:2454] T bffb870715a243d992107583d7c17f1b P 4da690c0fa13407a85c0689ce8de9ade [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 2.
I20240502 01:54:22.426940 17542 leader_election.cc:302] T bffb870715a243d992107583d7c17f1b P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, 4da690c0fa13407a85c0689ce8de9ade; no voters: fdc2ac372492454c85387589a91f63af
I20240502 01:54:22.427121 25917 raft_consensus.cc:482] T 5e3695e8ad0a44de9ba23ae3610aacec P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdc2ac372492454c85387589a91f63af)
I20240502 01:54:22.427307 25917 raft_consensus.cc:504] T 5e3695e8ad0a44de9ba23ae3610aacec P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:22.427528 25790 raft_consensus.cc:2781] T bffb870715a243d992107583d7c17f1b P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:22.427661 25790 raft_consensus.cc:686] T bffb870715a243d992107583d7c17f1b P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 LEADER]: Becoming Leader. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Running, Role: LEADER
I20240502 01:54:22.427927 25917 leader_election.cc:288] T 5e3695e8ad0a44de9ba23ae3610aacec P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:54:22.427934 25790 consensus_queue.cc:237] T bffb870715a243d992107583d7c17f1b P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } }
I20240502 01:54:22.429116 17473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e3695e8ad0a44de9ba23ae3610aacec" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:54:22.429674 24598 raft_consensus.cc:482] T b6d89235867f42418f8205167bb29755 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdc2ac372492454c85387589a91f63af)
I20240502 01:54:22.429836 24598 raft_consensus.cc:504] T b6d89235867f42418f8205167bb29755 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:22.430575 24598 leader_election.cc:288] T b6d89235867f42418f8205167bb29755 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:54:22.430792 24598 raft_consensus.cc:482] T 68bc23294ac64c9486bffebbdb16b508 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f19cc6cb1e76460aa6711e49a5775091)
I20240502 01:54:22.430935 24598 raft_consensus.cc:504] T 68bc23294ac64c9486bffebbdb16b508 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } }
I20240502 01:54:22.431099 20361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d89235867f42418f8205167bb29755" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:54:22.431329 20361 raft_consensus.cc:2454] T b6d89235867f42418f8205167bb29755 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:54:22.431524 24598 leader_election.cc:288] T 68bc23294ac64c9486bffebbdb16b508 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:54:22.432049 25813 raft_consensus.cc:482] T 4099f4b2ae164020839faab53957750f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dfa1152987644bd781b46c7e42a7a685)
I20240502 01:54:22.432050 17542 leader_election.cc:302] T b6d89235867f42418f8205167bb29755 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, 6a36c952c51542c0bf7ea3ec2857a9f9; no voters:
I20240502 01:54:22.432209 25813 raft_consensus.cc:504] T 4099f4b2ae164020839faab53957750f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } }
I20240502 01:54:22.432324 24598 raft_consensus.cc:2781] T b6d89235867f42418f8205167bb29755 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:54:22.432412 24598 raft_consensus.cc:482] T b6d89235867f42418f8205167bb29755 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting leader election (detected failure of leader fdc2ac372492454c85387589a91f63af)
I20240502 01:54:22.432493 24598 raft_consensus.cc:3037] T b6d89235867f42418f8205167bb29755 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:22.432818 25813 leader_election.cc:288] T 4099f4b2ae164020839faab53957750f P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:54:22.433933 25603 raft_consensus.cc:482] T a390a222562f425eabb11d7f52cbe9a6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dfa1152987644bd781b46c7e42a7a685)
I20240502 01:54:22.434119 25603 raft_consensus.cc:504] T a390a222562f425eabb11d7f52cbe9a6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:54:22.434798 25603 leader_election.cc:288] T a390a222562f425eabb11d7f52cbe9a6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:54:22.435190 20494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68bc23294ac64c9486bffebbdb16b508" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
I20240502 01:54:22.435200 24598 raft_consensus.cc:504] T b6d89235867f42418f8205167bb29755 P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:22.435866 24598 leader_election.cc:288] T b6d89235867f42418f8205167bb29755 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 election: Requested vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:54:22.437491 25813 raft_consensus.cc:482] T 062b9d90fb0444f4be37bf09770c9caf P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdc2ac372492454c85387589a91f63af)
I20240502 01:54:22.437651 25813 raft_consensus.cc:504] T 062b9d90fb0444f4be37bf09770c9caf P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } }
I20240502 01:54:22.438235 25813 leader_election.cc:288] T 062b9d90fb0444f4be37bf09770c9caf P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:54:22.438941 17473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a390a222562f425eabb11d7f52cbe9a6" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
W20240502 01:54:22.439143 20302 leader_election.cc:334] T a390a222562f425eabb11d7f52cbe9a6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
I20240502 01:54:22.439527 20302 leader_election.cc:302] T a390a222562f425eabb11d7f52cbe9a6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9; no voters: 4da690c0fa13407a85c0689ce8de9ade, dfa1152987644bd781b46c7e42a7a685
I20240502 01:54:22.439965 25603 raft_consensus.cc:2726] T a390a222562f425eabb11d7f52cbe9a6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:22.441337 17864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "062b9d90fb0444f4be37bf09770c9caf" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
W20240502 01:54:22.447443 20434 leader_election.cc:334] T 88e68acb7de346f0ad165cadabd077be P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
W20240502 01:54:22.447513 17411 leader_election.cc:334] T 4099f4b2ae164020839faab53957750f P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
I20240502 01:54:22.447610 20434 leader_election.cc:302] T 88e68acb7de346f0ad165cadabd077be P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1; no voters: dfa1152987644bd781b46c7e42a7a685, fdc2ac372492454c85387589a91f63af
W20240502 01:54:22.447705 17411 leader_election.cc:334] T 4099f4b2ae164020839faab53957750f P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
I20240502 01:54:22.447837 17411 leader_election.cc:302] T 4099f4b2ae164020839faab53957750f P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade; no voters: dfa1152987644bd781b46c7e42a7a685, f19cc6cb1e76460aa6711e49a5775091
I20240502 01:54:22.447882 25920 raft_consensus.cc:2726] T 88e68acb7de346f0ad165cadabd077be P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:22.448117 25813 raft_consensus.cc:2726] T 4099f4b2ae164020839faab53957750f P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:22.448865 25813 raft_consensus.cc:482] T a390a222562f425eabb11d7f52cbe9a6 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dfa1152987644bd781b46c7e42a7a685)
I20240502 01:54:22.449035 25813 raft_consensus.cc:504] T a390a222562f425eabb11d7f52cbe9a6 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
W20240502 01:54:22.449349 20434 leader_election.cc:334] T 5e3695e8ad0a44de9ba23ae3610aacec P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
I20240502 01:54:22.449458 20434 leader_election.cc:302] T 5e3695e8ad0a44de9ba23ae3610aacec P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1; no voters: 4da690c0fa13407a85c0689ce8de9ade, fdc2ac372492454c85387589a91f63af
I20240502 01:54:22.449647 25813 leader_election.cc:288] T a390a222562f425eabb11d7f52cbe9a6 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:54:22.449689 25920 raft_consensus.cc:2726] T 5e3695e8ad0a44de9ba23ae3610aacec P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:22.450430 20361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a390a222562f425eabb11d7f52cbe9a6" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:54:22.450678 20361 raft_consensus.cc:2454] T a390a222562f425eabb11d7f52cbe9a6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
W20240502 01:54:22.450737 17412 leader_election.cc:334] T 062b9d90fb0444f4be37bf09770c9caf P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
W20240502 01:54:22.451015 17411 leader_election.cc:334] T a390a222562f425eabb11d7f52cbe9a6 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
I20240502 01:54:22.450886 17412 leader_election.cc:302] T 062b9d90fb0444f4be37bf09770c9caf P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade; no voters: bb31bceba76d466db24b9341c04900c3, fdc2ac372492454c85387589a91f63af
I20240502 01:54:22.451993 25813 raft_consensus.cc:2726] T 062b9d90fb0444f4be37bf09770c9caf P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:22.452831 17543 leader_election.cc:302] T 57ab576948e346cbbe6715bf53f7c665 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29; no voters: bb31bceba76d466db24b9341c04900c3, dfa1152987644bd781b46c7e42a7a685
I20240502 01:54:22.453120 24598 raft_consensus.cc:2726] T 57ab576948e346cbbe6715bf53f7c665 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:22.453171 25920 raft_consensus.cc:482] T 25ad701a0fec4e80b37df5dec598796a P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdc2ac372492454c85387589a91f63af)
I20240502 01:54:22.453320 25920 raft_consensus.cc:504] T 25ad701a0fec4e80b37df5dec598796a P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:54:22.454013 25920 leader_election.cc:288] T 25ad701a0fec4e80b37df5dec598796a P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:54:22.454303 25604 raft_consensus.cc:482] T cf0237a4c65b495ba892871dd850bed9 P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dfa1152987644bd781b46c7e42a7a685)
I20240502 01:54:22.454509 20361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d89235867f42418f8205167bb29755" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
I20240502 01:54:22.454442 25604 raft_consensus.cc:504] T cf0237a4c65b495ba892871dd850bed9 P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } }
I20240502 01:54:22.454674 20361 raft_consensus.cc:3037] T b6d89235867f42418f8205167bb29755 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:22.454749 17473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25ad701a0fec4e80b37df5dec598796a" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:54:22.454939 17473 raft_consensus.cc:2454] T 25ad701a0fec4e80b37df5dec598796a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 1.
I20240502 01:54:22.455111 17411 leader_election.cc:302] T a390a222562f425eabb11d7f52cbe9a6 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, 6a36c952c51542c0bf7ea3ec2857a9f9; no voters: dfa1152987644bd781b46c7e42a7a685
I20240502 01:54:22.455183 25604 leader_election.cc:288] T cf0237a4c65b495ba892871dd850bed9 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:54:22.455340 20433 leader_election.cc:302] T 25ad701a0fec4e80b37df5dec598796a P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 4da690c0fa13407a85c0689ce8de9ade; no voters:
I20240502 01:54:22.455595 25920 raft_consensus.cc:2781] T 25ad701a0fec4e80b37df5dec598796a P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:54:22.455668 25920 raft_consensus.cc:482] T 25ad701a0fec4e80b37df5dec598796a P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader fdc2ac372492454c85387589a91f63af)
I20240502 01:54:22.455755 25920 raft_consensus.cc:3037] T 25ad701a0fec4e80b37df5dec598796a P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:22.457441 25813 raft_consensus.cc:2781] T a390a222562f425eabb11d7f52cbe9a6 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:54:22.457568 25813 raft_consensus.cc:482] T a390a222562f425eabb11d7f52cbe9a6 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting leader election (detected failure of leader dfa1152987644bd781b46c7e42a7a685)
I20240502 01:54:22.457653 25813 raft_consensus.cc:3037] T a390a222562f425eabb11d7f52cbe9a6 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:22.458297 20494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf0237a4c65b495ba892871dd850bed9" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
I20240502 01:54:22.458675 25920 raft_consensus.cc:504] T 25ad701a0fec4e80b37df5dec598796a P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:54:22.459328 25920 leader_election.cc:288] T 25ad701a0fec4e80b37df5dec598796a P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 election: Requested vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:54:22.459781 17473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25ad701a0fec4e80b37df5dec598796a" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:54:22.459968 17473 raft_consensus.cc:3037] T 25ad701a0fec4e80b37df5dec598796a P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:54:22.461088 20434 leader_election.cc:334] T 25ad701a0fec4e80b37df5dec598796a P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
W20240502 01:54:22.461396 20434 leader_election.cc:334] T 25ad701a0fec4e80b37df5dec598796a P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
I20240502 01:54:22.462424 17473 raft_consensus.cc:2454] T 25ad701a0fec4e80b37df5dec598796a P 4da690c0fa13407a85c0689ce8de9ade [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 2.
I20240502 01:54:22.462792 20433 leader_election.cc:302] T 25ad701a0fec4e80b37df5dec598796a P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 4da690c0fa13407a85c0689ce8de9ade; no voters: fdc2ac372492454c85387589a91f63af
I20240502 01:54:22.463129 25920 raft_consensus.cc:2781] T 25ad701a0fec4e80b37df5dec598796a P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:22.463301 25813 raft_consensus.cc:504] T a390a222562f425eabb11d7f52cbe9a6 P 4da690c0fa13407a85c0689ce8de9ade [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:54:22.463420 25920 raft_consensus.cc:686] T 25ad701a0fec4e80b37df5dec598796a P 2f306549407b4bb281ccf2994035e4f1 [term 2 LEADER]: Becoming Leader. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Running, Role: LEADER
I20240502 01:54:22.463742 25920 consensus_queue.cc:237] T 25ad701a0fec4e80b37df5dec598796a P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:54:22.463954 25813 leader_election.cc:288] T a390a222562f425eabb11d7f52cbe9a6 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 election: Requested vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:54:22.464358 20361 raft_consensus.cc:2454] T b6d89235867f42418f8205167bb29755 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 2.
I20240502 01:54:22.465072 20361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a390a222562f425eabb11d7f52cbe9a6" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
I20240502 01:54:22.465288 20361 raft_consensus.cc:3037] T a390a222562f425eabb11d7f52cbe9a6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:22.465432 25604 raft_consensus.cc:482] T 8c129f9289024ec9898fbbc06b2fb2cd P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdc2ac372492454c85387589a91f63af)
W20240502 01:54:22.465543 17411 leader_election.cc:334] T a390a222562f425eabb11d7f52cbe9a6 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
I20240502 01:54:22.465577 25604 raft_consensus.cc:504] T 8c129f9289024ec9898fbbc06b2fb2cd P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:22.466130 25604 leader_election.cc:288] T 8c129f9289024ec9898fbbc06b2fb2cd P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
W20240502 01:54:22.468199 17803 leader_election.cc:334] T 8c129f9289024ec9898fbbc06b2fb2cd P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
W20240502 01:54:22.469048 17543 leader_election.cc:334] T b6d89235867f42418f8205167bb29755 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
I20240502 01:54:22.469403 20361 raft_consensus.cc:2454] T a390a222562f425eabb11d7f52cbe9a6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 2.
I20240502 01:54:22.469848 17411 leader_election.cc:302] T a390a222562f425eabb11d7f52cbe9a6 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, 6a36c952c51542c0bf7ea3ec2857a9f9; no voters: dfa1152987644bd781b46c7e42a7a685
W20240502 01:54:22.471362 17543 leader_election.cc:334] T b6d89235867f42418f8205167bb29755 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
W20240502 01:54:22.471635 17802 leader_election.cc:334] T cf0237a4c65b495ba892871dd850bed9 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
I20240502 01:54:22.471772 17802 leader_election.cc:302] T cf0237a4c65b495ba892871dd850bed9 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb31bceba76d466db24b9341c04900c3; no voters: 2f306549407b4bb281ccf2994035e4f1, dfa1152987644bd781b46c7e42a7a685
W20240502 01:54:22.470341 17542 leader_election.cc:334] T 68bc23294ac64c9486bffebbdb16b508 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
I20240502 01:54:22.472388 17542 leader_election.cc:302] T 68bc23294ac64c9486bffebbdb16b508 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29; no voters: 2f306549407b4bb281ccf2994035e4f1, f19cc6cb1e76460aa6711e49a5775091
I20240502 01:54:22.472764 25917 raft_consensus.cc:482] T b29218a879e74895bfe00021ad358aa4 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dfa1152987644bd781b46c7e42a7a685)
I20240502 01:54:22.472803 24598 raft_consensus.cc:2726] T 68bc23294ac64c9486bffebbdb16b508 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:22.472919 25917 raft_consensus.cc:504] T b29218a879e74895bfe00021ad358aa4 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } }
I20240502 01:54:22.473502 25917 leader_election.cc:288] T b29218a879e74895bfe00021ad358aa4 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:54:22.473608 25604 raft_consensus.cc:2726] T cf0237a4c65b495ba892871dd850bed9 P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:54:22.474324 17473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b29218a879e74895bfe00021ad358aa4" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:54:22.474539 17473 raft_consensus.cc:2454] T b29218a879e74895bfe00021ad358aa4 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 1.
W20240502 01:54:22.474654 17802 leader_election.cc:334] T 8c129f9289024ec9898fbbc06b2fb2cd P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
I20240502 01:54:22.474745 17802 leader_election.cc:302] T 8c129f9289024ec9898fbbc06b2fb2cd P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb31bceba76d466db24b9341c04900c3; no voters: dfa1152987644bd781b46c7e42a7a685, fdc2ac372492454c85387589a91f63af
I20240502 01:54:22.474982 25604 raft_consensus.cc:2726] T 8c129f9289024ec9898fbbc06b2fb2cd P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:22.475346 20433 leader_election.cc:302] T b29218a879e74895bfe00021ad358aa4 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 4da690c0fa13407a85c0689ce8de9ade; no voters:
I20240502 01:54:22.475468 17542 leader_election.cc:302] T b6d89235867f42418f8205167bb29755 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, 6a36c952c51542c0bf7ea3ec2857a9f9; no voters: fdc2ac372492454c85387589a91f63af
W20240502 01:54:22.475631 20433 leader_election.cc:334] T b29218a879e74895bfe00021ad358aa4 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
I20240502 01:54:22.475639 25917 raft_consensus.cc:2781] T b29218a879e74895bfe00021ad358aa4 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:54:22.476225 25917 raft_consensus.cc:482] T b29218a879e74895bfe00021ad358aa4 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader dfa1152987644bd781b46c7e42a7a685)
I20240502 01:54:22.476312 25917 raft_consensus.cc:3037] T b29218a879e74895bfe00021ad358aa4 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:22.476377 25813 raft_consensus.cc:2781] T a390a222562f425eabb11d7f52cbe9a6 P 4da690c0fa13407a85c0689ce8de9ade [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:22.476837 24598 raft_consensus.cc:2781] T b6d89235867f42418f8205167bb29755 P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:22.476984 24598 raft_consensus.cc:686] T b6d89235867f42418f8205167bb29755 P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 LEADER]: Becoming Leader. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Running, Role: LEADER
I20240502 01:54:22.477340 24598 consensus_queue.cc:237] T b6d89235867f42418f8205167bb29755 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } }
I20240502 01:54:22.477632 25516 raft_consensus.cc:482] T 7e9be84ef6a043818f166f6b47898855 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dfa1152987644bd781b46c7e42a7a685)
I20240502 01:54:22.477784 25516 raft_consensus.cc:504] T 7e9be84ef6a043818f166f6b47898855 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } }
I20240502 01:54:22.478377 25516 leader_election.cc:288] T 7e9be84ef6a043818f166f6b47898855 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:54:22.479069 25917 raft_consensus.cc:504] T b29218a879e74895bfe00021ad358aa4 P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } }
I20240502 01:54:22.480058 25577 raft_consensus.cc:482] T 2544ba74e9d146ef963fd92a1a6f8a98 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdc2ac372492454c85387589a91f63af)
I20240502 01:54:22.480216 25577 raft_consensus.cc:504] T 2544ba74e9d146ef963fd92a1a6f8a98 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:54:22.480545 25917 leader_election.cc:288] T b29218a879e74895bfe00021ad358aa4 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 election: Requested vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
W20240502 01:54:22.481514 17542 leader_election.cc:334] T 7e9be84ef6a043818f166f6b47898855 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
I20240502 01:54:22.482792 17473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b29218a879e74895bfe00021ad358aa4" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:54:22.483014 17473 raft_consensus.cc:3037] T b29218a879e74895bfe00021ad358aa4 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:22.486033 17473 raft_consensus.cc:2454] T b29218a879e74895bfe00021ad358aa4 P 4da690c0fa13407a85c0689ce8de9ade [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 2.
I20240502 01:54:22.480968 25577 leader_election.cc:288] T 2544ba74e9d146ef963fd92a1a6f8a98 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
W20240502 01:54:22.482378 20433 leader_election.cc:334] T b29218a879e74895bfe00021ad358aa4 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
I20240502 01:54:22.482515 20494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9be84ef6a043818f166f6b47898855" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
I20240502 01:54:22.486728 20433 leader_election.cc:302] T b29218a879e74895bfe00021ad358aa4 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 4da690c0fa13407a85c0689ce8de9ade; no voters: dfa1152987644bd781b46c7e42a7a685
I20240502 01:54:22.487052 25577 raft_consensus.cc:2781] T b29218a879e74895bfe00021ad358aa4 P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:22.487186 25577 raft_consensus.cc:686] T b29218a879e74895bfe00021ad358aa4 P 2f306549407b4bb281ccf2994035e4f1 [term 2 LEADER]: Becoming Leader. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Running, Role: LEADER
I20240502 01:54:22.487547 25813 raft_consensus.cc:686] T a390a222562f425eabb11d7f52cbe9a6 P 4da690c0fa13407a85c0689ce8de9ade [term 2 LEADER]: Becoming Leader. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Running, Role: LEADER
I20240502 01:54:22.487493 25577 consensus_queue.cc:237] T b29218a879e74895bfe00021ad358aa4 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } }
I20240502 01:54:22.487921 25813 consensus_queue.cc:237] T a390a222562f425eabb11d7f52cbe9a6 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:54:22.489886 24959 raft_consensus.cc:482] T ad067712087e4f78885bd8e1cf102694 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dfa1152987644bd781b46c7e42a7a685)
I20240502 01:54:22.490024 24959 raft_consensus.cc:504] T ad067712087e4f78885bd8e1cf102694 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:22.490676 24959 leader_election.cc:288] T ad067712087e4f78885bd8e1cf102694 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:54:22.490761 17543 leader_election.cc:302] T 7e9be84ef6a043818f166f6b47898855 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29; no voters: 2f306549407b4bb281ccf2994035e4f1, dfa1152987644bd781b46c7e42a7a685
I20240502 01:54:22.491078 25516 raft_consensus.cc:2726] T 7e9be84ef6a043818f166f6b47898855 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:22.493201 17864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2544ba74e9d146ef963fd92a1a6f8a98" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
W20240502 01:54:22.494644 20433 leader_election.cc:334] T ad067712087e4f78885bd8e1cf102694 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
I20240502 01:54:22.496687 17330 catalog_manager.cc:5484] T a390a222562f425eabb11d7f52cbe9a6 P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: term changed from 1 to 2, leader changed from dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) to 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65). New cstate: current_term: 2 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:22.498315 25603 raft_consensus.cc:482] T 8e7f88b6c509477fb0900024e7de8f00 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dfa1152987644bd781b46c7e42a7a685)
I20240502 01:54:22.498485 25603 raft_consensus.cc:504] T 8e7f88b6c509477fb0900024e7de8f00 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } }
I20240502 01:54:22.499164 25603 leader_election.cc:288] T 8e7f88b6c509477fb0900024e7de8f00 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:54:22.499883 20361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad067712087e4f78885bd8e1cf102694" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
W20240502 01:54:22.500872 20303 leader_election.cc:334] T 8e7f88b6c509477fb0900024e7de8f00 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
W20240502 01:54:22.501727 20302 leader_election.cc:334] T 8e7f88b6c509477fb0900024e7de8f00 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
I20240502 01:54:22.501883 20302 leader_election.cc:302] T 8e7f88b6c509477fb0900024e7de8f00 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9; no voters: dfa1152987644bd781b46c7e42a7a685, fdc2ac372492454c85387589a91f63af
I20240502 01:54:22.502173 25603 raft_consensus.cc:2726] T 8e7f88b6c509477fb0900024e7de8f00 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:22.502635 20433 leader_election.cc:302] T ad067712087e4f78885bd8e1cf102694 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1; no voters: 6a36c952c51542c0bf7ea3ec2857a9f9, dfa1152987644bd781b46c7e42a7a685
I20240502 01:54:22.502966 25577 raft_consensus.cc:2726] T ad067712087e4f78885bd8e1cf102694 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:54:22.503448 20434 leader_election.cc:334] T 2544ba74e9d146ef963fd92a1a6f8a98 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
I20240502 01:54:22.503633 20434 leader_election.cc:302] T 2544ba74e9d146ef963fd92a1a6f8a98 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1; no voters: bb31bceba76d466db24b9341c04900c3, fdc2ac372492454c85387589a91f63af
I20240502 01:54:22.503906 25577 raft_consensus.cc:2726] T 2544ba74e9d146ef963fd92a1a6f8a98 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:22.506424 17328 catalog_manager.cc:5484] T bffb870715a243d992107583d7c17f1b P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: term changed from 1 to 2, leader changed from fdc2ac372492454c85387589a91f63af (127.11.42.69) to 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66). New cstate: current_term: 2 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:22.509569 17330 catalog_manager.cc:5484] T 25ad701a0fec4e80b37df5dec598796a P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: term changed from 1 to 2, leader changed from fdc2ac372492454c85387589a91f63af (127.11.42.69) to 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71). New cstate: current_term: 2 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:22.513515 25604 raft_consensus.cc:482] T 062b9d90fb0444f4be37bf09770c9caf P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdc2ac372492454c85387589a91f63af)
I20240502 01:54:22.513655 25604 raft_consensus.cc:504] T 062b9d90fb0444f4be37bf09770c9caf P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } }
I20240502 01:54:22.514259 25604 leader_election.cc:288] T 062b9d90fb0444f4be37bf09770c9caf P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:54:22.515959 17473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "062b9d90fb0444f4be37bf09770c9caf" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:54:22.516171 17473 raft_consensus.cc:2454] T 062b9d90fb0444f4be37bf09770c9caf P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 1.
I20240502 01:54:22.517766 17802 leader_election.cc:302] T 062b9d90fb0444f4be37bf09770c9caf P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, bb31bceba76d466db24b9341c04900c3; no voters:
W20240502 01:54:22.517951 17803 leader_election.cc:334] T 062b9d90fb0444f4be37bf09770c9caf P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
I20240502 01:54:22.518031 25849 raft_consensus.cc:482] T 4aefc61cb95f4e949e7a23a1c677749e P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f19cc6cb1e76460aa6711e49a5775091)
I20240502 01:54:22.518144 25849 raft_consensus.cc:504] T 4aefc61cb95f4e949e7a23a1c677749e P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:22.518712 25849 leader_election.cc:288] T 4aefc61cb95f4e949e7a23a1c677749e P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:54:22.518889 25849 raft_consensus.cc:2781] T 062b9d90fb0444f4be37bf09770c9caf P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:54:22.518950 25849 raft_consensus.cc:482] T 062b9d90fb0444f4be37bf09770c9caf P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader fdc2ac372492454c85387589a91f63af)
I20240502 01:54:22.519023 25849 raft_consensus.cc:3037] T 062b9d90fb0444f4be37bf09770c9caf P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:22.523368 17328 catalog_manager.cc:5484] T b6d89235867f42418f8205167bb29755 P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: term changed from 1 to 2, leader changed from fdc2ac372492454c85387589a91f63af (127.11.42.69) to 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66). New cstate: current_term: 2 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:22.525596 17330 catalog_manager.cc:5484] T b29218a879e74895bfe00021ad358aa4 P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: term changed from 1 to 2, leader changed from dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) to 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71). New cstate: current_term: 2 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:22.520928 25849 raft_consensus.cc:504] T 062b9d90fb0444f4be37bf09770c9caf P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } }
I20240502 01:54:22.527817 25849 leader_election.cc:288] T 062b9d90fb0444f4be37bf09770c9caf P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 election: Requested vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
W20240502 01:54:22.529865 17803 leader_election.cc:334] T 062b9d90fb0444f4be37bf09770c9caf P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
I20240502 01:54:22.533586 25849 raft_consensus.cc:482] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f19cc6cb1e76460aa6711e49a5775091)
I20240502 01:54:22.533710 25849 raft_consensus.cc:504] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:22.534296 25849 leader_election.cc:288] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:54:22.536231 25849 raft_consensus.cc:482] T 2544ba74e9d146ef963fd92a1a6f8a98 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdc2ac372492454c85387589a91f63af)
I20240502 01:54:22.536367 25849 raft_consensus.cc:504] T 2544ba74e9d146ef963fd92a1a6f8a98 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:54:22.536945 25849 leader_election.cc:288] T 2544ba74e9d146ef963fd92a1a6f8a98 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
W20240502 01:54:22.537258 17802 leader_election.cc:334] T 4aefc61cb95f4e949e7a23a1c677749e P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
W20240502 01:54:22.537431 17802 leader_election.cc:334] T 4aefc61cb95f4e949e7a23a1c677749e P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
I20240502 01:54:22.537535 17802 leader_election.cc:302] T 4aefc61cb95f4e949e7a23a1c677749e P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb31bceba76d466db24b9341c04900c3; no voters: dfa1152987644bd781b46c7e42a7a685, f19cc6cb1e76460aa6711e49a5775091
I20240502 01:54:22.537763 25849 raft_consensus.cc:2726] T 4aefc61cb95f4e949e7a23a1c677749e P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:54:22.537825 20494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2544ba74e9d146ef963fd92a1a6f8a98" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
I20240502 01:54:22.538058 20494 raft_consensus.cc:2454] T 2544ba74e9d146ef963fd92a1a6f8a98 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 1.
I20240502 01:54:22.538563 17803 leader_election.cc:302] T 2544ba74e9d146ef963fd92a1a6f8a98 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:54:22.538594 17473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "062b9d90fb0444f4be37bf09770c9caf" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:54:22.538816 17473 raft_consensus.cc:3037] T 062b9d90fb0444f4be37bf09770c9caf P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:54:22.539005 17803 leader_election.cc:334] T 2544ba74e9d146ef963fd92a1a6f8a98 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
I20240502 01:54:22.541235 25849 raft_consensus.cc:2781] T 2544ba74e9d146ef963fd92a1a6f8a98 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:54:22.541313 25849 raft_consensus.cc:482] T 2544ba74e9d146ef963fd92a1a6f8a98 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader fdc2ac372492454c85387589a91f63af)
I20240502 01:54:22.541374 25849 raft_consensus.cc:3037] T 2544ba74e9d146ef963fd92a1a6f8a98 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:22.544447 17473 raft_consensus.cc:2454] T 062b9d90fb0444f4be37bf09770c9caf P 4da690c0fa13407a85c0689ce8de9ade [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 2.
I20240502 01:54:22.546191 25848 raft_consensus.cc:482] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdc2ac372492454c85387589a91f63af)
I20240502 01:54:22.546423 25848 raft_consensus.cc:504] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
I20240502 01:54:22.547147 25848 leader_election.cc:288] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:54:22.545694 25849 raft_consensus.cc:504] T 2544ba74e9d146ef963fd92a1a6f8a98 P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
I20240502 01:54:22.547824 25849 leader_election.cc:288] T 2544ba74e9d146ef963fd92a1a6f8a98 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 election: Requested vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:54:22.548161 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbf05e3b3ad44354be7600c3c07756fd" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:54:22.548401 17604 raft_consensus.cc:2454] T bbf05e3b3ad44354be7600c3c07756fd P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 1.
I20240502 01:54:22.548933 20300 leader_election.cc:302] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, 6a36c952c51542c0bf7ea3ec2857a9f9; no voters:
I20240502 01:54:22.549288 25848 raft_consensus.cc:2781] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:54:22.549232 20494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2544ba74e9d146ef963fd92a1a6f8a98" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
I20240502 01:54:22.549383 25848 raft_consensus.cc:482] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader fdc2ac372492454c85387589a91f63af)
I20240502 01:54:22.549479 25848 raft_consensus.cc:3037] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:22.549455 20494 raft_consensus.cc:3037] T 2544ba74e9d146ef963fd92a1a6f8a98 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:22.551353 25790 raft_consensus.cc:482] T f6cac9478db94a21a9799b80d24c8dc9 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdc2ac372492454c85387589a91f63af)
I20240502 01:54:22.551529 25790 raft_consensus.cc:504] T f6cac9478db94a21a9799b80d24c8dc9 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } }
I20240502 01:54:22.552047 20494 raft_consensus.cc:2454] T 2544ba74e9d146ef963fd92a1a6f8a98 P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 2.
I20240502 01:54:22.552127 25790 leader_election.cc:288] T f6cac9478db94a21a9799b80d24c8dc9 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:54:22.552608 25848 raft_consensus.cc:504] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
I20240502 01:54:22.553809 17803 leader_election.cc:302] T 2544ba74e9d146ef963fd92a1a6f8a98 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:54:22.554148 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbf05e3b3ad44354be7600c3c07756fd" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:54:22.554273 25604 raft_consensus.cc:2781] T 2544ba74e9d146ef963fd92a1a6f8a98 P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:22.554344 17604 raft_consensus.cc:3037] T bbf05e3b3ad44354be7600c3c07756fd P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:22.554411 25604 raft_consensus.cc:686] T 2544ba74e9d146ef963fd92a1a6f8a98 P bb31bceba76d466db24b9341c04900c3 [term 2 LEADER]: Becoming Leader. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Running, Role: LEADER
I20240502 01:54:22.554742 25604 consensus_queue.cc:237] T 2544ba74e9d146ef963fd92a1a6f8a98 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } }
W20240502 01:54:22.555078 17803 leader_election.cc:334] T 2544ba74e9d146ef963fd92a1a6f8a98 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
I20240502 01:54:22.557377 17604 raft_consensus.cc:2454] T bbf05e3b3ad44354be7600c3c07756fd P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 2.
I20240502 01:54:22.557898 20300 leader_election.cc:302] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, 6a36c952c51542c0bf7ea3ec2857a9f9; no voters:
I20240502 01:54:22.558180 25603 raft_consensus.cc:2781] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:22.558344 25603 raft_consensus.cc:686] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 LEADER]: Becoming Leader. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Running, Role: LEADER
I20240502 01:54:22.558609 25603 consensus_queue.cc:237] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } }
W20240502 01:54:22.560297 20303 leader_election.cc:334] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
I20240502 01:54:22.560765 25848 leader_election.cc:288] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 election: Requested vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
W20240502 01:54:22.561746 17542 leader_election.cc:334] T f6cac9478db94a21a9799b80d24c8dc9 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
W20240502 01:54:22.563082 20303 leader_election.cc:334] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
I20240502 01:54:22.563107 25920 raft_consensus.cc:482] T b950e18825ac434ba44586d544667918 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dfa1152987644bd781b46c7e42a7a685)
I20240502 01:54:22.563228 25920 raft_consensus.cc:504] T b950e18825ac434ba44586d544667918 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:22.563819 25920 leader_election.cc:288] T b950e18825ac434ba44586d544667918 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:54:22.564965 20361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b950e18825ac434ba44586d544667918" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:54:22.565192 20361 raft_consensus.cc:2454] T b950e18825ac434ba44586d544667918 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 1.
I20240502 01:54:22.565862 20433 leader_election.cc:302] T b950e18825ac434ba44586d544667918 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 6a36c952c51542c0bf7ea3ec2857a9f9; no voters:
I20240502 01:54:22.566375 25920 raft_consensus.cc:2781] T b950e18825ac434ba44586d544667918 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:54:22.566483 25920 raft_consensus.cc:482] T b950e18825ac434ba44586d544667918 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader dfa1152987644bd781b46c7e42a7a685)
I20240502 01:54:22.566560 25920 raft_consensus.cc:3037] T b950e18825ac434ba44586d544667918 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:22.568712 24817 raft_consensus.cc:482] T b9d82510030b4503ac892175a8bb1bb7 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f19cc6cb1e76460aa6711e49a5775091)
I20240502 01:54:22.568884 24817 raft_consensus.cc:504] T b9d82510030b4503ac892175a8bb1bb7 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:22.569433 24817 leader_election.cc:288] T b9d82510030b4503ac892175a8bb1bb7 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
W20240502 01:54:22.570430 20433 leader_election.cc:334] T b950e18825ac434ba44586d544667918 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
I20240502 01:54:22.572719 25920 raft_consensus.cc:504] T b950e18825ac434ba44586d544667918 P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:22.573402 25920 leader_election.cc:288] T b950e18825ac434ba44586d544667918 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 election: Requested vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
W20240502 01:54:22.574960 20433 leader_election.cc:334] T b950e18825ac434ba44586d544667918 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
I20240502 01:54:22.576941 25813 raft_consensus.cc:482] T a212c17ba64842c99b3d6a78a3b7cc8a P 4da690c0fa13407a85c0689ce8de9ade [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dfa1152987644bd781b46c7e42a7a685)
I20240502 01:54:22.577163 25813 raft_consensus.cc:504] T a212c17ba64842c99b3d6a78a3b7cc8a P 4da690c0fa13407a85c0689ce8de9ade [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:22.577821 25813 leader_election.cc:288] T a212c17ba64842c99b3d6a78a3b7cc8a P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:54:22.580209 25577 raft_consensus.cc:1050] T adc2b57af23c41aebd1258ff8b60d074 P 2f306549407b4bb281ccf2994035e4f1: attempting to promote NON_VOTER 4da690c0fa13407a85c0689ce8de9ade to VOTER
W20240502 01:54:22.581048 17411 leader_election.cc:334] T a212c17ba64842c99b3d6a78a3b7cc8a P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
I20240502 01:54:22.581038 25577 consensus_queue.cc:237] T adc2b57af23c41aebd1258ff8b60d074 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } }
W20240502 01:54:22.581252 17411 leader_election.cc:334] T a212c17ba64842c99b3d6a78a3b7cc8a P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
I20240502 01:54:22.581362 17411 leader_election.cc:302] T a212c17ba64842c99b3d6a78a3b7cc8a P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade; no voters: dfa1152987644bd781b46c7e42a7a685, f19cc6cb1e76460aa6711e49a5775091
I20240502 01:54:22.581748 25813 raft_consensus.cc:2726] T a212c17ba64842c99b3d6a78a3b7cc8a P 4da690c0fa13407a85c0689ce8de9ade [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:54:22.581470 17328 catalog_manager.cc:5484] T 2544ba74e9d146ef963fd92a1a6f8a98 P bb31bceba76d466db24b9341c04900c3 reported cstate change: term changed from 1 to 2, leader changed from fdc2ac372492454c85387589a91f63af (127.11.42.69) to bb31bceba76d466db24b9341c04900c3 (127.11.42.68). New cstate: current_term: 2 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: NON_VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:22.582515 25813 raft_consensus.cc:482] T 5e3695e8ad0a44de9ba23ae3610aacec P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdc2ac372492454c85387589a91f63af)
I20240502 01:54:22.582671 25813 raft_consensus.cc:504] T 5e3695e8ad0a44de9ba23ae3610aacec P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:22.583269 25813 leader_election.cc:288] T 5e3695e8ad0a44de9ba23ae3610aacec P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:54:22.585031 20494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e3695e8ad0a44de9ba23ae3610aacec" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
W20240502 01:54:22.585085 17412 leader_election.cc:334] T 5e3695e8ad0a44de9ba23ae3610aacec P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
I20240502 01:54:22.585266 20494 raft_consensus.cc:2454] T 5e3695e8ad0a44de9ba23ae3610aacec P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
I20240502 01:54:22.585644 17412 leader_election.cc:302] T 5e3695e8ad0a44de9ba23ae3610aacec P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 4da690c0fa13407a85c0689ce8de9ade; no voters: fdc2ac372492454c85387589a91f63af
I20240502 01:54:22.586105 25813 raft_consensus.cc:2781] T 5e3695e8ad0a44de9ba23ae3610aacec P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:54:22.586228 25813 raft_consensus.cc:482] T 5e3695e8ad0a44de9ba23ae3610aacec P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting leader election (detected failure of leader fdc2ac372492454c85387589a91f63af)
I20240502 01:54:22.586369 25813 raft_consensus.cc:3037] T 5e3695e8ad0a44de9ba23ae3610aacec P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:22.589663 25813 raft_consensus.cc:504] T 5e3695e8ad0a44de9ba23ae3610aacec P 4da690c0fa13407a85c0689ce8de9ade [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:22.590049 20361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b950e18825ac434ba44586d544667918" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
I20240502 01:54:22.590302 20361 raft_consensus.cc:3037] T b950e18825ac434ba44586d544667918 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:22.591048 20494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e3695e8ad0a44de9ba23ae3610aacec" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
I20240502 01:54:22.591266 20494 raft_consensus.cc:3037] T 5e3695e8ad0a44de9ba23ae3610aacec P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:54:22.592427 17543 leader_election.cc:334] T f6cac9478db94a21a9799b80d24c8dc9 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
I20240502 01:54:22.592569 17543 leader_election.cc:302] T f6cac9478db94a21a9799b80d24c8dc9 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29; no voters: f19cc6cb1e76460aa6711e49a5775091, fdc2ac372492454c85387589a91f63af
I20240502 01:54:22.592849 25790 raft_consensus.cc:2726] T f6cac9478db94a21a9799b80d24c8dc9 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:22.592908 20361 raft_consensus.cc:2454] T b950e18825ac434ba44586d544667918 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 2.
I20240502 01:54:22.594410 25813 leader_election.cc:288] T 5e3695e8ad0a44de9ba23ae3610aacec P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 election: Requested vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:54:22.594566 25848 raft_consensus.cc:482] T ad067712087e4f78885bd8e1cf102694 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dfa1152987644bd781b46c7e42a7a685)
I20240502 01:54:22.594758 25848 raft_consensus.cc:504] T ad067712087e4f78885bd8e1cf102694 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
W20240502 01:54:22.594498 17412 leader_election.cc:334] T 5e3695e8ad0a44de9ba23ae3610aacec P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
I20240502 01:54:22.595790 20497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad067712087e4f78885bd8e1cf102694" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
I20240502 01:54:22.595376 25848 leader_election.cc:288] T ad067712087e4f78885bd8e1cf102694 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
I20240502 01:54:22.595975 20497 raft_consensus.cc:2454] T ad067712087e4f78885bd8e1cf102694 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 1.
I20240502 01:54:22.596429 20494 raft_consensus.cc:2454] T 5e3695e8ad0a44de9ba23ae3610aacec P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 2.
I20240502 01:54:22.596565 20303 leader_election.cc:302] T ad067712087e4f78885bd8e1cf102694 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 6a36c952c51542c0bf7ea3ec2857a9f9; no voters:
I20240502 01:54:22.597026 17412 leader_election.cc:302] T 5e3695e8ad0a44de9ba23ae3610aacec P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 4da690c0fa13407a85c0689ce8de9ade; no voters: fdc2ac372492454c85387589a91f63af
W20240502 01:54:22.597370 20302 leader_election.cc:334] T b9d82510030b4503ac892175a8bb1bb7 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
W20240502 01:54:22.597733 20302 leader_election.cc:334] T b9d82510030b4503ac892175a8bb1bb7 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
I20240502 01:54:22.597899 20302 leader_election.cc:302] T b9d82510030b4503ac892175a8bb1bb7 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9; no voters: dfa1152987644bd781b46c7e42a7a685, f19cc6cb1e76460aa6711e49a5775091
I20240502 01:54:22.598182 24817 raft_consensus.cc:2781] T ad067712087e4f78885bd8e1cf102694 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:54:22.598188 20433 leader_election.cc:302] T b950e18825ac434ba44586d544667918 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 6a36c952c51542c0bf7ea3ec2857a9f9; no voters: dfa1152987644bd781b46c7e42a7a685
I20240502 01:54:22.598268 24817 raft_consensus.cc:482] T ad067712087e4f78885bd8e1cf102694 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader dfa1152987644bd781b46c7e42a7a685)
I20240502 01:54:22.598381 24817 raft_consensus.cc:3037] T ad067712087e4f78885bd8e1cf102694 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:22.598428 25917 raft_consensus.cc:482] T 7e9be84ef6a043818f166f6b47898855 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dfa1152987644bd781b46c7e42a7a685)
I20240502 01:54:22.598579 25917 raft_consensus.cc:504] T 7e9be84ef6a043818f166f6b47898855 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } }
I20240502 01:54:22.599203 25917 leader_election.cc:288] T 7e9be84ef6a043818f166f6b47898855 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
W20240502 01:54:22.599237 20302 leader_election.cc:334] T ad067712087e4f78885bd8e1cf102694 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
I20240502 01:54:22.599354 25917 raft_consensus.cc:2781] T b950e18825ac434ba44586d544667918 P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:22.599498 25917 raft_consensus.cc:686] T b950e18825ac434ba44586d544667918 P 2f306549407b4bb281ccf2994035e4f1 [term 2 LEADER]: Becoming Leader. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Running, Role: LEADER
I20240502 01:54:22.599738 25917 consensus_queue.cc:237] T b950e18825ac434ba44586d544667918 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } }
I20240502 01:54:22.600659 25358 raft_consensus.cc:2726] T b9d82510030b4503ac892175a8bb1bb7 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:22.601231 24817 raft_consensus.cc:504] T ad067712087e4f78885bd8e1cf102694 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:22.602061 17606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9be84ef6a043818f166f6b47898855" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:54:22.602299 17606 raft_consensus.cc:2454] T 7e9be84ef6a043818f166f6b47898855 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 1.
I20240502 01:54:22.601871 24817 leader_election.cc:288] T ad067712087e4f78885bd8e1cf102694 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 election: Requested vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
W20240502 01:54:22.603070 20303 consensus_peers.cc:462] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
I20240502 01:54:22.602953 17330 catalog_manager.cc:5484] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: term changed from 1 to 2, leader changed from fdc2ac372492454c85387589a91f63af (127.11.42.69) to 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70). New cstate: current_term: 2 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:22.603382 20494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad067712087e4f78885bd8e1cf102694" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1"
I20240502 01:54:22.603585 20494 raft_consensus.cc:3037] T ad067712087e4f78885bd8e1cf102694 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:22.603751 25751 raft_consensus.cc:2781] T 5e3695e8ad0a44de9ba23ae3610aacec P 4da690c0fa13407a85c0689ce8de9ade [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:22.603788 20431 leader_election.cc:302] T 7e9be84ef6a043818f166f6b47898855 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 3fceccefba274b1dbbd8216af8b0bc29; no voters:
I20240502 01:54:22.603906 25751 raft_consensus.cc:686] T 5e3695e8ad0a44de9ba23ae3610aacec P 4da690c0fa13407a85c0689ce8de9ade [term 2 LEADER]: Becoming Leader. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Running, Role: LEADER
I20240502 01:54:22.604104 24959 raft_consensus.cc:2781] T 7e9be84ef6a043818f166f6b47898855 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:54:22.604218 24959 raft_consensus.cc:482] T 7e9be84ef6a043818f166f6b47898855 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader dfa1152987644bd781b46c7e42a7a685)
I20240502 01:54:22.604171 25751 consensus_queue.cc:237] T 5e3695e8ad0a44de9ba23ae3610aacec P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:22.604306 24959 raft_consensus.cc:3037] T 7e9be84ef6a043818f166f6b47898855 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:54:22.604418 20302 leader_election.cc:334] T ad067712087e4f78885bd8e1cf102694 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
I20240502 01:54:22.605208 25790 raft_consensus.cc:482] T 81f335c023444796ab933e19d809cc03 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f19cc6cb1e76460aa6711e49a5775091)
I20240502 01:54:22.605384 25790 raft_consensus.cc:504] T 81f335c023444796ab933e19d809cc03 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } }
I20240502 01:54:22.606036 25790 leader_election.cc:288] T 81f335c023444796ab933e19d809cc03 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
W20240502 01:54:22.606235 20433 consensus_peers.cc:485] T adc2b57af23c41aebd1258ff8b60d074 P 2f306549407b4bb281ccf2994035e4f1 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:22.607630 17542 leader_election.cc:334] T 81f335c023444796ab933e19d809cc03 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
W20240502 01:54:22.609375 17543 leader_election.cc:334] T 81f335c023444796ab933e19d809cc03 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
I20240502 01:54:22.609578 17543 leader_election.cc:302] T 81f335c023444796ab933e19d809cc03 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29; no voters: f19cc6cb1e76460aa6711e49a5775091, fdc2ac372492454c85387589a91f63af
I20240502 01:54:22.609683 25790 raft_consensus.cc:482] T b671a9ceacea4a819f750fd37130be0f P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f19cc6cb1e76460aa6711e49a5775091)
I20240502 01:54:22.609865 24598 raft_consensus.cc:2726] T 81f335c023444796ab933e19d809cc03 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:22.609835 25790 raft_consensus.cc:504] T b671a9ceacea4a819f750fd37130be0f P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:22.610437 25790 leader_election.cc:288] T b671a9ceacea4a819f750fd37130be0f P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623)
W20240502 01:54:22.612306 17542 leader_election.cc:334] T b671a9ceacea4a819f750fd37130be0f P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
W20240502 01:54:22.613754 20433 leader_election.cc:334] T 7e9be84ef6a043818f166f6b47898855 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
I20240502 01:54:22.614602 20497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b671a9ceacea4a819f750fd37130be0f" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
I20240502 01:54:22.614439 17328 catalog_manager.cc:5484] T b950e18825ac434ba44586d544667918 P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: term changed from 1 to 2, leader changed from dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) to 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71). New cstate: current_term: 2 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:22.615226 17543 leader_election.cc:302] T b671a9ceacea4a819f750fd37130be0f P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29; no voters: 2f306549407b4bb281ccf2994035e4f1, f19cc6cb1e76460aa6711e49a5775091
I20240502 01:54:22.615834 25790 raft_consensus.cc:2726] T b671a9ceacea4a819f750fd37130be0f P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:22.617290 17473 raft_consensus.cc:1261] T adc2b57af23c41aebd1258ff8b60d074 P 4da690c0fa13407a85c0689ce8de9ade [term 1 LEARNER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:54:22.617978 25920 consensus_queue.cc:1035] T adc2b57af23c41aebd1258ff8b60d074 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:22.619930 25790 raft_consensus.cc:482] T 074a08b2b64640eb9784b6a23921ce8d P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f19cc6cb1e76460aa6711e49a5775091)
I20240502 01:54:22.620070 25790 raft_consensus.cc:504] T 074a08b2b64640eb9784b6a23921ce8d P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:22.621193 20361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "074a08b2b64640eb9784b6a23921ce8d" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:54:22.621798 25790 leader_election.cc:288] T 074a08b2b64640eb9784b6a23921ce8d P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
W20240502 01:54:22.622695 17542 leader_election.cc:334] T 074a08b2b64640eb9784b6a23921ce8d P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
I20240502 01:54:22.622814 17542 leader_election.cc:302] T 074a08b2b64640eb9784b6a23921ce8d P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29; no voters: 6a36c952c51542c0bf7ea3ec2857a9f9, f19cc6cb1e76460aa6711e49a5775091
I20240502 01:54:22.623207 25790 raft_consensus.cc:2726] T 074a08b2b64640eb9784b6a23921ce8d P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:22.623627 17802 leader_election.cc:302] T 062b9d90fb0444f4be37bf09770c9caf P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, bb31bceba76d466db24b9341c04900c3; no voters: fdc2ac372492454c85387589a91f63af
I20240502 01:54:22.623945 25604 raft_consensus.cc:2781] T 062b9d90fb0444f4be37bf09770c9caf P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:22.624397 25604 raft_consensus.cc:686] T 062b9d90fb0444f4be37bf09770c9caf P bb31bceba76d466db24b9341c04900c3 [term 2 LEADER]: Becoming Leader. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Running, Role: LEADER
I20240502 01:54:22.624691 25604 consensus_queue.cc:237] T 062b9d90fb0444f4be37bf09770c9caf P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } }
W20240502 01:54:22.629113 17802 leader_election.cc:334] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
I20240502 01:54:22.629179 20361 raft_consensus.cc:1261] T adc2b57af23c41aebd1258ff8b60d074 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20240502 01:54:22.630868 17802 leader_election.cc:334] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
I20240502 01:54:22.630981 17802 leader_election.cc:302] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb31bceba76d466db24b9341c04900c3; no voters: dfa1152987644bd781b46c7e42a7a685, f19cc6cb1e76460aa6711e49a5775091
I20240502 01:54:22.631209 25849 raft_consensus.cc:2726] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:22.632156 25920 consensus_queue.cc:1035] T adc2b57af23c41aebd1258ff8b60d074 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:22.637122 25849 raft_consensus.cc:482] T 164e04f7c9454c189f6907b319023d99 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdc2ac372492454c85387589a91f63af)
I20240502 01:54:22.637238 25849 raft_consensus.cc:504] T 164e04f7c9454c189f6907b319023d99 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:22.637818 25849 leader_election.cc:288] T 164e04f7c9454c189f6907b319023d99 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:54:22.638374 17606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "164e04f7c9454c189f6907b319023d99" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:54:22.638594 17606 raft_consensus.cc:2454] T 164e04f7c9454c189f6907b319023d99 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 1.
I20240502 01:54:22.638998 17800 leader_election.cc:302] T 164e04f7c9454c189f6907b319023d99 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:54:22.640096 25849 raft_consensus.cc:2781] T 164e04f7c9454c189f6907b319023d99 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:54:22.640173 25849 raft_consensus.cc:482] T 164e04f7c9454c189f6907b319023d99 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader fdc2ac372492454c85387589a91f63af)
I20240502 01:54:22.640237 25849 raft_consensus.cc:3037] T 164e04f7c9454c189f6907b319023d99 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:22.642925 25849 raft_consensus.cc:504] T 164e04f7c9454c189f6907b319023d99 P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:22.643815 17606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "164e04f7c9454c189f6907b319023d99" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:54:22.644011 17606 raft_consensus.cc:3037] T 164e04f7c9454c189f6907b319023d99 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:22.646601 17606 raft_consensus.cc:2454] T 164e04f7c9454c189f6907b319023d99 P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 2.
I20240502 01:54:22.647042 17800 leader_election.cc:302] T 164e04f7c9454c189f6907b319023d99 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, bb31bceba76d466db24b9341c04900c3; no voters:
I20240502 01:54:22.647405 25429 raft_consensus.cc:2781] T 164e04f7c9454c189f6907b319023d99 P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:22.647529 25429 raft_consensus.cc:686] T 164e04f7c9454c189f6907b319023d99 P bb31bceba76d466db24b9341c04900c3 [term 2 LEADER]: Becoming Leader. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Running, Role: LEADER
I20240502 01:54:22.647797 25429 consensus_queue.cc:237] T 164e04f7c9454c189f6907b319023d99 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
W20240502 01:54:22.648038 17803 leader_election.cc:334] T 164e04f7c9454c189f6907b319023d99 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
I20240502 01:54:22.652705 25603 raft_consensus.cc:482] T f3e4c8f98b5e48e3ba6affc2df08943a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f19cc6cb1e76460aa6711e49a5775091)
I20240502 01:54:22.652880 25603 raft_consensus.cc:504] T f3e4c8f98b5e48e3ba6affc2df08943a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } }
I20240502 01:54:22.653349 25849 leader_election.cc:288] T 164e04f7c9454c189f6907b319023d99 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 election: Requested vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:54:22.653633 25603 leader_election.cc:288] T f3e4c8f98b5e48e3ba6affc2df08943a P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71:38623), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:54:22.654182 20497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3e4c8f98b5e48e3ba6affc2df08943a" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: false dest_uuid: "2f306549407b4bb281ccf2994035e4f1" is_pre_election: true
W20240502 01:54:22.654795 17803 leader_election.cc:334] T 164e04f7c9454c189f6907b319023d99 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
W20240502 01:54:22.656221 20302 leader_election.cc:334] T f3e4c8f98b5e48e3ba6affc2df08943a P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
I20240502 01:54:22.656370 20302 leader_election.cc:302] T f3e4c8f98b5e48e3ba6affc2df08943a P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9; no voters: 2f306549407b4bb281ccf2994035e4f1, f19cc6cb1e76460aa6711e49a5775091
I20240502 01:54:22.656607 25603 raft_consensus.cc:2726] T f3e4c8f98b5e48e3ba6affc2df08943a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:54:22.681511 25849 raft_consensus.cc:482] T 57ab576948e346cbbe6715bf53f7c665 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dfa1152987644bd781b46c7e42a7a685)
I20240502 01:54:22.681656 25849 raft_consensus.cc:504] T 57ab576948e346cbbe6715bf53f7c665 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:22.682240 25849 leader_election.cc:288] T 57ab576948e346cbbe6715bf53f7c665 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
W20240502 01:54:22.683818 17802 leader_election.cc:334] T 57ab576948e346cbbe6715bf53f7c665 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
I20240502 01:54:22.684052 17328 catalog_manager.cc:5484] T 5e3695e8ad0a44de9ba23ae3610aacec P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: term changed from 1 to 2, leader changed from fdc2ac372492454c85387589a91f63af (127.11.42.69) to 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65). New cstate: current_term: 2 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:22.690516 20494 raft_consensus.cc:2454] T ad067712087e4f78885bd8e1cf102694 P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 2.
I20240502 01:54:22.691097 20303 leader_election.cc:302] T ad067712087e4f78885bd8e1cf102694 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 6a36c952c51542c0bf7ea3ec2857a9f9; no voters: dfa1152987644bd781b46c7e42a7a685
I20240502 01:54:22.691421 25603 raft_consensus.cc:2781] T ad067712087e4f78885bd8e1cf102694 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:22.691565 25603 raft_consensus.cc:686] T ad067712087e4f78885bd8e1cf102694 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 LEADER]: Becoming Leader. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Running, Role: LEADER
I20240502 01:54:22.691830 25603 consensus_queue.cc:237] T ad067712087e4f78885bd8e1cf102694 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:22.693081 24959 raft_consensus.cc:504] T 7e9be84ef6a043818f166f6b47898855 P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } }
I20240502 01:54:22.694289 24959 leader_election.cc:288] T 7e9be84ef6a043818f166f6b47898855 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 election: Requested vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:54:22.695382 17606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9be84ef6a043818f166f6b47898855" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:54:22.695667 17606 raft_consensus.cc:3037] T 7e9be84ef6a043818f166f6b47898855 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:22.695498 24959 raft_consensus.cc:2932] T adc2b57af23c41aebd1258ff8b60d074 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:22.698709 17606 raft_consensus.cc:2454] T 7e9be84ef6a043818f166f6b47898855 P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 2.
I20240502 01:54:22.699347 20431 leader_election.cc:302] T 7e9be84ef6a043818f166f6b47898855 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 3fceccefba274b1dbbd8216af8b0bc29; no voters:
I20240502 01:54:22.699632 25577 raft_consensus.cc:2781] T 7e9be84ef6a043818f166f6b47898855 P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:22.699760 25577 raft_consensus.cc:686] T 7e9be84ef6a043818f166f6b47898855 P 2f306549407b4bb281ccf2994035e4f1 [term 2 LEADER]: Becoming Leader. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Running, Role: LEADER
W20240502 01:54:22.699802 20433 leader_election.cc:334] T 7e9be84ef6a043818f166f6b47898855 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
I20240502 01:54:22.700037 25577 consensus_queue.cc:237] T 7e9be84ef6a043818f166f6b47898855 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } }
I20240502 01:54:22.702188 17473 raft_consensus.cc:2932] T adc2b57af23c41aebd1258ff8b60d074 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:22.703682 17330 catalog_manager.cc:5484] T ad067712087e4f78885bd8e1cf102694 P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: term changed from 1 to 2, leader changed from dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) to 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70). New cstate: current_term: 2 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:22.704952 17328 catalog_manager.cc:5484] T adc2b57af23c41aebd1258ff8b60d074 P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:22.714577 25920 raft_consensus.cc:482] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dfa1152987644bd781b46c7e42a7a685)
I20240502 01:54:22.714725 25920 raft_consensus.cc:504] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } }
I20240502 01:54:22.715306 25920 leader_election.cc:288] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
W20240502 01:54:22.718560 20433 leader_election.cc:334] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
I20240502 01:54:22.718425 20361 raft_consensus.cc:2932] T adc2b57af23c41aebd1258ff8b60d074 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { replace: true } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:22.722316 20494 consensus_queue.cc:237] T adc2b57af23c41aebd1258ff8b60d074 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } }
I20240502 01:54:22.723254 25813 raft_consensus.cc:482] T 75bb4e19cf57405fabcb532b55a29ced P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f19cc6cb1e76460aa6711e49a5775091)
I20240502 01:54:22.723429 25813 raft_consensus.cc:504] T 75bb4e19cf57405fabcb532b55a29ced P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } }
I20240502 01:54:22.724059 25813 leader_election.cc:288] T 75bb4e19cf57405fabcb532b55a29ced P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
W20240502 01:54:22.726662 17411 leader_election.cc:334] T 75bb4e19cf57405fabcb532b55a29ced P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
I20240502 01:54:22.727608 17864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75bb4e19cf57405fabcb532b55a29ced" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:54:22.727813 17864 raft_consensus.cc:2454] T 75bb4e19cf57405fabcb532b55a29ced P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
I20240502 01:54:22.728288 17412 leader_election.cc:302] T 75bb4e19cf57405fabcb532b55a29ced P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, bb31bceba76d466db24b9341c04900c3; no voters: f19cc6cb1e76460aa6711e49a5775091
I20240502 01:54:22.728585 25813 raft_consensus.cc:2781] T 75bb4e19cf57405fabcb532b55a29ced P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:54:22.728667 25813 raft_consensus.cc:482] T 75bb4e19cf57405fabcb532b55a29ced P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting leader election (detected failure of leader f19cc6cb1e76460aa6711e49a5775091)
I20240502 01:54:22.728799 25813 raft_consensus.cc:3037] T 75bb4e19cf57405fabcb532b55a29ced P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:22.730016 17473 raft_consensus.cc:1261] T adc2b57af23c41aebd1258ff8b60d074 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:22.730585 25920 consensus_queue.cc:1035] T adc2b57af23c41aebd1258ff8b60d074 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:54:22.730965 17800 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.42.66:38531, user_credentials={real_user=slave}} blocked reactor thread for 83939.4us
I20240502 01:54:22.731541 25813 raft_consensus.cc:504] T 75bb4e19cf57405fabcb532b55a29ced P 4da690c0fa13407a85c0689ce8de9ade [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } }
I20240502 01:54:22.731875 17606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ab576948e346cbbe6715bf53f7c665" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:54:22.732115 17606 raft_consensus.cc:2454] T 57ab576948e346cbbe6715bf53f7c665 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 1.
I20240502 01:54:22.732172 25813 leader_election.cc:288] T 75bb4e19cf57405fabcb532b55a29ced P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 election: Requested vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:54:22.733840 25920 raft_consensus.cc:482] T 68bc23294ac64c9486bffebbdb16b508 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f19cc6cb1e76460aa6711e49a5775091)
I20240502 01:54:22.733983 25920 raft_consensus.cc:504] T 68bc23294ac64c9486bffebbdb16b508 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } }
W20240502 01:54:22.734047 17411 leader_election.cc:334] T 75bb4e19cf57405fabcb532b55a29ced P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
I20240502 01:54:22.734603 25920 leader_election.cc:288] T 68bc23294ac64c9486bffebbdb16b508 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:54:22.734581 17864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75bb4e19cf57405fabcb532b55a29ced" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:54:22.734762 17864 raft_consensus.cc:3037] T 75bb4e19cf57405fabcb532b55a29ced P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:54:22.736289 20433 leader_election.cc:334] T 68bc23294ac64c9486bffebbdb16b508 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
I20240502 01:54:22.737165 17606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68bc23294ac64c9486bffebbdb16b508" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:54:22.737388 17606 raft_consensus.cc:2454] T 68bc23294ac64c9486bffebbdb16b508 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 1.
I20240502 01:54:22.737680 25751 raft_consensus.cc:482] T 52430bbf347a4c5191edb23e1aa5d385 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dfa1152987644bd781b46c7e42a7a685)
I20240502 01:54:22.737793 20431 leader_election.cc:302] T 68bc23294ac64c9486bffebbdb16b508 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 3fceccefba274b1dbbd8216af8b0bc29; no voters: f19cc6cb1e76460aa6711e49a5775091
I20240502 01:54:22.737843 25751 raft_consensus.cc:504] T 52430bbf347a4c5191edb23e1aa5d385 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } }
I20240502 01:54:22.738107 17864 raft_consensus.cc:2454] T 75bb4e19cf57405fabcb532b55a29ced P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 2.
I20240502 01:54:22.738168 25920 raft_consensus.cc:2781] T 68bc23294ac64c9486bffebbdb16b508 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:54:22.738270 25920 raft_consensus.cc:482] T 68bc23294ac64c9486bffebbdb16b508 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f19cc6cb1e76460aa6711e49a5775091)
I20240502 01:54:22.738359 25920 raft_consensus.cc:3037] T 68bc23294ac64c9486bffebbdb16b508 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:22.738440 25751 leader_election.cc:288] T 52430bbf347a4c5191edb23e1aa5d385 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:54:22.738512 17412 leader_election.cc:302] T 75bb4e19cf57405fabcb532b55a29ced P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, bb31bceba76d466db24b9341c04900c3; no voters: f19cc6cb1e76460aa6711e49a5775091
I20240502 01:54:22.738751 25751 raft_consensus.cc:2781] T 75bb4e19cf57405fabcb532b55a29ced P 4da690c0fa13407a85c0689ce8de9ade [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:22.738893 25751 raft_consensus.cc:686] T 75bb4e19cf57405fabcb532b55a29ced P 4da690c0fa13407a85c0689ce8de9ade [term 2 LEADER]: Becoming Leader. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Running, Role: LEADER
I20240502 01:54:22.739198 25751 consensus_queue.cc:237] T 75bb4e19cf57405fabcb532b55a29ced P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } }
I20240502 01:54:22.740921 25920 raft_consensus.cc:504] T 68bc23294ac64c9486bffebbdb16b508 P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } }
W20240502 01:54:22.741058 17411 leader_election.cc:334] T 52430bbf347a4c5191edb23e1aa5d385 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
W20240502 01:54:22.741927 17412 leader_election.cc:334] T 52430bbf347a4c5191edb23e1aa5d385 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
I20240502 01:54:22.742082 17412 leader_election.cc:302] T 52430bbf347a4c5191edb23e1aa5d385 P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade; no voters: dfa1152987644bd781b46c7e42a7a685, fdc2ac372492454c85387589a91f63af
I20240502 01:54:22.742624 25790 raft_consensus.cc:482] T 1b14ac3413004127b72904b631ec1b94 P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dfa1152987644bd781b46c7e42a7a685)
I20240502 01:54:22.742794 25790 raft_consensus.cc:504] T 1b14ac3413004127b72904b631ec1b94 P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:22.743475 25790 leader_election.cc:288] T 1b14ac3413004127b72904b631ec1b94 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
W20240502 01:54:22.745282 17542 leader_election.cc:334] T 1b14ac3413004127b72904b631ec1b94 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
I20240502 01:54:22.746151 25813 raft_consensus.cc:2726] T 52430bbf347a4c5191edb23e1aa5d385 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:54:22.747135 20433 leader_election.cc:334] T 68bc23294ac64c9486bffebbdb16b508 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
W20240502 01:54:22.747361 17543 leader_election.cc:334] T 1b14ac3413004127b72904b631ec1b94 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
I20240502 01:54:22.747478 17543 leader_election.cc:302] T 1b14ac3413004127b72904b631ec1b94 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29; no voters: dfa1152987644bd781b46c7e42a7a685, fdc2ac372492454c85387589a91f63af
I20240502 01:54:22.749262 25790 raft_consensus.cc:2726] T 1b14ac3413004127b72904b631ec1b94 P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:54:22.749886 25920 leader_election.cc:288] T 68bc23294ac64c9486bffebbdb16b508 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 election: Requested vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:54:22.749959 17606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68bc23294ac64c9486bffebbdb16b508" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:54:22.750141 17606 raft_consensus.cc:3037] T 68bc23294ac64c9486bffebbdb16b508 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:22.752763 17606 raft_consensus.cc:2454] T 68bc23294ac64c9486bffebbdb16b508 P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 2.
I20240502 01:54:22.753441 25813 raft_consensus.cc:482] T 975f253ff03447ff96fe84169c05097e P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f19cc6cb1e76460aa6711e49a5775091)
I20240502 01:54:22.753614 25813 raft_consensus.cc:504] T 975f253ff03447ff96fe84169c05097e P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } }
I20240502 01:54:22.753851 20431 leader_election.cc:302] T 68bc23294ac64c9486bffebbdb16b508 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 3fceccefba274b1dbbd8216af8b0bc29; no voters: f19cc6cb1e76460aa6711e49a5775091
I20240502 01:54:22.754261 25920 raft_consensus.cc:2781] T 68bc23294ac64c9486bffebbdb16b508 P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:22.754433 25920 raft_consensus.cc:686] T 68bc23294ac64c9486bffebbdb16b508 P 2f306549407b4bb281ccf2994035e4f1 [term 2 LEADER]: Becoming Leader. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Running, Role: LEADER
I20240502 01:54:22.754606 25813 leader_election.cc:288] T 975f253ff03447ff96fe84169c05097e P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
I20240502 01:54:22.754746 25920 consensus_queue.cc:237] T 68bc23294ac64c9486bffebbdb16b508 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } }
I20240502 01:54:22.755892 20367 raft_consensus.cc:1261] T adc2b57af23c41aebd1258ff8b60d074 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:22.756873 25917 consensus_queue.cc:1035] T adc2b57af23c41aebd1258ff8b60d074 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:54:22.758325 17411 leader_election.cc:334] T 975f253ff03447ff96fe84169c05097e P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
I20240502 01:54:22.758658 17606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975f253ff03447ff96fe84169c05097e" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:54:22.759217 17409 leader_election.cc:302] T 975f253ff03447ff96fe84169c05097e P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade; no voters: 3fceccefba274b1dbbd8216af8b0bc29, f19cc6cb1e76460aa6711e49a5775091
I20240502 01:54:22.759676 25813 raft_consensus.cc:2726] T 975f253ff03447ff96fe84169c05097e P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:22.749796 17330 catalog_manager.cc:5484] T 7e9be84ef6a043818f166f6b47898855 P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: term changed from 1 to 2, leader changed from dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) to 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71). New cstate: current_term: 2 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:22.762033 25849 raft_consensus.cc:482] T ee28f4aa31504dbfac977c4178aefdb1 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f19cc6cb1e76460aa6711e49a5775091)
I20240502 01:54:22.762185 25849 raft_consensus.cc:504] T ee28f4aa31504dbfac977c4178aefdb1 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } }
I20240502 01:54:22.763756 25849 leader_election.cc:288] T ee28f4aa31504dbfac977c4178aefdb1 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
W20240502 01:54:22.764818 17802 leader_election.cc:334] T ee28f4aa31504dbfac977c4178aefdb1 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
W20240502 01:54:22.765962 17802 leader_election.cc:334] T ee28f4aa31504dbfac977c4178aefdb1 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
I20240502 01:54:22.766080 17802 leader_election.cc:302] T ee28f4aa31504dbfac977c4178aefdb1 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb31bceba76d466db24b9341c04900c3; no voters: dfa1152987644bd781b46c7e42a7a685, f19cc6cb1e76460aa6711e49a5775091
I20240502 01:54:22.766613 25849 raft_consensus.cc:2726] T ee28f4aa31504dbfac977c4178aefdb1 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:22.769493 17330 catalog_manager.cc:5484] T 75bb4e19cf57405fabcb532b55a29ced P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: term changed from 1 to 2, leader changed from f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) to 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65). New cstate: current_term: 2 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:22.774960 17864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf0237a4c65b495ba892871dd850bed9" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:54:22.775211 17864 raft_consensus.cc:2454] T cf0237a4c65b495ba892871dd850bed9 P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 2.
I20240502 01:54:22.776059 20434 leader_election.cc:302] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, bb31bceba76d466db24b9341c04900c3; no voters: dfa1152987644bd781b46c7e42a7a685
I20240502 01:54:22.776512 24959 raft_consensus.cc:2781] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 01:54:22.776662 24959 raft_consensus.cc:482] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Starting leader election (detected failure of leader dfa1152987644bd781b46c7e42a7a685)
I20240502 01:54:22.776818 24959 raft_consensus.cc:3037] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:22.781461 24959 raft_consensus.cc:504] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } }
W20240502 01:54:22.783358 20433 leader_election.cc:334] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
I20240502 01:54:22.783885 24959 leader_election.cc:288] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 3 election: Requested vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:54:22.785046 25917 raft_consensus.cc:2932] T adc2b57af23c41aebd1258ff8b60d074 P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:22.788750 17322 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet adc2b57af23c41aebd1258ff8b60d074 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:22.791016 25790 raft_consensus.cc:482] T 975f253ff03447ff96fe84169c05097e P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f19cc6cb1e76460aa6711e49a5775091)
I20240502 01:54:22.791167 25790 raft_consensus.cc:504] T 975f253ff03447ff96fe84169c05097e P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } }
I20240502 01:54:22.791764 25790 leader_election.cc:288] T 975f253ff03447ff96fe84169c05097e P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:54:22.792629 17473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975f253ff03447ff96fe84169c05097e" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade" is_pre_election: true
I20240502 01:54:22.792838 17473 raft_consensus.cc:2454] T 975f253ff03447ff96fe84169c05097e P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 1.
I20240502 01:54:22.793881 17542 leader_election.cc:302] T 975f253ff03447ff96fe84169c05097e P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, 4da690c0fa13407a85c0689ce8de9ade; no voters:
W20240502 01:54:22.794837 17542 leader_election.cc:334] T 975f253ff03447ff96fe84169c05097e P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
I20240502 01:54:22.795178 25790 raft_consensus.cc:2781] T 975f253ff03447ff96fe84169c05097e P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:54:22.795289 25790 raft_consensus.cc:482] T 975f253ff03447ff96fe84169c05097e P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f19cc6cb1e76460aa6711e49a5775091)
I20240502 01:54:22.795361 25790 raft_consensus.cc:3037] T 975f253ff03447ff96fe84169c05097e P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:22.798219 25790 raft_consensus.cc:504] T 975f253ff03447ff96fe84169c05097e P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } }
I20240502 01:54:22.800861 25790 leader_election.cc:288] T 975f253ff03447ff96fe84169c05097e P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 election: Requested vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:54:22.800746 20361 raft_consensus.cc:2932] T adc2b57af23c41aebd1258ff8b60d074 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:22.802089 17473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975f253ff03447ff96fe84169c05097e" candidate_uuid: "3fceccefba274b1dbbd8216af8b0bc29" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4da690c0fa13407a85c0689ce8de9ade"
I20240502 01:54:22.802287 17473 raft_consensus.cc:3037] T 975f253ff03447ff96fe84169c05097e P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:22.804939 17473 raft_consensus.cc:2454] T 975f253ff03447ff96fe84169c05097e P 4da690c0fa13407a85c0689ce8de9ade [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fceccefba274b1dbbd8216af8b0bc29 in term 2.
I20240502 01:54:22.805668 25577 raft_consensus.cc:482] T 38bb8dd915f743f49b26ee2572871375 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f19cc6cb1e76460aa6711e49a5775091)
I20240502 01:54:22.805859 25577 raft_consensus.cc:504] T 38bb8dd915f743f49b26ee2572871375 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } }
I20240502 01:54:22.806535 25577 leader_election.cc:288] T 38bb8dd915f743f49b26ee2572871375 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
W20240502 01:54:22.808634 17542 leader_election.cc:334] T 975f253ff03447ff96fe84169c05097e P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
I20240502 01:54:22.809666 17328 catalog_manager.cc:5484] T adc2b57af23c41aebd1258ff8b60d074 P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: config changed from index 3 to 4, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New cstate: current_term: 1 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:22.810887 25603 raft_consensus.cc:482] T 074a08b2b64640eb9784b6a23921ce8d P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f19cc6cb1e76460aa6711e49a5775091)
I20240502 01:54:22.811058 25603 raft_consensus.cc:504] T 074a08b2b64640eb9784b6a23921ce8d P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:22.811676 25603 leader_election.cc:288] T 074a08b2b64640eb9784b6a23921ce8d P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:54:22.812043 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "074a08b2b64640eb9784b6a23921ce8d" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:54:22.812250 17604 raft_consensus.cc:2454] T 074a08b2b64640eb9784b6a23921ce8d P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 1.
I20240502 01:54:22.812655 20300 leader_election.cc:302] T 074a08b2b64640eb9784b6a23921ce8d P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, 6a36c952c51542c0bf7ea3ec2857a9f9; no voters:
I20240502 01:54:22.813170 25603 raft_consensus.cc:2781] T 074a08b2b64640eb9784b6a23921ce8d P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:54:22.813268 25603 raft_consensus.cc:482] T 074a08b2b64640eb9784b6a23921ce8d P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f19cc6cb1e76460aa6711e49a5775091)
W20240502 01:54:22.813331 20302 leader_election.cc:334] T 074a08b2b64640eb9784b6a23921ce8d P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
I20240502 01:54:22.813395 25603 raft_consensus.cc:3037] T 074a08b2b64640eb9784b6a23921ce8d P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:22.815925 25603 raft_consensus.cc:504] T 074a08b2b64640eb9784b6a23921ce8d P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:22.816854 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "074a08b2b64640eb9784b6a23921ce8d" candidate_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:54:22.817096 17604 raft_consensus.cc:3037] T 074a08b2b64640eb9784b6a23921ce8d P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:54:22.818024 17321 catalog_manager.cc:4573] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet adc2b57af23c41aebd1258ff8b60d074: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
I20240502 01:54:22.819774 17604 raft_consensus.cc:2454] T 074a08b2b64640eb9784b6a23921ce8d P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a36c952c51542c0bf7ea3ec2857a9f9 in term 2.
I20240502 01:54:22.820659 20300 leader_election.cc:302] T 074a08b2b64640eb9784b6a23921ce8d P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, 6a36c952c51542c0bf7ea3ec2857a9f9; no voters:
I20240502 01:54:22.820981 25848 raft_consensus.cc:2781] T 074a08b2b64640eb9784b6a23921ce8d P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:22.821143 25848 raft_consensus.cc:686] T 074a08b2b64640eb9784b6a23921ce8d P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 LEADER]: Becoming Leader. State: Replica: 6a36c952c51542c0bf7ea3ec2857a9f9, State: Running, Role: LEADER
I20240502 01:54:22.821410 25848 consensus_queue.cc:237] T 074a08b2b64640eb9784b6a23921ce8d P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:22.823374 25603 leader_election.cc:288] T 074a08b2b64640eb9784b6a23921ce8d P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 election: Requested vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
W20240502 01:54:22.824012 20302 leader_election.cc:334] T 074a08b2b64640eb9784b6a23921ce8d P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
I20240502 01:54:22.824266 25577 raft_consensus.cc:482] T b671a9ceacea4a819f750fd37130be0f P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f19cc6cb1e76460aa6711e49a5775091)
I20240502 01:54:22.824416 25577 raft_consensus.cc:504] T b671a9ceacea4a819f750fd37130be0f P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:22.824950 25577 leader_election.cc:288] T b671a9ceacea4a819f750fd37130be0f P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:54:22.825662 17605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b671a9ceacea4a819f750fd37130be0f" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29" is_pre_election: true
I20240502 01:54:22.825887 17605 raft_consensus.cc:2454] T b671a9ceacea4a819f750fd37130be0f P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 1.
I20240502 01:54:22.826398 20431 leader_election.cc:302] T b671a9ceacea4a819f750fd37130be0f P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 3fceccefba274b1dbbd8216af8b0bc29; no voters:
I20240502 01:54:22.826716 25577 raft_consensus.cc:2781] T b671a9ceacea4a819f750fd37130be0f P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:54:22.826840 25577 raft_consensus.cc:482] T b671a9ceacea4a819f750fd37130be0f P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f19cc6cb1e76460aa6711e49a5775091)
I20240502 01:54:22.826931 25577 raft_consensus.cc:3037] T b671a9ceacea4a819f750fd37130be0f P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:22.834384 17328 catalog_manager.cc:5484] T 074a08b2b64640eb9784b6a23921ce8d P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: term changed from 1 to 2, leader changed from f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) to 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70). New cstate: current_term: 2 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:22.835386 25577 raft_consensus.cc:504] T b671a9ceacea4a819f750fd37130be0f P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:22.836232 17605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b671a9ceacea4a819f750fd37130be0f" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:54:22.836433 17605 raft_consensus.cc:3037] T b671a9ceacea4a819f750fd37130be0f P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:22.839107 17605 raft_consensus.cc:2454] T b671a9ceacea4a819f750fd37130be0f P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 2.
I20240502 01:54:22.840204 20431 leader_election.cc:302] T b671a9ceacea4a819f750fd37130be0f P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 3fceccefba274b1dbbd8216af8b0bc29; no voters:
I20240502 01:54:22.840742 25751 raft_consensus.cc:482] T d25f2ab1a7544cec893dd02c7c89285c P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f19cc6cb1e76460aa6711e49a5775091)
I20240502 01:54:22.841306 25751 raft_consensus.cc:504] T d25f2ab1a7544cec893dd02c7c89285c P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:22.841583 17542 leader_election.cc:302] T 975f253ff03447ff96fe84169c05097e P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, 4da690c0fa13407a85c0689ce8de9ade; no voters: f19cc6cb1e76460aa6711e49a5775091
I20240502 01:54:22.841851 24598 raft_consensus.cc:2781] T 975f253ff03447ff96fe84169c05097e P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:22.841953 25751 leader_election.cc:288] T d25f2ab1a7544cec893dd02c7c89285c P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:54:22.841979 24598 raft_consensus.cc:686] T 975f253ff03447ff96fe84169c05097e P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 LEADER]: Becoming Leader. State: Replica: 3fceccefba274b1dbbd8216af8b0bc29, State: Running, Role: LEADER
I20240502 01:54:22.842268 24598 consensus_queue.cc:237] T 975f253ff03447ff96fe84169c05097e P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } }
W20240502 01:54:22.844053 17411 leader_election.cc:334] T d25f2ab1a7544cec893dd02c7c89285c P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
I20240502 01:54:22.844671 17864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d25f2ab1a7544cec893dd02c7c89285c" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3" is_pre_election: true
I20240502 01:54:22.844899 17864 raft_consensus.cc:2454] T d25f2ab1a7544cec893dd02c7c89285c P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 1.
I20240502 01:54:22.845381 17412 leader_election.cc:302] T d25f2ab1a7544cec893dd02c7c89285c P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, bb31bceba76d466db24b9341c04900c3; no voters: f19cc6cb1e76460aa6711e49a5775091
I20240502 01:54:22.845656 25751 raft_consensus.cc:2781] T d25f2ab1a7544cec893dd02c7c89285c P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:54:22.845741 25751 raft_consensus.cc:482] T d25f2ab1a7544cec893dd02c7c89285c P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Starting leader election (detected failure of leader f19cc6cb1e76460aa6711e49a5775091)
I20240502 01:54:22.845844 25751 raft_consensus.cc:3037] T d25f2ab1a7544cec893dd02c7c89285c P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:22.848448 25751 raft_consensus.cc:504] T d25f2ab1a7544cec893dd02c7c89285c P 4da690c0fa13407a85c0689ce8de9ade [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:22.848464 17328 catalog_manager.cc:5484] T 975f253ff03447ff96fe84169c05097e P 3fceccefba274b1dbbd8216af8b0bc29 reported cstate change: term changed from 1 to 2, leader changed from f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) to 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66). New cstate: current_term: 2 leader_uuid: "3fceccefba274b1dbbd8216af8b0bc29" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 01:54:22.850302 17411 leader_election.cc:334] T d25f2ab1a7544cec893dd02c7c89285c P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
I20240502 01:54:22.850807 25751 leader_election.cc:288] T d25f2ab1a7544cec893dd02c7c89285c P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 election: Requested vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
I20240502 01:54:22.851127 25917 raft_consensus.cc:482] T da06571e020948059fb06f65f26c831d P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f19cc6cb1e76460aa6711e49a5775091)
I20240502 01:54:22.851305 17864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d25f2ab1a7544cec893dd02c7c89285c" candidate_uuid: "4da690c0fa13407a85c0689ce8de9ade" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:54:22.851284 25917 raft_consensus.cc:504] T da06571e020948059fb06f65f26c831d P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } }
I20240502 01:54:22.851480 17864 raft_consensus.cc:3037] T d25f2ab1a7544cec893dd02c7c89285c P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:22.851835 25917 leader_election.cc:288] T da06571e020948059fb06f65f26c831d P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
I20240502 01:54:22.851975 25917 raft_consensus.cc:2781] T b671a9ceacea4a819f750fd37130be0f P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:22.852092 25917 raft_consensus.cc:686] T b671a9ceacea4a819f750fd37130be0f P 2f306549407b4bb281ccf2994035e4f1 [term 2 LEADER]: Becoming Leader. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Running, Role: LEADER
I20240502 01:54:22.852382 25917 consensus_queue.cc:237] T b671a9ceacea4a819f750fd37130be0f P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } }
I20240502 01:54:22.854190 17864 raft_consensus.cc:2454] T d25f2ab1a7544cec893dd02c7c89285c P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4da690c0fa13407a85c0689ce8de9ade in term 2.
I20240502 01:54:22.854741 17412 leader_election.cc:302] T d25f2ab1a7544cec893dd02c7c89285c P 4da690c0fa13407a85c0689ce8de9ade [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4da690c0fa13407a85c0689ce8de9ade, bb31bceba76d466db24b9341c04900c3; no voters: f19cc6cb1e76460aa6711e49a5775091
I20240502 01:54:22.855183 25751 raft_consensus.cc:2781] T d25f2ab1a7544cec893dd02c7c89285c P 4da690c0fa13407a85c0689ce8de9ade [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:22.855387 25751 raft_consensus.cc:686] T d25f2ab1a7544cec893dd02c7c89285c P 4da690c0fa13407a85c0689ce8de9ade [term 2 LEADER]: Becoming Leader. State: Replica: 4da690c0fa13407a85c0689ce8de9ade, State: Running, Role: LEADER
I20240502 01:54:22.855687 25751 consensus_queue.cc:237] T d25f2ab1a7544cec893dd02c7c89285c P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:22.859453 25790 raft_consensus.cc:482] T 3ff84ce60ac1438d92b2c944bb27aabb P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f19cc6cb1e76460aa6711e49a5775091)
I20240502 01:54:22.859634 25790 raft_consensus.cc:504] T 3ff84ce60ac1438d92b2c944bb27aabb P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } }
W20240502 01:54:22.861851 17543 leader_election.cc:334] T 3ff84ce60ac1438d92b2c944bb27aabb P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
I20240502 01:54:22.880367 25577 leader_election.cc:288] T b671a9ceacea4a819f750fd37130be0f P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 election: Requested vote from peers 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:54:22.895093 20361 raft_consensus.cc:1261] T a390a222562f425eabb11d7f52cbe9a6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 01:54:22.895695 25813 consensus_queue.cc:1035] T a390a222562f425eabb11d7f52cbe9a6 P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:22.899583 17800 leader_election.cc:302] T 57ab576948e346cbbe6715bf53f7c665 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, bb31bceba76d466db24b9341c04900c3; no voters: dfa1152987644bd781b46c7e42a7a685
I20240502 01:54:22.900094 25849 raft_consensus.cc:2781] T 57ab576948e346cbbe6715bf53f7c665 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:54:22.900197 25849 raft_consensus.cc:482] T 57ab576948e346cbbe6715bf53f7c665 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader dfa1152987644bd781b46c7e42a7a685)
I20240502 01:54:22.900295 25849 raft_consensus.cc:3037] T 57ab576948e346cbbe6715bf53f7c665 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:22.903183 25849 raft_consensus.cc:504] T 57ab576948e346cbbe6715bf53f7c665 P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:22.904455 25849 leader_election.cc:288] T 57ab576948e346cbbe6715bf53f7c665 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 election: Requested vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
W20240502 01:54:22.905558 17802 leader_election.cc:334] T 57ab576948e346cbbe6715bf53f7c665 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
I20240502 01:54:22.908587 25849 raft_consensus.cc:482] T 1a837e7dce89462596b2b6049069ec8e P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f19cc6cb1e76460aa6711e49a5775091)
I20240502 01:54:22.908747 25849 raft_consensus.cc:504] T 1a837e7dce89462596b2b6049069ec8e P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:22.909794 25849 leader_election.cc:288] T 1a837e7dce89462596b2b6049069ec8e P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:54:22.911695 17864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf0237a4c65b495ba892871dd850bed9" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "bb31bceba76d466db24b9341c04900c3"
I20240502 01:54:22.911901 17864 raft_consensus.cc:3037] T cf0237a4c65b495ba892871dd850bed9 P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Advancing to term 3
W20240502 01:54:22.917464 17542 consensus_peers.cc:462] T 3dcd890bdd2d413e95401897e9f1bcab P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
I20240502 01:54:22.923938 17473 raft_consensus.cc:1261] T bffb870715a243d992107583d7c17f1b P 4da690c0fa13407a85c0689ce8de9ade [term 2 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:54:22.924551 24598 consensus_queue.cc:1035] T bffb870715a243d992107583d7c17f1b P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 01:54:22.929103 17802 leader_election.cc:334] T 1a837e7dce89462596b2b6049069ec8e P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
I20240502 01:54:22.930500 17330 catalog_manager.cc:5484] T 062b9d90fb0444f4be37bf09770c9caf P bb31bceba76d466db24b9341c04900c3 reported cstate change: term changed from 1 to 2, leader changed from fdc2ac372492454c85387589a91f63af (127.11.42.69) to bb31bceba76d466db24b9341c04900c3 (127.11.42.68). New cstate: current_term: 2 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240502 01:54:22.933713 17803 leader_election.cc:334] T 1a837e7dce89462596b2b6049069ec8e P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
I20240502 01:54:22.933859 17803 leader_election.cc:302] T 1a837e7dce89462596b2b6049069ec8e P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb31bceba76d466db24b9341c04900c3; no voters: f19cc6cb1e76460aa6711e49a5775091, fdc2ac372492454c85387589a91f63af
I20240502 01:54:22.934089 25604 raft_consensus.cc:2726] T 1a837e7dce89462596b2b6049069ec8e P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:54:22.934499 17411 consensus_peers.cc:485] T a390a222562f425eabb11d7f52cbe9a6 P 4da690c0fa13407a85c0689ce8de9ade -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:22.934782 20433 leader_election.cc:334] T 38bb8dd915f743f49b26ee2572871375 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
I20240502 01:54:22.934967 17864 raft_consensus.cc:2454] T cf0237a4c65b495ba892871dd850bed9 P bb31bceba76d466db24b9341c04900c3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 3.
W20240502 01:54:22.935021 20433 leader_election.cc:334] T 38bb8dd915f743f49b26ee2572871375 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
I20240502 01:54:22.935132 20433 leader_election.cc:302] T 38bb8dd915f743f49b26ee2572871375 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1; no voters: dfa1152987644bd781b46c7e42a7a685, f19cc6cb1e76460aa6711e49a5775091
I20240502 01:54:22.936327 20434 leader_election.cc:302] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, bb31bceba76d466db24b9341c04900c3; no voters: dfa1152987644bd781b46c7e42a7a685
I20240502 01:54:22.938352 17604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ab576948e346cbbe6715bf53f7c665" candidate_uuid: "bb31bceba76d466db24b9341c04900c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3fceccefba274b1dbbd8216af8b0bc29"
I20240502 01:54:22.938517 17604 raft_consensus.cc:3037] T 57ab576948e346cbbe6715bf53f7c665 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:22.941429 17604 raft_consensus.cc:2454] T 57ab576948e346cbbe6715bf53f7c665 P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb31bceba76d466db24b9341c04900c3 in term 2.
I20240502 01:54:22.942027 17800 leader_election.cc:302] T 57ab576948e346cbbe6715bf53f7c665 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29, bb31bceba76d466db24b9341c04900c3; no voters: dfa1152987644bd781b46c7e42a7a685
I20240502 01:54:22.942442 25604 raft_consensus.cc:2781] T 57ab576948e346cbbe6715bf53f7c665 P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:22.942588 25604 raft_consensus.cc:686] T 57ab576948e346cbbe6715bf53f7c665 P bb31bceba76d466db24b9341c04900c3 [term 2 LEADER]: Becoming Leader. State: Replica: bb31bceba76d466db24b9341c04900c3, State: Running, Role: LEADER
I20240502 01:54:22.942888 25604 consensus_queue.cc:237] T 57ab576948e346cbbe6715bf53f7c665 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } }
I20240502 01:54:22.944082 25577 raft_consensus.cc:2781] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:54:22.944294 25577 raft_consensus.cc:686] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1 [term 3 LEADER]: Becoming Leader. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Running, Role: LEADER
I20240502 01:54:22.944702 25577 consensus_queue.cc:237] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } }
I20240502 01:54:22.940117 17328 catalog_manager.cc:5484] T d25f2ab1a7544cec893dd02c7c89285c P 4da690c0fa13407a85c0689ce8de9ade reported cstate change: term changed from 1 to 2, leader changed from f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) to 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65). New cstate: current_term: 2 leader_uuid: "4da690c0fa13407a85c0689ce8de9ade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:22.954011 25920 raft_consensus.cc:2726] T 38bb8dd915f743f49b26ee2572871375 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:22.954012 17476 raft_consensus.cc:2932] T adc2b57af23c41aebd1258ff8b60d074 P 4da690c0fa13407a85c0689ce8de9ade [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } } }
I20240502 01:54:22.954380 25920 raft_consensus.cc:482] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f19cc6cb1e76460aa6711e49a5775091)
I20240502 01:54:22.954512 25920 raft_consensus.cc:504] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } }
I20240502 01:54:22.951115 17326 catalog_manager.cc:5484] T 68bc23294ac64c9486bffebbdb16b508 P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: term changed from 1 to 2, leader changed from f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) to 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71). New cstate: current_term: 2 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:22.955108 25920 leader_election.cc:288] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
W20240502 01:54:22.956635 17803 consensus_peers.cc:485] T 2544ba74e9d146ef963fd92a1a6f8a98 P bb31bceba76d466db24b9341c04900c3 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:22.958933 20433 leader_election.cc:334] T b671a9ceacea4a819f750fd37130be0f P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
I20240502 01:54:22.960479 20367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3e4c8f98b5e48e3ba6affc2df08943a" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" is_pre_election: true
I20240502 01:54:22.960748 20367 raft_consensus.cc:2454] T f3e4c8f98b5e48e3ba6affc2df08943a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 2.
I20240502 01:54:22.961483 20367 raft_consensus.cc:1261] T b6d89235867f42418f8205167bb29755 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240502 01:54:22.961596 20433 leader_election.cc:334] T b671a9ceacea4a819f750fd37130be0f P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
W20240502 01:54:22.961891 20433 leader_election.cc:334] T da06571e020948059fb06f65f26c831d P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
I20240502 01:54:22.963641 24918 consensus_queue.cc:1035] T b6d89235867f42418f8205167bb29755 P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:22.964056 20433 leader_election.cc:302] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 6a36c952c51542c0bf7ea3ec2857a9f9; no voters:
W20240502 01:54:22.964056 17543 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.42.69:34339, user_credentials={real_user=slave}} blocked reactor thread for 102236us
I20240502 01:54:22.964583 25920 raft_consensus.cc:2781] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 01:54:22.964685 25920 raft_consensus.cc:482] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Starting leader election (detected failure of leader f19cc6cb1e76460aa6711e49a5775091)
I20240502 01:54:22.964759 25920 raft_consensus.cc:3037] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:22.965257 25790 leader_election.cc:288] T 3ff84ce60ac1438d92b2c944bb27aabb P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:54:22.967978 25920 raft_consensus.cc:504] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } }
I20240502 01:54:22.968582 25920 leader_election.cc:288] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 3 election: Requested vote from peers 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
W20240502 01:54:22.969112 17542 leader_election.cc:334] T 3ff84ce60ac1438d92b2c944bb27aabb P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
I20240502 01:54:22.969290 17542 leader_election.cc:302] T 3ff84ce60ac1438d92b2c944bb27aabb P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29; no voters: f19cc6cb1e76460aa6711e49a5775091, fdc2ac372492454c85387589a91f63af
I20240502 01:54:22.969686 25790 raft_consensus.cc:2726] T 3ff84ce60ac1438d92b2c944bb27aabb P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:22.970223 17330 catalog_manager.cc:5484] T 164e04f7c9454c189f6907b319023d99 P bb31bceba76d466db24b9341c04900c3 reported cstate change: term changed from 1 to 2, leader changed from fdc2ac372492454c85387589a91f63af (127.11.42.69) to bb31bceba76d466db24b9341c04900c3 (127.11.42.68). New cstate: current_term: 2 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:22.970162 17326 catalog_manager.cc:5484] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: term changed from 2 to 3, leader changed from dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) to 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71). New cstate: current_term: 3 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:22.970896 17326 catalog_manager.cc:5484] T b671a9ceacea4a819f750fd37130be0f P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: term changed from 1 to 2, leader changed from f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) to 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71). New cstate: current_term: 2 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:22.971971 17330 catalog_manager.cc:5484] T 57ab576948e346cbbe6715bf53f7c665 P bb31bceba76d466db24b9341c04900c3 reported cstate change: term changed from 1 to 2, leader changed from dfa1152987644bd781b46c7e42a7a685 (127.11.42.67) to bb31bceba76d466db24b9341c04900c3 (127.11.42.68). New cstate: current_term: 2 leader_uuid: "bb31bceba76d466db24b9341c04900c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: UNKNOWN } } }
W20240502 01:54:22.972571 20433 leader_election.cc:334] T da06571e020948059fb06f65f26c831d P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
I20240502 01:54:22.973100 20433 leader_election.cc:302] T da06571e020948059fb06f65f26c831d P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1; no voters: dfa1152987644bd781b46c7e42a7a685, f19cc6cb1e76460aa6711e49a5775091
W20240502 01:54:22.974805 20433 consensus_peers.cc:485] T b29218a879e74895bfe00021ad358aa4 P 2f306549407b4bb281ccf2994035e4f1 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:22.975207 26066 raft_consensus.cc:2726] T da06571e020948059fb06f65f26c831d P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:22.976811 20367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3e4c8f98b5e48e3ba6affc2df08943a" candidate_uuid: "2f306549407b4bb281ccf2994035e4f1" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: false dest_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9"
I20240502 01:54:22.977034 20367 raft_consensus.cc:3037] T f3e4c8f98b5e48e3ba6affc2df08943a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Advancing to term 3
W20240502 01:54:22.977859 20433 leader_election.cc:334] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
W20240502 01:54:22.979406 20433 leader_election.cc:334] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
I20240502 01:54:22.980913 20367 raft_consensus.cc:2454] T f3e4c8f98b5e48e3ba6affc2df08943a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f306549407b4bb281ccf2994035e4f1 in term 3.
I20240502 01:54:22.984580 20433 leader_election.cc:302] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1, 6a36c952c51542c0bf7ea3ec2857a9f9; no voters: f19cc6cb1e76460aa6711e49a5775091
I20240502 01:54:22.984840 26066 raft_consensus.cc:2781] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:54:22.985093 26066 raft_consensus.cc:686] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1 [term 3 LEADER]: Becoming Leader. State: Replica: 2f306549407b4bb281ccf2994035e4f1, State: Running, Role: LEADER
I20240502 01:54:22.985442 26066 consensus_queue.cc:237] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } }
I20240502 01:54:22.999531 17326 catalog_manager.cc:5484] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: term changed from 2 to 3, leader changed from f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72) to 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71). New cstate: current_term: 3 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:23.001874 17473 raft_consensus.cc:1261] T 25ad701a0fec4e80b37df5dec598796a P 4da690c0fa13407a85c0689ce8de9ade [term 2 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 01:54:23.002530 26066 consensus_queue.cc:1035] T 25ad701a0fec4e80b37df5dec598796a P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 01:54:23.006325 20433 consensus_peers.cc:485] T b950e18825ac434ba44586d544667918 P 2f306549407b4bb281ccf2994035e4f1 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:23.009727 17454 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 01:54:23.019376 17543 consensus_peers.cc:485] T b6d89235867f42418f8205167bb29755 P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:23.019486 17543 consensus_peers.cc:485] T bffb870715a243d992107583d7c17f1b P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:23.026223 17803 consensus_peers.cc:485] T 062b9d90fb0444f4be37bf09770c9caf P bb31bceba76d466db24b9341c04900c3 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:23.032263 17411 consensus_peers.cc:485] T a390a222562f425eabb11d7f52cbe9a6 P 4da690c0fa13407a85c0689ce8de9ade -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:23.033572 17585 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 01:54:23.038530 20494 raft_consensus.cc:1261] T 2544ba74e9d146ef963fd92a1a6f8a98 P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 01:54:23.039253 26050 consensus_queue.cc:1035] T 2544ba74e9d146ef963fd92a1a6f8a98 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 01:54:23.044313 17412 consensus_peers.cc:485] T 5e3695e8ad0a44de9ba23ae3610aacec P 4da690c0fa13407a85c0689ce8de9ade -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:23.054869 20303 consensus_peers.cc:462] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
W20240502 01:54:23.063956 20434 consensus_peers.cc:485] T 25ad701a0fec4e80b37df5dec598796a P 2f306549407b4bb281ccf2994035e4f1 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:23.070389 20362 raft_consensus.cc:1261] T b950e18825ac434ba44586d544667918 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:54:23.071292 25577 consensus_queue.cc:1035] T b950e18825ac434ba44586d544667918 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:54:23.089519 20302 consensus_peers.cc:485] T ad067712087e4f78885bd8e1cf102694 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:23.097505 20477 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240502 01:54:23.098083 17802 consensus_peers.cc:485] T 2544ba74e9d146ef963fd92a1a6f8a98 P bb31bceba76d466db24b9341c04900c3 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:23.104171 20433 consensus_peers.cc:485] T 25ad701a0fec4e80b37df5dec598796a P 2f306549407b4bb281ccf2994035e4f1 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:23.104808 17606 raft_consensus.cc:1261] T bbf05e3b3ad44354be7600c3c07756fd P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 01:54:23.105306 17604 raft_consensus.cc:1261] T 164e04f7c9454c189f6907b319023d99 P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:54:23.105763 17473 raft_consensus.cc:1261] T b29218a879e74895bfe00021ad358aa4 P 4da690c0fa13407a85c0689ce8de9ade [term 2 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:54:23.107693 25603 consensus_queue.cc:1035] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:23.108592 25849 consensus_queue.cc:1035] T 164e04f7c9454c189f6907b319023d99 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:23.112712 17846 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 01:54:23.119303 17605 raft_consensus.cc:1261] T 7e9be84ef6a043818f166f6b47898855 P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:54:23.121115 25577 consensus_queue.cc:1035] T 7e9be84ef6a043818f166f6b47898855 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 01:54:23.133096 17412 consensus_peers.cc:485] T 0615786e80314766892988f3cff6a74d P 4da690c0fa13407a85c0689ce8de9ade -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:54:23.141782 25920 consensus_queue.cc:1035] T b29218a879e74895bfe00021ad358aa4 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:23.143322 20496 raft_consensus.cc:3037] T bbf05e3b3ad44354be7600c3c07756fd P 2f306549407b4bb281ccf2994035e4f1 [term 1 LEARNER]: Advancing to term 2
I20240502 01:54:23.146247 20496 raft_consensus.cc:1261] T bbf05e3b3ad44354be7600c3c07756fd P 2f306549407b4bb281ccf2994035e4f1 [term 2 LEARNER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 01:54:23.148880 25358 consensus_queue.cc:1035] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: NON_VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:54:23.151877 20345 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 01:54:23.158528 25848 raft_consensus.cc:1050] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9: attempting to promote NON_VOTER 2f306549407b4bb281ccf2994035e4f1 to VOTER
W20240502 01:54:23.158697 25848 raft_consensus.cc:1054] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9: Unable to promote non-voter 2f306549407b4bb281ccf2994035e4f1: Illegal state: Leader has not yet committed an operation in its own term
W20240502 01:54:23.169641 20303 consensus_peers.cc:485] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:23.172801 17864 raft_consensus.cc:1261] T 75bb4e19cf57405fabcb532b55a29ced P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:54:23.173454 25813 consensus_queue.cc:1035] T 75bb4e19cf57405fabcb532b55a29ced P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 01:54:23.175266 20433 consensus_peers.cc:485] T 7e9be84ef6a043818f166f6b47898855 P 2f306549407b4bb281ccf2994035e4f1 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:23.177208 20433 consensus_peers.cc:485] T 68bc23294ac64c9486bffebbdb16b508 P 2f306549407b4bb281ccf2994035e4f1 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:23.185067 20496 raft_consensus.cc:1261] T 5e3695e8ad0a44de9ba23ae3610aacec P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:54:23.185727 25813 consensus_queue.cc:1035] T 5e3695e8ad0a44de9ba23ae3610aacec P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:54:23.195809 20434 consensus_peers.cc:485] T 7b2eae408b9843acaa61e698a3e44cff P 2f306549407b4bb281ccf2994035e4f1 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:54:23.196959 17475 raft_consensus.cc:1261] T 062b9d90fb0444f4be37bf09770c9caf P 4da690c0fa13407a85c0689ce8de9ade [term 2 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:54:23.197923 26050 consensus_queue.cc:1035] T 062b9d90fb0444f4be37bf09770c9caf P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 01:54:23.199795 17803 consensus_peers.cc:485] T 164e04f7c9454c189f6907b319023d99 P bb31bceba76d466db24b9341c04900c3 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:23.201035 17411 consensus_peers.cc:485] T 75bb4e19cf57405fabcb532b55a29ced P 4da690c0fa13407a85c0689ce8de9ade -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:23.213037 17607 raft_consensus.cc:1261] T 68bc23294ac64c9486bffebbdb16b508 P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:54:23.213687 25920 consensus_queue.cc:1035] T 68bc23294ac64c9486bffebbdb16b508 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:23.248661 25358 raft_consensus.cc:1050] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9: attempting to promote NON_VOTER 2f306549407b4bb281ccf2994035e4f1 to VOTER
I20240502 01:54:23.249493 25358 consensus_queue.cc:237] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } }
I20240502 01:54:23.257952 20497 raft_consensus.cc:1261] T bbf05e3b3ad44354be7600c3c07756fd P 2f306549407b4bb281ccf2994035e4f1 [term 2 LEARNER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:54:23.258646 24817 consensus_queue.cc:1035] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:23.261588 17864 raft_consensus.cc:1261] T d25f2ab1a7544cec893dd02c7c89285c P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Refusing update from remote peer 4da690c0fa13407a85c0689ce8de9ade: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:54:23.261855 17607 raft_consensus.cc:1261] T bbf05e3b3ad44354be7600c3c07756fd P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:54:23.262145 25813 consensus_queue.cc:1035] T d25f2ab1a7544cec893dd02c7c89285c P 4da690c0fa13407a85c0689ce8de9ade [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:23.262997 25603 consensus_queue.cc:1035] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:54:23.265441 20303 consensus_peers.cc:485] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:23.273145 24817 raft_consensus.cc:2932] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:23.290599 20494 raft_consensus.cc:2932] T bbf05e3b3ad44354be7600c3c07756fd P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:23.299870 17329 catalog_manager.cc:5484] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9 reported cstate change: config changed from index 2 to 4, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:23.302525 20494 raft_consensus.cc:1261] T ad067712087e4f78885bd8e1cf102694 P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:54:23.303905 25848 consensus_queue.cc:1035] T ad067712087e4f78885bd8e1cf102694 P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:54:23.298017 17605 raft_consensus.cc:2932] T bbf05e3b3ad44354be7600c3c07756fd P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 2f306549407b4bb281ccf2994035e4f1 (127.11.42.71) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } }
I20240502 01:54:23.309999 17607 raft_consensus.cc:1261] T b671a9ceacea4a819f750fd37130be0f P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:54:23.315383 24959 consensus_queue.cc:1035] T b671a9ceacea4a819f750fd37130be0f P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:54:23.329121 17542 consensus_peers.cc:485] T 975f253ff03447ff96fe84169c05097e P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:23.333547 20433 consensus_peers.cc:485] T b671a9ceacea4a819f750fd37130be0f P 2f306549407b4bb281ccf2994035e4f1 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:23.349032 17411 consensus_peers.cc:485] T d25f2ab1a7544cec893dd02c7c89285c P 4da690c0fa13407a85c0689ce8de9ade -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:23.352599 20433 consensus_peers.cc:485] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:23.358516 17607 raft_consensus.cc:1261] T 57ab576948e346cbbe6715bf53f7c665 P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 FOLLOWER]: Refusing update from remote peer bb31bceba76d466db24b9341c04900c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:54:23.359131 26050 consensus_queue.cc:1035] T 57ab576948e346cbbe6715bf53f7c665 P bb31bceba76d466db24b9341c04900c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:54:23.366945 17542 consensus_peers.cc:462] T 3dcd890bdd2d413e95401897e9f1bcab P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
I20240502 01:54:23.373106 17607 raft_consensus.cc:1261] T 074a08b2b64640eb9784b6a23921ce8d P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 FOLLOWER]: Refusing update from remote peer 6a36c952c51542c0bf7ea3ec2857a9f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:54:23.373854 25603 consensus_queue.cc:1035] T 074a08b2b64640eb9784b6a23921ce8d P 6a36c952c51542c0bf7ea3ec2857a9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 01:54:23.375880 17802 consensus_peers.cc:485] T 57ab576948e346cbbe6715bf53f7c665 P bb31bceba76d466db24b9341c04900c3 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:23.391201 20302 consensus_peers.cc:485] T 074a08b2b64640eb9784b6a23921ce8d P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:23.432092 17475 raft_consensus.cc:1261] T 975f253ff03447ff96fe84169c05097e P 4da690c0fa13407a85c0689ce8de9ade [term 2 FOLLOWER]: Refusing update from remote peer 3fceccefba274b1dbbd8216af8b0bc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:54:23.432597 24598 consensus_queue.cc:1035] T 975f253ff03447ff96fe84169c05097e P 3fceccefba274b1dbbd8216af8b0bc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:23.450558 17607 raft_consensus.cc:3037] T f3e4c8f98b5e48e3ba6affc2df08943a P 3fceccefba274b1dbbd8216af8b0bc29 [term 2 LEARNER]: Advancing to term 3
I20240502 01:54:23.452641 17607 raft_consensus.cc:1261] T f3e4c8f98b5e48e3ba6affc2df08943a P 3fceccefba274b1dbbd8216af8b0bc29 [term 3 LEARNER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240502 01:54:23.453423 24959 consensus_queue.cc:1035] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: NON_VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:23.457557 26066 raft_consensus.cc:1050] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1: attempting to promote NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 to VOTER
W20240502 01:54:23.457680 26066 raft_consensus.cc:1054] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1: Unable to promote non-voter 3fceccefba274b1dbbd8216af8b0bc29: Illegal state: Leader has not yet committed an operation in its own term
I20240502 01:54:23.506110 17864 raft_consensus.cc:1261] T cf0237a4c65b495ba892871dd850bed9 P bb31bceba76d466db24b9341c04900c3 [term 3 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240502 01:54:23.506569 24959 consensus_queue.cc:1035] T cf0237a4c65b495ba892871dd850bed9 P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:23.544008 20361 raft_consensus.cc:1261] T f3e4c8f98b5e48e3ba6affc2df08943a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 3 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240502 01:54:23.544943 26066 consensus_queue.cc:1035] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 01:54:23.545732 20433 consensus_peers.cc:485] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:23.550583 25920 raft_consensus.cc:1050] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1: attempting to promote NON_VOTER 3fceccefba274b1dbbd8216af8b0bc29 to VOTER
I20240502 01:54:23.551268 25920 consensus_queue.cc:237] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.7, Last appended by leader: 6, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } }
I20240502 01:54:23.554013 17607 raft_consensus.cc:1261] T f3e4c8f98b5e48e3ba6affc2df08943a P 3fceccefba274b1dbbd8216af8b0bc29 [term 3 LEARNER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20240502 01:54:23.554718 26066 consensus_queue.cc:1035] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:23.556155 20362 raft_consensus.cc:1261] T f3e4c8f98b5e48e3ba6affc2df08943a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 3 FOLLOWER]: Refusing update from remote peer 2f306549407b4bb281ccf2994035e4f1: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
W20240502 01:54:23.556826 20433 consensus_peers.cc:485] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:23.557255 24959 consensus_queue.cc:1035] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 01:54:23.561617 20302 consensus_peers.cc:485] T 0399c9d1d7b2408292dc78c185605ee8 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:54:23.563494 26066 raft_consensus.cc:2932] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1 [term 3 LEADER]: Committing config change with OpId 3.8: config changed from index 6 to 8, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:23.567312 17607 raft_consensus.cc:2932] T f3e4c8f98b5e48e3ba6affc2df08943a P 3fceccefba274b1dbbd8216af8b0bc29 [term 3 FOLLOWER]: Committing config change with OpId 3.8: config changed from index 6 to 8, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:23.569561 20367 raft_consensus.cc:2932] T f3e4c8f98b5e48e3ba6affc2df08943a P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 3 FOLLOWER]: Committing config change with OpId 3.8: config changed from index 6 to 8, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } } }
I20240502 01:54:23.573467 17326 catalog_manager.cc:5484] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1 reported cstate change: config changed from index 6 to 8, 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "2f306549407b4bb281ccf2994035e4f1" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 01:54:23.583721 17412 consensus_peers.cc:485] T 277c6ae5c8dc4f8eae261bf06890d793 P 4da690c0fa13407a85c0689ce8de9ade -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:54:23.586911 20303 consensus_peers.cc:462] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
W20240502 01:54:23.599947 17802 consensus_peers.cc:485] T 5124abbb09db4be0955c6bc831af08fa P bb31bceba76d466db24b9341c04900c3 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:54:23.709182 17802 consensus_peers.cc:485] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:54:23.723811 17803 consensus_peers.cc:485] T 59c288e7279f4119afb6c3bf860e6dc3 P bb31bceba76d466db24b9341c04900c3 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:54:23.741745 17803 consensus_peers.cc:485] T f0b3862e504d4b43a60477e5efef29b1 P bb31bceba76d466db24b9341c04900c3 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:54:23.752188 17542 consensus_peers.cc:485] T fa4d38dc9b674a9b90d4793dad2687ec P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:54:23.778115 17543 consensus_peers.cc:485] T 5152c002eadd4d54ab10a093bce45f17 P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:54:23.782387 17411 consensus_peers.cc:485] T 9642b9b37a784cf881bfc4971d8c9cb6 P 4da690c0fa13407a85c0689ce8de9ade -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:1203: Failure
Value of: s.ok()
Actual: false
Expected: true
Runtime error: /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 1
Maximum Replica Count | 43
Average Replica Count | 21.750000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 1
Maximum | 3
Average | 2.900000
stderr: W20240502 01:53:47.627740 20682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:47.628194 20682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:47.636920 20682 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
E20240502 01:53:48.068609 20776 ksck_remote.cc:634] unable to open table rebalancer_dynamic_table_new_name: Not found: the table does not exist: table_name: "rebalancer_dynamic_table_new_name"
I20240502 01:53:48.757758 20682 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 01:53:49.840019 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9631b34b6ef449edb5476c988aed9d7a: could not find any suitable replica to move from server dfa1152987644bd781b46c7e42a7a685 to server 2f306549407b4bb281ccf2994035e4f1
W20240502 01:53:49.841138 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9631b34b6ef449edb5476c988aed9d7a: could not find any suitable replica to move from server 4da690c0fa13407a85c0689ce8de9ade to server 6a36c952c51542c0bf7ea3ec2857a9f9
W20240502 01:53:49.841586 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b996bd60793c4d3c83bf34f4989da909: could not find any suitable replica to move from server 3fceccefba274b1dbbd8216af8b0bc29 to server 6a36c952c51542c0bf7ea3ec2857a9f9
W20240502 01:53:49.841738 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2444c341261d4eb29c484622291d0517: could not find any suitable replica to move from server fdc2ac372492454c85387589a91f63af to server dfa1152987644bd781b46c7e42a7a685
W20240502 01:53:49.841894 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 128b9c63621549098646906560fbc531: could not find any suitable replica to move from server dfa1152987644bd781b46c7e42a7a685 to server fdc2ac372492454c85387589a91f63af
W20240502 01:53:49.842164 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9631b34b6ef449edb5476c988aed9d7a: could not find any suitable replica to move from server dfa1152987644bd781b46c7e42a7a685 to server f19cc6cb1e76460aa6711e49a5775091
W20240502 01:53:49.842320 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c6474d0c5256497eb6fa5844f3530b62: could not find any suitable replica to move from server 3fceccefba274b1dbbd8216af8b0bc29 to server 2f306549407b4bb281ccf2994035e4f1
W20240502 01:53:49.842555 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b69ea3782c3c43dfa2760155beffc87b: could not find any suitable replica to move from server 3fceccefba274b1dbbd8216af8b0bc29 to server 6a36c952c51542c0bf7ea3ec2857a9f9
W20240502 01:53:49.842736 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f27ebc659702427c996fda2492c7e5d0: could not find any suitable replica to move from server dfa1152987644bd781b46c7e42a7a685 to server f19cc6cb1e76460aa6711e49a5775091
W20240502 01:53:49.842972 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2444c341261d4eb29c484622291d0517: could not find any suitable replica to move from server 6a36c952c51542c0bf7ea3ec2857a9f9 to server bb31bceba76d466db24b9341c04900c3
W20240502 01:53:49.843155 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c6c3a53b24874f31aa38118254253dcb: could not find any suitable replica to move from server 3fceccefba274b1dbbd8216af8b0bc29 to server 2f306549407b4bb281ccf2994035e4f1
W20240502 01:53:49.843297 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 21fe1df2eeed4efd801d10edf592ecce: could not find any suitable replica to move from server 3fceccefba274b1dbbd8216af8b0bc29 to server fdc2ac372492454c85387589a91f63af
W20240502 01:53:49.843422 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5e2315325f6c4e7f8834471d07555cf5: could not find any suitable replica to move from server dfa1152987644bd781b46c7e42a7a685 to server f19cc6cb1e76460aa6711e49a5775091
W20240502 01:53:49.843523 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9631b34b6ef449edb5476c988aed9d7a: could not find any suitable replica to move from server 4da690c0fa13407a85c0689ce8de9ade to server fdc2ac372492454c85387589a91f63af
W20240502 01:53:49.843621 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e431affbb02a4744b67a522521a3ee04: could not find any suitable replica to move from server bb31bceba76d466db24b9341c04900c3 to server f19cc6cb1e76460aa6711e49a5775091
W20240502 01:53:49.843914 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c6474d0c5256497eb6fa5844f3530b62: could not find any suitable replica to move from server bb31bceba76d466db24b9341c04900c3 to server 6a36c952c51542c0bf7ea3ec2857a9f9
W20240502 01:53:49.844192 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a07db144a5294a4dbfc503167983593e: could not find any suitable replica to move from server bb31bceba76d466db24b9341c04900c3 to server f19cc6cb1e76460aa6711e49a5775091
W20240502 01:53:49.844350 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1567095c8de34d6a98c56d3bb926b668: could not find any suitable replica to move from server 3fceccefba274b1dbbd8216af8b0bc29 to server f19cc6cb1e76460aa6711e49a5775091
W20240502 01:53:49.844472 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ffe6d29170f24ad48433354f548b9b2a: could not find any suitable replica to move from server dfa1152987644bd781b46c7e42a7a685 to server 2f306549407b4bb281ccf2994035e4f1
W20240502 01:53:49.844621 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b69ea3782c3c43dfa2760155beffc87b: could not find any suitable replica to move from server 4da690c0fa13407a85c0689ce8de9ade to server 2f306549407b4bb281ccf2994035e4f1
W20240502 01:53:49.844730 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa75b6d8fec44d1998a1c84ded9c9998: could not find any suitable replica to move from server 3fceccefba274b1dbbd8216af8b0bc29 to server fdc2ac372492454c85387589a91f63af
W20240502 01:53:49.844859 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dbf97af96f07462286eb1fabeddf25fa: could not find any suitable replica to move from server dfa1152987644bd781b46c7e42a7a685 to server 6a36c952c51542c0bf7ea3ec2857a9f9
W20240502 01:53:49.845103 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3f381b46ce4c4598900dada557836f91: could not find any suitable replica to move from server 4da690c0fa13407a85c0689ce8de9ade to server 6a36c952c51542c0bf7ea3ec2857a9f9
W20240502 01:53:49.845234 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b996bd60793c4d3c83bf34f4989da909: could not find any suitable replica to move from server bb31bceba76d466db24b9341c04900c3 to server f19cc6cb1e76460aa6711e49a5775091
W20240502 01:53:49.845366 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b69eb497d10340fea8df98ad03a1e465: could not find any suitable replica to move from server bb31bceba76d466db24b9341c04900c3 to server 6a36c952c51542c0bf7ea3ec2857a9f9
W20240502 01:53:49.845525 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c6c3a53b24874f31aa38118254253dcb: could not find any suitable replica to move from server dfa1152987644bd781b46c7e42a7a685 to server f19cc6cb1e76460aa6711e49a5775091
W20240502 01:53:49.845690 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2c26265a42364c878d9c982a6cbb7f48: could not find any suitable replica to move from server 3fceccefba274b1dbbd8216af8b0bc29 to server 2f306549407b4bb281ccf2994035e4f1
W20240502 01:53:49.845839 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e431affbb02a4744b67a522521a3ee04: could not find any suitable replica to move from server dfa1152987644bd781b46c7e42a7a685 to server fdc2ac372492454c85387589a91f63af
W20240502 01:53:49.845952 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1567095c8de34d6a98c56d3bb926b668: could not find any suitable replica to move from server bb31bceba76d466db24b9341c04900c3 to server fdc2ac372492454c85387589a91f63af
W20240502 01:53:49.846058 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9631b34b6ef449edb5476c988aed9d7a: could not find any suitable replica to move from server 3fceccefba274b1dbbd8216af8b0bc29 to server 2f306549407b4bb281ccf2994035e4f1
I20240502 01:53:50.169574 20682 rebalancer_tool.cc:1074] tablet 377e804dbeed4913a7d435cfdd1e2d8e: '4da690c0fa13407a85c0689ce8de9ade' -> '2f306549407b4bb281ccf2994035e4f1' move scheduled
I20240502 01:53:50.305611 20682 rebalancer_tool.cc:1074] tablet 52430bbf347a4c5191edb23e1aa5d385: '3fceccefba274b1dbbd8216af8b0bc29' -> 'fdc2ac372492454c85387589a91f63af' move scheduled
I20240502 01:53:50.340363 20682 rebalancer_tool.cc:1074] tablet 5b092dd1cc004c26a33e17c28af4d3ba: 'dfa1152987644bd781b46c7e42a7a685' -> '6a36c952c51542c0bf7ea3ec2857a9f9' move scheduled
I20240502 01:53:50.343293 20682 rebalancer_tool.cc:1084] tablet 4cd692f7f2ca4dce901b7b2d2dc3a726: 'bb31bceba76d466db24b9341c04900c3' -> 'f19cc6cb1e76460aa6711e49a5775091' move ignored: Not found: 4cd692f7f2ca4dce901b7b2d2dc3a726: tablet not found
I20240502 01:53:50.343457 20682 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:53:51.350365 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 21fe1df2eeed4efd801d10edf592ecce: could not find any suitable replica to move from server 3fceccefba274b1dbbd8216af8b0bc29 to server f19cc6cb1e76460aa6711e49a5775091
W20240502 01:53:51.350476 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f27ebc659702427c996fda2492c7e5d0: could not find any suitable replica to move from server dfa1152987644bd781b46c7e42a7a685 to server 6a36c952c51542c0bf7ea3ec2857a9f9
W20240502 01:53:51.350569 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dbf97af96f07462286eb1fabeddf25fa: could not find any suitable replica to move from server 3fceccefba274b1dbbd8216af8b0bc29 to server 2f306549407b4bb281ccf2994035e4f1
W20240502 01:53:51.350661 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b69eb497d10340fea8df98ad03a1e465: could not find any suitable replica to move from server dfa1152987644bd781b46c7e42a7a685 to server 2f306549407b4bb281ccf2994035e4f1
W20240502 01:53:51.350783 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 128b9c63621549098646906560fbc531: could not find any suitable replica to move from server 3fceccefba274b1dbbd8216af8b0bc29 to server 6a36c952c51542c0bf7ea3ec2857a9f9
W20240502 01:53:51.350845 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a07db144a5294a4dbfc503167983593e: could not find any suitable replica to move from server 3fceccefba274b1dbbd8216af8b0bc29 to server f19cc6cb1e76460aa6711e49a5775091
W20240502 01:53:51.350993 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1567095c8de34d6a98c56d3bb926b668: could not find any suitable replica to move from server bb31bceba76d466db24b9341c04900c3 to server 2f306549407b4bb281ccf2994035e4f1
W20240502 01:53:51.351188 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5e2315325f6c4e7f8834471d07555cf5: could not find any suitable replica to move from server dfa1152987644bd781b46c7e42a7a685 to server f19cc6cb1e76460aa6711e49a5775091
W20240502 01:53:51.351333 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b996bd60793c4d3c83bf34f4989da909: could not find any suitable replica to move from server 3fceccefba274b1dbbd8216af8b0bc29 to server fdc2ac372492454c85387589a91f63af
W20240502 01:53:51.351488 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 128b9c63621549098646906560fbc531: could not find any suitable replica to move from server 4da690c0fa13407a85c0689ce8de9ade to server f19cc6cb1e76460aa6711e49a5775091
W20240502 01:53:51.351588 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e431affbb02a4744b67a522521a3ee04: could not find any suitable replica to move from server 3fceccefba274b1dbbd8216af8b0bc29 to server f19cc6cb1e76460aa6711e49a5775091
W20240502 01:53:51.351655 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ffe6d29170f24ad48433354f548b9b2a: could not find any suitable replica to move from server dfa1152987644bd781b46c7e42a7a685 to server 6a36c952c51542c0bf7ea3ec2857a9f9
W20240502 01:53:51.351763 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa75b6d8fec44d1998a1c84ded9c9998: could not find any suitable replica to move from server bb31bceba76d466db24b9341c04900c3 to server 6a36c952c51542c0bf7ea3ec2857a9f9
W20240502 01:53:51.351861 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c6474d0c5256497eb6fa5844f3530b62: could not find any suitable replica to move from server 3fceccefba274b1dbbd8216af8b0bc29 to server f19cc6cb1e76460aa6711e49a5775091
W20240502 01:53:51.351998 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dca50b50b02e4a8c8b90e0de6f305bd8: could not find any suitable replica to move from server fdc2ac372492454c85387589a91f63af to server dfa1152987644bd781b46c7e42a7a685
W20240502 01:53:51.352250 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c6c3a53b24874f31aa38118254253dcb: could not find any suitable replica to move from server bb31bceba76d466db24b9341c04900c3 to server f19cc6cb1e76460aa6711e49a5775091
W20240502 01:53:51.352381 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 128b9c63621549098646906560fbc531: could not find any suitable replica to move from server bb31bceba76d466db24b9341c04900c3 to server 2f306549407b4bb281ccf2994035e4f1
W20240502 01:53:51.352473 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dbf97af96f07462286eb1fabeddf25fa: could not find any suitable replica to move from server dfa1152987644bd781b46c7e42a7a685 to server 6a36c952c51542c0bf7ea3ec2857a9f9
W20240502 01:53:51.352581 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a07db144a5294a4dbfc503167983593e: could not find any suitable replica to move from server 4da690c0fa13407a85c0689ce8de9ade to server 2f306549407b4bb281ccf2994035e4f1
I20240502 01:53:51.423514 20682 rebalancer_tool.cc:1074] tablet 31aa6a26f64f49368ad43cc97c66f2f1: 'f19cc6cb1e76460aa6711e49a5775091' -> 'bb31bceba76d466db24b9341c04900c3' move scheduled
I20240502 01:53:51.424975 20682 rebalancer_tool.cc:1084] tablet a12dd8d5f6164fb8a77d6a197aa0eb7e: '4da690c0fa13407a85c0689ce8de9ade' -> '2f306549407b4bb281ccf2994035e4f1' move ignored: Not found: a12dd8d5f6164fb8a77d6a197aa0eb7e: tablet not found
I20240502 01:53:51.425118 20682 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:53:52.336299 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3e245c8534e84c55922b7cd77670a4a3: could not find any suitable replica to move from server 2f306549407b4bb281ccf2994035e4f1 to server dfa1152987644bd781b46c7e42a7a685
W20240502 01:53:52.336427 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b996bd60793c4d3c83bf34f4989da909: could not find any suitable replica to move from server 3fceccefba274b1dbbd8216af8b0bc29 to server 6a36c952c51542c0bf7ea3ec2857a9f9
W20240502 01:53:52.336505 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dbf97af96f07462286eb1fabeddf25fa: could not find any suitable replica to move from server 3fceccefba274b1dbbd8216af8b0bc29 to server f19cc6cb1e76460aa6711e49a5775091
W20240502 01:53:52.336624 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2444c341261d4eb29c484622291d0517: could not find any suitable replica to move from server f19cc6cb1e76460aa6711e49a5775091 to server dfa1152987644bd781b46c7e42a7a685
W20240502 01:53:52.336722 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ffe6d29170f24ad48433354f548b9b2a: could not find any suitable replica to move from server dfa1152987644bd781b46c7e42a7a685 to server fdc2ac372492454c85387589a91f63af
W20240502 01:53:52.336794 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c6474d0c5256497eb6fa5844f3530b62: could not find any suitable replica to move from server 3fceccefba274b1dbbd8216af8b0bc29 to server fdc2ac372492454c85387589a91f63af
W20240502 01:53:52.336900 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fa75b6d8fec44d1998a1c84ded9c9998: could not find any suitable replica to move from server bb31bceba76d466db24b9341c04900c3 to server f19cc6cb1e76460aa6711e49a5775091
W20240502 01:53:52.337143 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c6c3a53b24874f31aa38118254253dcb: could not find any suitable replica to move from server 3fceccefba274b1dbbd8216af8b0bc29 to server fdc2ac372492454c85387589a91f63af
W20240502 01:53:52.337400 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c6474d0c5256497eb6fa5844f3530b62: could not find any suitable replica to move from server bb31bceba76d466db24b9341c04900c3 to server 2f306549407b4bb281ccf2994035e4f1
W20240502 01:53:52.337488 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dca50b50b02e4a8c8b90e0de6f305bd8: could not find any suitable replica to move from server 6a36c952c51542c0bf7ea3ec2857a9f9 to server 3fceccefba274b1dbbd8216af8b0bc29
W20240502 01:53:52.337672 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e431affbb02a4744b67a522521a3ee04: could not find any suitable replica to move from server 3fceccefba274b1dbbd8216af8b0bc29 to server fdc2ac372492454c85387589a91f63af
W20240502 01:53:52.337769 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1567095c8de34d6a98c56d3bb926b668: could not find any suitable replica to move from server bb31bceba76d466db24b9341c04900c3 to server 6a36c952c51542c0bf7ea3ec2857a9f9
W20240502 01:53:52.337857 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 21fe1df2eeed4efd801d10edf592ecce: could not find any suitable replica to move from server 4da690c0fa13407a85c0689ce8de9ade to server fdc2ac372492454c85387589a91f63af
W20240502 01:53:52.337920 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f27ebc659702427c996fda2492c7e5d0: could not find any suitable replica to move from server dfa1152987644bd781b46c7e42a7a685 to server 6a36c952c51542c0bf7ea3ec2857a9f9
W20240502 01:53:52.338028 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5e2315325f6c4e7f8834471d07555cf5: could not find any suitable replica to move from server bb31bceba76d466db24b9341c04900c3 to server fdc2ac372492454c85387589a91f63af
W20240502 01:53:52.338089 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b69eb497d10340fea8df98ad03a1e465: could not find any suitable replica to move from server 4da690c0fa13407a85c0689ce8de9ade to server 2f306549407b4bb281ccf2994035e4f1
W20240502 01:53:52.338161 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1567095c8de34d6a98c56d3bb926b668: could not find any suitable replica to move from server 3fceccefba274b1dbbd8216af8b0bc29 to server f19cc6cb1e76460aa6711e49a5775091
W20240502 01:53:52.338209 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e33e661e202249fca162bb1e059c31a8: could not find any suitable replica to move from server fdc2ac372492454c85387589a91f63af to server f19cc6cb1e76460aa6711e49a5775091
W20240502 01:53:52.338276 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2da12dd1e8bc4159bc3e1040e3db33fe: could not find any suitable replica to move from server 6a36c952c51542c0bf7ea3ec2857a9f9 to server 2f306549407b4bb281ccf2994035e4f1
I20240502 01:53:52.383502 20682 rebalancer_tool.cc:1074] tablet 62887e332443457f8a5b41a982354dc4: '2f306549407b4bb281ccf2994035e4f1' -> 'bb31bceba76d466db24b9341c04900c3' move scheduled
I20240502 01:53:52.514429 20682 rebalancer_tool.cc:1074] tablet 4bb42883b1a0475b863d008bd01fa70a: 'dfa1152987644bd781b46c7e42a7a685' -> 'f19cc6cb1e76460aa6711e49a5775091' move scheduled
I20240502 01:53:52.570272 20682 rebalancer_tool.cc:1074] tablet 52ac2775e02844d195677035796c3847: '4da690c0fa13407a85c0689ce8de9ade' -> '6a36c952c51542c0bf7ea3ec2857a9f9' move scheduled
I20240502 01:53:52.603921 20682 rebalancer_tool.cc:1074] tablet 975f253ff03447ff96fe84169c05097e: 'fdc2ac372492454c85387589a91f63af' -> '3fceccefba274b1dbbd8216af8b0bc29' move scheduled
I20240502 01:53:52.634542 20682 rebalancer_tool.cc:1074] tablet 6aacf7c4f19546f581c9a54ad171916b: 'bb31bceba76d466db24b9341c04900c3' -> '2f306549407b4bb281ccf2994035e4f1' move scheduled
I20240502 01:53:52.688340 20682 rebalancer_tool.cc:1074] tablet 9642b9b37a784cf881bfc4971d8c9cb6: 'dfa1152987644bd781b46c7e42a7a685' -> 'f19cc6cb1e76460aa6711e49a5775091' move scheduled
I20240502 01:53:52.885509 20682 rebalancer_tool.cc:1074] tablet e42126e16cda4d908b00e2e4a6807d6b: '4da690c0fa13407a85c0689ce8de9ade' -> 'bb31bceba76d466db24b9341c04900c3' move scheduled
I20240502 01:53:52.915084 20682 rebalancer_tool.cc:1074] tablet 65bdab2c9fba48a6828a864278319cc0: '3fceccefba274b1dbbd8216af8b0bc29' -> '6a36c952c51542c0bf7ea3ec2857a9f9' move scheduled
I20240502 01:53:52.943523 20682 rebalancer_tool.cc:1074] tablet ebb9bca9f1bd41b0857939909b1110e4: 'fdc2ac372492454c85387589a91f63af' -> '2f306549407b4bb281ccf2994035e4f1' move scheduled
I20240502 01:53:52.979936 20682 rebalancer_tool.cc:1074] tablet 1f01949cf70440b38e59a5020fd48c44: 'dfa1152987644bd781b46c7e42a7a685' -> 'f19cc6cb1e76460aa6711e49a5775091' move scheduled
I20240502 01:53:52.989219 20682 rebalancer_tool.cc:1084] tablet 5c56837c0a2e49f69fab0aaf01d113bd: '4da690c0fa13407a85c0689ce8de9ade' -> '2f306549407b4bb281ccf2994035e4f1' move ignored: Not found: 5c56837c0a2e49f69fab0aaf01d113bd: tablet not found
I20240502 01:53:52.989370 20682 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:53:54.744465 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 21fe1df2eeed4efd801d10edf592ecce: could not find any suitable replica to move from server 3fceccefba274b1dbbd8216af8b0bc29 to server f19cc6cb1e76460aa6711e49a5775091
W20240502 01:53:54.744598 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c6474d0c5256497eb6fa5844f3530b62: could not find any suitable replica to move from server bb31bceba76d466db24b9341c04900c3 to server 2f306549407b4bb281ccf2994035e4f1
W20240502 01:53:54.744810 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2444c341261d4eb29c484622291d0517: could not find any suitable replica to move from server f19cc6cb1e76460aa6711e49a5775091 to server bb31bceba76d466db24b9341c04900c3
W20240502 01:53:54.744972 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 96aabcc026484fffa9ea0c0dc8d37340: could not find any suitable replica to move from server f19cc6cb1e76460aa6711e49a5775091 to server dfa1152987644bd781b46c7e42a7a685
W20240502 01:53:54.745095 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b69eb497d10340fea8df98ad03a1e465: could not find any suitable replica to move from server dfa1152987644bd781b46c7e42a7a685 to server 2f306549407b4bb281ccf2994035e4f1
W20240502 01:53:54.745388 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b996bd60793c4d3c83bf34f4989da909: could not find any suitable replica to move from server bb31bceba76d466db24b9341c04900c3 to server 6a36c952c51542c0bf7ea3ec2857a9f9
W20240502 01:53:54.745527 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c6c3a53b24874f31aa38118254253dcb: could not find any suitable replica to move from server 3fceccefba274b1dbbd8216af8b0bc29 to server fdc2ac372492454c85387589a91f63af
W20240502 01:53:54.745620 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dca50b50b02e4a8c8b90e0de6f305bd8: could not find any suitable replica to move from server 6a36c952c51542c0bf7ea3ec2857a9f9 to server dfa1152987644bd781b46c7e42a7a685
W20240502 01:53:54.745705 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dbf97af96f07462286eb1fabeddf25fa: could not find any suitable replica to move from server dfa1152987644bd781b46c7e42a7a685 to server 2f306549407b4bb281ccf2994035e4f1
W20240502 01:53:54.745828 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e431affbb02a4744b67a522521a3ee04: could not find any suitable replica to move from server dfa1152987644bd781b46c7e42a7a685 to server 6a36c952c51542c0bf7ea3ec2857a9f9
W20240502 01:53:54.745921 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ffe6d29170f24ad48433354f548b9b2a: could not find any suitable replica to move from server bb31bceba76d466db24b9341c04900c3 to server fdc2ac372492454c85387589a91f63af
W20240502 01:53:54.746037 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1567095c8de34d6a98c56d3bb926b668: could not find any suitable replica to move from server 3fceccefba274b1dbbd8216af8b0bc29 to server 6a36c952c51542c0bf7ea3ec2857a9f9
W20240502 01:53:54.746187 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5e2315325f6c4e7f8834471d07555cf5: could not find any suitable replica to move from server 4da690c0fa13407a85c0689ce8de9ade to server 6a36c952c51542c0bf7ea3ec2857a9f9
W20240502 01:53:54.746340 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 21fe1df2eeed4efd801d10edf592ecce: could not find any suitable replica to move from server dfa1152987644bd781b46c7e42a7a685 to server f19cc6cb1e76460aa6711e49a5775091
W20240502 01:53:54.746438 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3e245c8534e84c55922b7cd77670a4a3: could not find any suitable replica to move from server fdc2ac372492454c85387589a91f63af to server 2f306549407b4bb281ccf2994035e4f1
I20240502 01:53:55.084758 20682 rebalancer_tool.cc:1074] tablet 5e8e78bd417d46cdb166087dbfa86446: '6a36c952c51542c0bf7ea3ec2857a9f9' -> '3fceccefba274b1dbbd8216af8b0bc29' move scheduled
I20240502 01:53:55.118558 20682 rebalancer_tool.cc:1074] tablet 0618761d30f34cfe9650d13c9275cdcb: '4da690c0fa13407a85c0689ce8de9ade' -> 'fdc2ac372492454c85387589a91f63af' move scheduled
I20240502 01:53:55.206739 20682 rebalancer_tool.cc:1074] tablet 5df83f83d2674155afba93c2d992af0a: 'bb31bceba76d466db24b9341c04900c3' -> 'f19cc6cb1e76460aa6711e49a5775091' move scheduled
I20240502 01:53:55.312928 20682 rebalancer_tool.cc:1074] tablet 648b1cc4d8b54c5eb0948ca8c1984a1f: '2f306549407b4bb281ccf2994035e4f1' -> 'dfa1152987644bd781b46c7e42a7a685' move scheduled
I20240502 01:53:55.377117 20682 rebalancer_tool.cc:1074] tablet f0b3862e504d4b43a60477e5efef29b1: '3fceccefba274b1dbbd8216af8b0bc29' -> 'fdc2ac372492454c85387589a91f63af' move scheduled
I20240502 01:53:55.398279 20682 rebalancer_tool.cc:1136] tablet 4bb42883b1a0475b863d008bd01fa70a: 'dfa1152987644bd781b46c7e42a7a685' -> 'f19cc6cb1e76460aa6711e49a5775091' move completed: OK
I20240502 01:53:55.408443 20682 rebalancer_tool.cc:1136] tablet 62887e332443457f8a5b41a982354dc4: '2f306549407b4bb281ccf2994035e4f1' -> 'bb31bceba76d466db24b9341c04900c3' move completed: OK
I20240502 01:53:55.427201 20682 rebalancer_tool.cc:1136] tablet 975f253ff03447ff96fe84169c05097e: 'fdc2ac372492454c85387589a91f63af' -> '3fceccefba274b1dbbd8216af8b0bc29' move completed: OK
I20240502 01:53:55.436267 20682 rebalancer_tool.cc:1136] tablet ebb9bca9f1bd41b0857939909b1110e4: 'fdc2ac372492454c85387589a91f63af' -> '2f306549407b4bb281ccf2994035e4f1' move completed: OK
I20240502 01:53:55.453979 20682 rebalancer_tool.cc:1136] tablet 31aa6a26f64f49368ad43cc97c66f2f1: 'f19cc6cb1e76460aa6711e49a5775091' -> 'bb31bceba76d466db24b9341c04900c3' move completed: OK
I20240502 01:53:55.463145 20682 rebalancer_tool.cc:1136] tablet 1f01949cf70440b38e59a5020fd48c44: 'dfa1152987644bd781b46c7e42a7a685' -> 'f19cc6cb1e76460aa6711e49a5775091' move completed: OK
I20240502 01:53:55.474329 20682 rebalancer_tool.cc:1136] tablet 5b092dd1cc004c26a33e17c28af4d3ba: 'dfa1152987644bd781b46c7e42a7a685' -> '6a36c952c51542c0bf7ea3ec2857a9f9' move completed: OK
I20240502 01:53:55.499199 20682 rebalancer_tool.cc:1136] tablet 52430bbf347a4c5191edb23e1aa5d385: '3fceccefba274b1dbbd8216af8b0bc29' -> 'fdc2ac372492454c85387589a91f63af' move completed: OK
I20240502 01:53:55.500372 20682 rebalancer_tool.cc:1123] tablet 377e804dbeed4913a7d435cfdd1e2d8e: '4da690c0fa13407a85c0689ce8de9ade' -> '2f306549407b4bb281ccf2994035e4f1' move is abandoned: Not found: 377e804dbeed4913a7d435cfdd1e2d8e: tablet not found
I20240502 01:53:55.509542 20682 rebalancer_tool.cc:1136] tablet 52ac2775e02844d195677035796c3847: '4da690c0fa13407a85c0689ce8de9ade' -> '6a36c952c51542c0bf7ea3ec2857a9f9' move completed: OK
I20240502 01:53:55.517946 20682 rebalancer_tool.cc:1136] tablet e42126e16cda4d908b00e2e4a6807d6b: '4da690c0fa13407a85c0689ce8de9ade' -> 'bb31bceba76d466db24b9341c04900c3' move completed: OK
I20240502 01:53:55.526515 20682 rebalancer_tool.cc:1136] tablet 6aacf7c4f19546f581c9a54ad171916b: 'bb31bceba76d466db24b9341c04900c3' -> '2f306549407b4bb281ccf2994035e4f1' move completed: OK
I20240502 01:53:55.537919 20682 rebalancer_tool.cc:1136] tablet 9642b9b37a784cf881bfc4971d8c9cb6: 'dfa1152987644bd781b46c7e42a7a685' -> 'f19cc6cb1e76460aa6711e49a5775091' move completed: OK
I20240502 01:53:55.561498 20682 rebalancer_tool.cc:1136] tablet 65bdab2c9fba48a6828a864278319cc0: '3fceccefba274b1dbbd8216af8b0bc29' -> '6a36c952c51542c0bf7ea3ec2857a9f9' move completed: OK
I20240502 01:53:55.721164 20682 rebalancer_tool.cc:1074] tablet 1fdd104f1a28446cbc25d4ca16d03d7d: 'bb31bceba76d466db24b9341c04900c3' -> '6a36c952c51542c0bf7ea3ec2857a9f9' move scheduled
I20240502 01:53:55.825027 20682 rebalancer_tool.cc:1074] tablet 4aefc61cb95f4e949e7a23a1c677749e: 'dfa1152987644bd781b46c7e42a7a685' -> 'f19cc6cb1e76460aa6711e49a5775091' move scheduled
I20240502 01:53:56.131110 20682 rebalancer_tool.cc:1074] tablet 617443200e2c47b69d63ef056294f40c: 'fdc2ac372492454c85387589a91f63af' -> '4da690c0fa13407a85c0689ce8de9ade' move scheduled
I20240502 01:53:56.295820 20682 rebalancer_tool.cc:1074] tablet 5124abbb09db4be0955c6bc831af08fa: '3fceccefba274b1dbbd8216af8b0bc29' -> '2f306549407b4bb281ccf2994035e4f1' move scheduled
I20240502 01:53:56.422159 20682 rebalancer_tool.cc:1074] tablet 71cbc012efe14fdb937302796a9f84e5: 'bb31bceba76d466db24b9341c04900c3' -> '6a36c952c51542c0bf7ea3ec2857a9f9' move scheduled
I20240502 01:53:56.611089 20682 rebalancer_tool.cc:1074] tablet 10ec78a08ab04678897a6892988e515f: 'dfa1152987644bd781b46c7e42a7a685' -> 'f19cc6cb1e76460aa6711e49a5775091' move scheduled
I20240502 01:53:56.827685 20682 rebalancer_tool.cc:1074] tablet 8e7f88b6c509477fb0900024e7de8f00: '3fceccefba274b1dbbd8216af8b0bc29' -> 'fdc2ac372492454c85387589a91f63af' move scheduled
I20240502 01:53:56.936105 20682 rebalancer_tool.cc:1074] tablet 03bd47783e884a70bfea1dfab4f2e563: '4da690c0fa13407a85c0689ce8de9ade' -> '2f306549407b4bb281ccf2994035e4f1' move scheduled
I20240502 01:53:57.097388 20682 rebalancer_tool.cc:1074] tablet 31a4ca0cc44e492fbd13a70ce38add35: 'bb31bceba76d466db24b9341c04900c3' -> '3fceccefba274b1dbbd8216af8b0bc29' move scheduled
I20240502 01:53:57.512892 20682 rebalancer_tool.cc:1074] tablet cf0237a4c65b495ba892871dd850bed9: '6a36c952c51542c0bf7ea3ec2857a9f9' -> 'dfa1152987644bd781b46c7e42a7a685' move scheduled
I20240502 01:53:57.794423 20682 rebalancer_tool.cc:1074] tablet a212c17ba64842c99b3d6a78a3b7cc8a: 'fdc2ac372492454c85387589a91f63af' -> 'f19cc6cb1e76460aa6711e49a5775091' move scheduled
I20240502 01:53:57.960652 20682 rebalancer_tool.cc:1074] tablet 1d8e2e0f6ed34e4cb004e1c5309c0f1f: 'f19cc6cb1e76460aa6711e49a5775091' -> '4da690c0fa13407a85c0689ce8de9ade' move scheduled
I20240502 01:53:58.470554 20682 rebalancer_tool.cc:1074] tablet 7e9be84ef6a043818f166f6b47898855: 'bb31bceba76d466db24b9341c04900c3' -> '2f306549407b4bb281ccf2994035e4f1' move scheduled
I20240502 01:53:58.772485 20682 rebalancer_tool.cc:1074] tablet de174ba4f7d848718ab8dbeb74369055: 'dfa1152987644bd781b46c7e42a7a685' -> '6a36c952c51542c0bf7ea3ec2857a9f9' move scheduled
I20240502 01:53:58.905131 20682 rebalancer_tool.cc:1136] tablet 31a4ca0cc44e492fbd13a70ce38add35: 'bb31bceba76d466db24b9341c04900c3' -> '3fceccefba274b1dbbd8216af8b0bc29' move completed: OK
I20240502 01:53:58.914335 20682 rebalancer_tool.cc:1136] tablet 10ec78a08ab04678897a6892988e515f: 'dfa1152987644bd781b46c7e42a7a685' -> 'f19cc6cb1e76460aa6711e49a5775091' move completed: OK
I20240502 01:53:58.923660 20682 rebalancer_tool.cc:1136] tablet 03bd47783e884a70bfea1dfab4f2e563: '4da690c0fa13407a85c0689ce8de9ade' -> '2f306549407b4bb281ccf2994035e4f1' move completed: OK
I20240502 01:53:58.961782 20682 rebalancer_tool.cc:1136] tablet 71cbc012efe14fdb937302796a9f84e5: 'bb31bceba76d466db24b9341c04900c3' -> '6a36c952c51542c0bf7ea3ec2857a9f9' move completed: OK
I20240502 01:53:59.013965 20682 rebalancer_tool.cc:1136] tablet 8e7f88b6c509477fb0900024e7de8f00: '3fceccefba274b1dbbd8216af8b0bc29' -> 'fdc2ac372492454c85387589a91f63af' move completed: OK
I20240502 01:53:59.043543 20682 rebalancer_tool.cc:1136] tablet 5124abbb09db4be0955c6bc831af08fa: '3fceccefba274b1dbbd8216af8b0bc29' -> '2f306549407b4bb281ccf2994035e4f1' move completed: OK
I20240502 01:53:59.071421 20682 rebalancer_tool.cc:1136] tablet 617443200e2c47b69d63ef056294f40c: 'fdc2ac372492454c85387589a91f63af' -> '4da690c0fa13407a85c0689ce8de9ade' move completed: OK
I20240502 01:53:59.102831 20682 rebalancer_tool.cc:1136] tablet 4aefc61cb95f4e949e7a23a1c677749e: 'dfa1152987644bd781b46c7e42a7a685' -> 'f19cc6cb1e76460aa6711e49a5775091' move completed: OK
I20240502 01:53:59.112903 20682 rebalancer_tool.cc:1136] tablet 1fdd104f1a28446cbc25d4ca16d03d7d: 'bb31bceba76d466db24b9341c04900c3' -> '6a36c952c51542c0bf7ea3ec2857a9f9' move completed: OK
I20240502 01:53:59.224169 20682 rebalancer_tool.cc:1136] tablet 5df83f83d2674155afba93c2d992af0a: 'bb31bceba76d466db24b9341c04900c3' -> 'f19cc6cb1e76460aa6711e49a5775091' move completed: OK
I20240502 01:53:59.385051 20682 rebalancer_tool.cc:1136] tablet 5e8e78bd417d46cdb166087dbfa86446: '6a36c952c51542c0bf7ea3ec2857a9f9' -> '3fceccefba274b1dbbd8216af8b0bc29' move completed: OK
I20240502 01:53:59.535291 20682 rebalancer_tool.cc:1074] tablet 5152c002eadd4d54ab10a093bce45f17: 'bb31bceba76d466db24b9341c04900c3' -> 'fdc2ac372492454c85387589a91f63af' move scheduled
I20240502 01:53:59.729223 20682 rebalancer_tool.cc:1074] tablet 7c697210a14d4982a939ea7965cb769d: '3fceccefba274b1dbbd8216af8b0bc29' -> 'f19cc6cb1e76460aa6711e49a5775091' move scheduled
I20240502 01:53:59.789330 20682 rebalancer_tool.cc:1074] tablet b39b88755cd74b0294d264215aea99cf: '3fceccefba274b1dbbd8216af8b0bc29' -> '2f306549407b4bb281ccf2994035e4f1' move scheduled
I20240502 01:53:59.884867 20682 rebalancer_tool.cc:1074] tablet 59c288e7279f4119afb6c3bf860e6dc3: 'dfa1152987644bd781b46c7e42a7a685' -> 'fdc2ac372492454c85387589a91f63af' move scheduled
I20240502 01:54:00.275133 20682 rebalancer_tool.cc:1074] tablet ba1e82a39c2e439889da60007372a99b: 'f19cc6cb1e76460aa6711e49a5775091' -> '3fceccefba274b1dbbd8216af8b0bc29' move scheduled
I20240502 01:54:00.658514 20682 rebalancer_tool.cc:1074] tablet aebde87a7f114537abd9c1d8b7e97004: '2f306549407b4bb281ccf2994035e4f1' -> 'dfa1152987644bd781b46c7e42a7a685' move scheduled
I20240502 01:54:00.734349 20682 rebalancer_tool.cc:1074] tablet 0d9f168de0d1410da2518abf936035ff: '3fceccefba274b1dbbd8216af8b0bc29' -> 'f19cc6cb1e76460aa6711e49a5775091' move scheduled
I20240502 01:54:01.007486 20682 rebalancer_tool.cc:1136] tablet de174ba4f7d848718ab8dbeb74369055: 'dfa1152987644bd781b46c7e42a7a685' -> '6a36c952c51542c0bf7ea3ec2857a9f9' move completed: OK
I20240502 01:54:01.047502 20682 rebalancer_tool.cc:1136] tablet 7e9be84ef6a043818f166f6b47898855: 'bb31bceba76d466db24b9341c04900c3' -> '2f306549407b4bb281ccf2994035e4f1' move completed: OK
I20240502 01:54:01.070134 20682 rebalancer_tool.cc:1136] tablet a212c17ba64842c99b3d6a78a3b7cc8a: 'fdc2ac372492454c85387589a91f63af' -> 'f19cc6cb1e76460aa6711e49a5775091' move completed: OK
I20240502 01:54:01.080523 20682 rebalancer_tool.cc:1136] tablet 1d8e2e0f6ed34e4cb004e1c5309c0f1f: 'f19cc6cb1e76460aa6711e49a5775091' -> '4da690c0fa13407a85c0689ce8de9ade' move completed: OK
I20240502 01:54:01.090240 20682 rebalancer_tool.cc:1136] tablet cf0237a4c65b495ba892871dd850bed9: '6a36c952c51542c0bf7ea3ec2857a9f9' -> 'dfa1152987644bd781b46c7e42a7a685' move completed: OK
I20240502 01:54:01.100097 20682 rebalancer_tool.cc:1136] tablet f0b3862e504d4b43a60477e5efef29b1: '3fceccefba274b1dbbd8216af8b0bc29' -> 'fdc2ac372492454c85387589a91f63af' move completed: OK
I20240502 01:54:01.153762 20682 rebalancer_tool.cc:1136] tablet 648b1cc4d8b54c5eb0948ca8c1984a1f: '2f306549407b4bb281ccf2994035e4f1' -> 'dfa1152987644bd781b46c7e42a7a685' move completed: OK
I20240502 01:54:01.329195 20682 rebalancer_tool.cc:1136] tablet 0618761d30f34cfe9650d13c9275cdcb: '4da690c0fa13407a85c0689ce8de9ade' -> 'fdc2ac372492454c85387589a91f63af' move completed: OK
I20240502 01:54:01.472913 20682 rebalancer_tool.cc:1074] tablet 30f43088bd0f48ee86907df16194b9be: 'dfa1152987644bd781b46c7e42a7a685' -> 'fdc2ac372492454c85387589a91f63af' move scheduled
I20240502 01:54:01.621912 20682 rebalancer_tool.cc:1074] tablet 4099f4b2ae164020839faab53957750f: '3fceccefba274b1dbbd8216af8b0bc29' -> 'f19cc6cb1e76460aa6711e49a5775091' move scheduled
I20240502 01:54:01.697983 20682 rebalancer_tool.cc:1074] tablet 377f8b554ad1448d981d2283b865f8fe: 'dfa1152987644bd781b46c7e42a7a685' -> '2f306549407b4bb281ccf2994035e4f1' move scheduled
I20240502 01:54:01.741489 20682 rebalancer_tool.cc:1074] tablet ed1522ec150a46f28fd553f9b563651f: 'dfa1152987644bd781b46c7e42a7a685' -> 'fdc2ac372492454c85387589a91f63af' move scheduled
I20240502 01:54:01.949224 20682 rebalancer_tool.cc:1136] tablet ba1e82a39c2e439889da60007372a99b: 'f19cc6cb1e76460aa6711e49a5775091' -> '3fceccefba274b1dbbd8216af8b0bc29' move completed: OK
I20240502 01:54:01.958670 20682 rebalancer_tool.cc:1136] tablet b39b88755cd74b0294d264215aea99cf: '3fceccefba274b1dbbd8216af8b0bc29' -> '2f306549407b4bb281ccf2994035e4f1' move completed: OK
I20240502 01:54:01.979565 20682 rebalancer_tool.cc:1136] tablet 5152c002eadd4d54ab10a093bce45f17: 'bb31bceba76d466db24b9341c04900c3' -> 'fdc2ac372492454c85387589a91f63af' move completed: OK
I20240502 01:54:02.048028 20682 rebalancer_tool.cc:1136] tablet 59c288e7279f4119afb6c3bf860e6dc3: 'dfa1152987644bd781b46c7e42a7a685' -> 'fdc2ac372492454c85387589a91f63af' move completed: OK
I20240502 01:54:02.100973 20682 rebalancer_tool.cc:1074] tablet b29218a879e74895bfe00021ad358aa4: '3fceccefba274b1dbbd8216af8b0bc29' -> '2f306549407b4bb281ccf2994035e4f1' move scheduled
W20240502 01:54:02.543975 20716 kernel_stack_watchdog.cc:198] Thread 20682 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:637 for 342ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fcfde080980 (unknown) at ??:0
@ 0x90ef77 __sanitizer::internal_sched_yield() at ??:?
@ 0x87462d __interceptor_pthread_create at ??:?
@ 0x7fcfdaceed11 kudu::Thread::StartThread() at ??:0
@ 0xa4d187 kudu::Thread::Create() at ??:?
@ 0x7fcfe1946530 kudu::rpc::ReactorThread::Init() at ??:0
@ 0x7fcfe1951797 kudu::rpc::Reactor::Init() at ??:0
@ 0x7fcfe18fa5c5 kudu::rpc::Messenger::Init() at ??:0
@ 0x7fcfe18f99d1 kudu::rpc::MessengerBuilder::Build() at ??:0
@ 0x7fcfeebd5bb2 kudu::tools::BuildMessenger() at ??:0
@ 0x7fcfeebef9c8 kudu::tools::BuildProxy<>() at ??:0
@ 0x7fcfefb0f2a9 kudu::tools::CheckCompleteMove() at ??:0
@ 0x7fcfefa951df kudu::tools::RebalancerTool::AlgoBasedRunner::UpdateMovesInProgressStatus() at ??:0
@ 0x7fcfefa8c82b kudu::tools::RebalancerTool::RunWith() at ??:0
@ 0x7fcfefa8914c kudu::tools::RebalancerTool::Run() at ??:0
@ 0x97996b kudu::tools::(anonymous namespace)::RunRebalance() at ??:?
I20240502 01:54:02.944434 20682 rebalancer_tool.cc:1136] tablet 0d9f168de0d1410da2518abf936035ff: '3fceccefba274b1dbbd8216af8b0bc29' -> 'f19cc6cb1e76460aa6711e49a5775091' move completed: OK
I20240502 01:54:03.058880 20682 rebalancer_tool.cc:1136] tablet 7c697210a14d4982a939ea7965cb769d: '3fceccefba274b1dbbd8216af8b0bc29' -> 'f19cc6cb1e76460aa6711e49a5775091' move completed: OK
I20240502 01:54:03.133359 20682 rebalancer_tool.cc:1136] tablet 30f43088bd0f48ee86907df16194b9be: 'dfa1152987644bd781b46c7e42a7a685' -> 'fdc2ac372492454c85387589a91f63af' move completed: OK
I20240502 01:54:03.700495 20682 rebalancer_tool.cc:1136] tablet 377f8b554ad1448d981d2283b865f8fe: 'dfa1152987644bd781b46c7e42a7a685' -> '2f306549407b4bb281ccf2994035e4f1' move completed: OK
I20240502 01:54:04.041791 20682 rebalancer_tool.cc:1136] tablet ed1522ec150a46f28fd553f9b563651f: 'dfa1152987644bd781b46c7e42a7a685' -> 'fdc2ac372492454c85387589a91f63af' move completed: OK
I20240502 01:54:04.055881 20682 rebalancer_tool.cc:1136] tablet b29218a879e74895bfe00021ad358aa4: '3fceccefba274b1dbbd8216af8b0bc29' -> '2f306549407b4bb281ccf2994035e4f1' move completed: OK
I20240502 01:54:04.065943 20682 rebalancer_tool.cc:1136] tablet 4099f4b2ae164020839faab53957750f: '3fceccefba274b1dbbd8216af8b0bc29' -> 'f19cc6cb1e76460aa6711e49a5775091' move completed: OK
I20240502 01:54:04.748132 20682 rebalancer_tool.cc:1136] tablet aebde87a7f114537abd9c1d8b7e97004: '2f306549407b4bb281ccf2994035e4f1' -> 'dfa1152987644bd781b46c7e42a7a685' move completed: OK
W20240502 01:54:05.668244 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6cf4b74847bf491c93c43661f9c6d7a5: could not find any suitable replica to move from server f19cc6cb1e76460aa6711e49a5775091 to server bb31bceba76d466db24b9341c04900c3
W20240502 01:54:05.668525 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 448fbe791a344997b564da10a31bc5d2: could not find any suitable replica to move from server 2f306549407b4bb281ccf2994035e4f1 to server 4da690c0fa13407a85c0689ce8de9ade
W20240502 01:54:05.668653 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 61898cfacac64133a52ca6484799789a: could not find any suitable replica to move from server f19cc6cb1e76460aa6711e49a5775091 to server bb31bceba76d466db24b9341c04900c3
W20240502 01:54:05.668814 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 43fc79d9e659415194e6983d21d5384a: could not find any suitable replica to move from server 4da690c0fa13407a85c0689ce8de9ade to server dfa1152987644bd781b46c7e42a7a685
W20240502 01:54:05.668932 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9c8c7af6d77740e1a130a1e68baee62d: could not find any suitable replica to move from server f19cc6cb1e76460aa6711e49a5775091 to server fdc2ac372492454c85387589a91f63af
W20240502 01:54:05.669270 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 736004bf70524edd905a6bb90bb25cc0: could not find any suitable replica to move from server 2f306549407b4bb281ccf2994035e4f1 to server bb31bceba76d466db24b9341c04900c3
W20240502 01:54:05.669370 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6077652d45454150b5c7d7072780d895: could not find any suitable replica to move from server bb31bceba76d466db24b9341c04900c3 to server 3fceccefba274b1dbbd8216af8b0bc29
W20240502 01:54:05.669454 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6f9eef247379455187a9871e7d04643f: could not find any suitable replica to move from server bb31bceba76d466db24b9341c04900c3 to server 4da690c0fa13407a85c0689ce8de9ade
I20240502 01:54:05.703860 20682 rebalancer_tool.cc:1074] tablet 28463d45bfde44429d56179c1eec56b0: 'f19cc6cb1e76460aa6711e49a5775091' -> '4da690c0fa13407a85c0689ce8de9ade' move scheduled
I20240502 01:54:05.796542 20682 rebalancer_tool.cc:1074] tablet d10d76ac7b094fe9b230b1afcbed9ca6: '6a36c952c51542c0bf7ea3ec2857a9f9' -> 'dfa1152987644bd781b46c7e42a7a685' move scheduled
I20240502 01:54:05.837718 20682 rebalancer_tool.cc:1074] tablet 0d9f168de0d1410da2518abf936035ff: 'bb31bceba76d466db24b9341c04900c3' -> 'fdc2ac372492454c85387589a91f63af' move scheduled
I20240502 01:54:05.894308 20682 rebalancer_tool.cc:1074] tablet 81f335c023444796ab933e19d809cc03: '2f306549407b4bb281ccf2994035e4f1' -> '3fceccefba274b1dbbd8216af8b0bc29' move scheduled
I20240502 01:54:05.937664 20682 rebalancer_tool.cc:1074] tablet fac0ba26fabd4643bc38b7559faa471c: 'f19cc6cb1e76460aa6711e49a5775091' -> '4da690c0fa13407a85c0689ce8de9ade' move scheduled
I20240502 01:54:05.967116 20682 rebalancer_tool.cc:1074] tablet a212c17ba64842c99b3d6a78a3b7cc8a: '6a36c952c51542c0bf7ea3ec2857a9f9' -> 'dfa1152987644bd781b46c7e42a7a685' move scheduled
I20240502 01:54:06.016026 20682 rebalancer_tool.cc:1074] tablet 139304da92ab4f48b8a08fa24ed60c95: 'fdc2ac372492454c85387589a91f63af' -> 'bb31bceba76d466db24b9341c04900c3' move scheduled
I20240502 01:54:06.089578 20682 rebalancer_tool.cc:1074] tablet 3ff84ce60ac1438d92b2c944bb27aabb: '2f306549407b4bb281ccf2994035e4f1' -> '3fceccefba274b1dbbd8216af8b0bc29' move scheduled
I20240502 01:54:06.135116 20682 rebalancer_tool.cc:1074] tablet bffb870715a243d992107583d7c17f1b: 'f19cc6cb1e76460aa6711e49a5775091' -> '4da690c0fa13407a85c0689ce8de9ade' move scheduled
I20240502 01:54:06.535724 20682 rebalancer_tool.cc:1074] tablet bfd7b1d1002c4b57a14d43ed9c6d9f50: '6a36c952c51542c0bf7ea3ec2857a9f9' -> 'dfa1152987644bd781b46c7e42a7a685' move scheduled
I20240502 01:54:06.837682 20682 rebalancer_tool.cc:1074] tablet 246cec994d274786aad7672f2a30e5d8: 'fdc2ac372492454c85387589a91f63af' -> 'bb31bceba76d466db24b9341c04900c3' move scheduled
I20240502 01:54:06.933490 20682 rebalancer_tool.cc:1074] tablet 18549019e31b47d899d011cef0637221: '2f306549407b4bb281ccf2994035e4f1' -> 'f19cc6cb1e76460aa6711e49a5775091' move scheduled
I20240502 01:54:07.021600 20682 rebalancer_tool.cc:1074] tablet 377f8b554ad1448d981d2283b865f8fe: '3fceccefba274b1dbbd8216af8b0bc29' -> '6a36c952c51542c0bf7ea3ec2857a9f9' move scheduled
I20240502 01:54:07.214780 20682 rebalancer_tool.cc:1074] tablet 062b9d90fb0444f4be37bf09770c9caf: '2f306549407b4bb281ccf2994035e4f1' -> '4da690c0fa13407a85c0689ce8de9ade' move scheduled
I20240502 01:54:08.161602 20682 rebalancer_tool.cc:1074] tablet 5df83f83d2674155afba93c2d992af0a: 'dfa1152987644bd781b46c7e42a7a685' -> 'fdc2ac372492454c85387589a91f63af' move scheduled
I20240502 01:54:08.441254 20682 rebalancer_tool.cc:1074] tablet 2ea539ebb37f42f68c166242a06f8d2e: 'f19cc6cb1e76460aa6711e49a5775091' -> 'bb31bceba76d466db24b9341c04900c3' move scheduled
I20240502 01:54:08.802130 20682 rebalancer_tool.cc:1074] tablet 4aefc61cb95f4e949e7a23a1c677749e: '3fceccefba274b1dbbd8216af8b0bc29' -> 'dfa1152987644bd781b46c7e42a7a685' move scheduled
I20240502 01:54:08.989239 20682 rebalancer_tool.cc:1074] tablet 2152c07e448b4582abff22b034363c70: '6a36c952c51542c0bf7ea3ec2857a9f9' -> '4da690c0fa13407a85c0689ce8de9ade' move scheduled
I20240502 01:54:09.207497 20682 rebalancer_tool.cc:1074] tablet da06571e020948059fb06f65f26c831d: 'fdc2ac372492454c85387589a91f63af' -> '2f306549407b4bb281ccf2994035e4f1' move scheduled
I20240502 01:54:09.715008 20682 rebalancer_tool.cc:1136] tablet 062b9d90fb0444f4be37bf09770c9caf: '2f306549407b4bb281ccf2994035e4f1' -> '4da690c0fa13407a85c0689ce8de9ade' move completed: OK
I20240502 01:54:10.011910 20682 rebalancer_tool.cc:1136] tablet 377f8b554ad1448d981d2283b865f8fe: '3fceccefba274b1dbbd8216af8b0bc29' -> '6a36c952c51542c0bf7ea3ec2857a9f9' move completed: OK
I20240502 01:54:10.076409 20682 rebalancer_tool.cc:1136] tablet 18549019e31b47d899d011cef0637221: '2f306549407b4bb281ccf2994035e4f1' -> 'f19cc6cb1e76460aa6711e49a5775091' move completed: OK
I20240502 01:54:10.133873 20682 rebalancer_tool.cc:1136] tablet 246cec994d274786aad7672f2a30e5d8: 'fdc2ac372492454c85387589a91f63af' -> 'bb31bceba76d466db24b9341c04900c3' move completed: OK
I20240502 01:54:10.203672 20682 rebalancer_tool.cc:1136] tablet bfd7b1d1002c4b57a14d43ed9c6d9f50: '6a36c952c51542c0bf7ea3ec2857a9f9' -> 'dfa1152987644bd781b46c7e42a7a685' move completed: OK
I20240502 01:54:10.241560 20682 rebalancer_tool.cc:1136] tablet bffb870715a243d992107583d7c17f1b: 'f19cc6cb1e76460aa6711e49a5775091' -> '4da690c0fa13407a85c0689ce8de9ade' move completed: OK
I20240502 01:54:10.253343 20682 rebalancer_tool.cc:1136] tablet 3ff84ce60ac1438d92b2c944bb27aabb: '2f306549407b4bb281ccf2994035e4f1' -> '3fceccefba274b1dbbd8216af8b0bc29' move completed: OK
I20240502 01:54:10.319710 20682 rebalancer_tool.cc:1136] tablet 139304da92ab4f48b8a08fa24ed60c95: 'fdc2ac372492454c85387589a91f63af' -> 'bb31bceba76d466db24b9341c04900c3' move completed: OK
I20240502 01:54:10.374953 20682 rebalancer_tool.cc:1136] tablet fac0ba26fabd4643bc38b7559faa471c: 'f19cc6cb1e76460aa6711e49a5775091' -> '4da690c0fa13407a85c0689ce8de9ade' move completed: OK
I20240502 01:54:10.400988 20682 rebalancer_tool.cc:1136] tablet 81f335c023444796ab933e19d809cc03: '2f306549407b4bb281ccf2994035e4f1' -> '3fceccefba274b1dbbd8216af8b0bc29' move completed: OK
I20240502 01:54:10.434423 20682 rebalancer_tool.cc:1136] tablet 0d9f168de0d1410da2518abf936035ff: 'bb31bceba76d466db24b9341c04900c3' -> 'fdc2ac372492454c85387589a91f63af' move completed: OK
I20240502 01:54:10.449909 20682 rebalancer_tool.cc:1136] tablet d10d76ac7b094fe9b230b1afcbed9ca6: '6a36c952c51542c0bf7ea3ec2857a9f9' -> 'dfa1152987644bd781b46c7e42a7a685' move completed: OK
I20240502 01:54:10.459271 20682 rebalancer_tool.cc:1136] tablet 28463d45bfde44429d56179c1eec56b0: 'f19cc6cb1e76460aa6711e49a5775091' -> '4da690c0fa13407a85c0689ce8de9ade' move completed: OK
I20240502 01:54:10.497380 20682 rebalancer_tool.cc:1074] tablet 4bb42883b1a0475b863d008bd01fa70a: 'bb31bceba76d466db24b9341c04900c3' -> '2f306549407b4bb281ccf2994035e4f1' move scheduled
I20240502 01:54:10.527388 20682 rebalancer_tool.cc:1074] tablet 472d7827b3074236a81a3ede098f6c0a: 'fdc2ac372492454c85387589a91f63af' -> '3fceccefba274b1dbbd8216af8b0bc29' move scheduled
I20240502 01:54:10.553412 20682 rebalancer_tool.cc:1074] tablet cf0237a4c65b495ba892871dd850bed9: 'f19cc6cb1e76460aa6711e49a5775091' -> 'bb31bceba76d466db24b9341c04900c3' move scheduled
I20240502 01:54:10.705329 20682 rebalancer_tool.cc:1074] tablet ed1522ec150a46f28fd553f9b563651f: '4da690c0fa13407a85c0689ce8de9ade' -> '6a36c952c51542c0bf7ea3ec2857a9f9' move scheduled
I20240502 01:54:11.303234 20682 rebalancer_tool.cc:1074] tablet fbd94103cae04473ba81709da39d7ed8: '2f306549407b4bb281ccf2994035e4f1' -> 'dfa1152987644bd781b46c7e42a7a685' move scheduled
I20240502 01:54:11.464592 20682 rebalancer_tool.cc:1074] tablet 68bc23294ac64c9486bffebbdb16b508: 'fdc2ac372492454c85387589a91f63af' -> '3fceccefba274b1dbbd8216af8b0bc29' move scheduled
I20240502 01:54:11.510532 20682 rebalancer_tool.cc:1074] tablet 1b14ac3413004127b72904b631ec1b94: 'f19cc6cb1e76460aa6711e49a5775091' -> 'dfa1152987644bd781b46c7e42a7a685' move scheduled
I20240502 01:54:11.540656 20682 rebalancer_tool.cc:1074] tablet 75bb4e19cf57405fabcb532b55a29ced: '6a36c952c51542c0bf7ea3ec2857a9f9' -> '4da690c0fa13407a85c0689ce8de9ade' move scheduled
I20240502 01:54:11.591434 20682 rebalancer_tool.cc:1074] tablet 8e7f88b6c509477fb0900024e7de8f00: 'bb31bceba76d466db24b9341c04900c3' -> '6a36c952c51542c0bf7ea3ec2857a9f9' move scheduled
I20240502 01:54:11.739184 20682 rebalancer_tool.cc:1074] tablet 2714919062f5465c8cde1a3c4720978a: 'fdc2ac372492454c85387589a91f63af' -> '3fceccefba274b1dbbd8216af8b0bc29' move scheduled
I20240502 01:54:13.420715 20682 rebalancer_tool.cc:1136] tablet da06571e020948059fb06f65f26c831d: 'fdc2ac372492454c85387589a91f63af' -> '2f306549407b4bb281ccf2994035e4f1' move completed: OK
I20240502 01:54:13.432004 20682 rebalancer_tool.cc:1136] tablet 2ea539ebb37f42f68c166242a06f8d2e: 'f19cc6cb1e76460aa6711e49a5775091' -> 'bb31bceba76d466db24b9341c04900c3' move completed: OK
I20240502 01:54:13.613435 20682 rebalancer_tool.cc:1136] tablet a212c17ba64842c99b3d6a78a3b7cc8a: '6a36c952c51542c0bf7ea3ec2857a9f9' -> 'dfa1152987644bd781b46c7e42a7a685' move completed: OK
I20240502 01:54:13.832543 20682 rebalancer_tool.cc:1136] tablet 472d7827b3074236a81a3ede098f6c0a: 'fdc2ac372492454c85387589a91f63af' -> '3fceccefba274b1dbbd8216af8b0bc29' move completed: OK
I20240502 01:54:14.084471 20682 rebalancer_tool.cc:1136] tablet 2152c07e448b4582abff22b034363c70: '6a36c952c51542c0bf7ea3ec2857a9f9' -> '4da690c0fa13407a85c0689ce8de9ade' move completed: OK
I20240502 01:54:14.106241 20682 rebalancer_tool.cc:1074] tablet 1fdd104f1a28446cbc25d4ca16d03d7d: 'dfa1152987644bd781b46c7e42a7a685' -> '2f306549407b4bb281ccf2994035e4f1' move scheduled
I20240502 01:54:14.163194 20682 rebalancer_tool.cc:1074] tablet f3e4c8f98b5e48e3ba6affc2df08943a: 'fdc2ac372492454c85387589a91f63af' -> 'f19cc6cb1e76460aa6711e49a5775091' move scheduled
I20240502 01:54:14.276263 20682 rebalancer_tool.cc:1074] tablet ee418fc4028a422eabac02ed63ea9047: '6a36c952c51542c0bf7ea3ec2857a9f9' -> 'bb31bceba76d466db24b9341c04900c3' move scheduled
I20240502 01:54:14.383965 20682 rebalancer_tool.cc:1074] tablet 29d8cb76dc514e7b9002bd2291d8ee8d: '6a36c952c51542c0bf7ea3ec2857a9f9' -> 'fdc2ac372492454c85387589a91f63af' move scheduled
I20240502 01:54:14.423421 20682 rebalancer_tool.cc:1136] tablet 2714919062f5465c8cde1a3c4720978a: 'fdc2ac372492454c85387589a91f63af' -> '3fceccefba274b1dbbd8216af8b0bc29' move completed: OK
I20240502 01:54:14.440779 20682 rebalancer_tool.cc:1136] tablet 8e7f88b6c509477fb0900024e7de8f00: 'bb31bceba76d466db24b9341c04900c3' -> '6a36c952c51542c0bf7ea3ec2857a9f9' move completed: OK
I20240502 01:54:14.498446 20682 rebalancer_tool.cc:1136] tablet 75bb4e19cf57405fabcb532b55a29ced: '6a36c952c51542c0bf7ea3ec2857a9f9' -> '4da690c0fa13407a85c0689ce8de9ade' move completed: OK
I20240502 01:54:14.507236 20682 rebalancer_tool.cc:1136] tablet fbd94103cae04473ba81709da39d7ed8: '2f306549407b4bb281ccf2994035e4f1' -> 'dfa1152987644bd781b46c7e42a7a685' move completed: OK
I20240502 01:54:14.522603 20682 rebalancer_tool.cc:1136] tablet ed1522ec150a46f28fd553f9b563651f: '4da690c0fa13407a85c0689ce8de9ade' -> '6a36c952c51542c0bf7ea3ec2857a9f9' move completed: OK
I20240502 01:54:14.533087 20682 rebalancer_tool.cc:1136] tablet 4bb42883b1a0475b863d008bd01fa70a: 'bb31bceba76d466db24b9341c04900c3' -> '2f306549407b4bb281ccf2994035e4f1' move completed: OK
I20240502 01:54:14.720927 20682 rebalancer_tool.cc:1136] tablet 5df83f83d2674155afba93c2d992af0a: 'dfa1152987644bd781b46c7e42a7a685' -> 'fdc2ac372492454c85387589a91f63af' move completed: OK
I20240502 01:54:14.832139 20682 rebalancer_tool.cc:1136] tablet 68bc23294ac64c9486bffebbdb16b508: 'fdc2ac372492454c85387589a91f63af' -> '3fceccefba274b1dbbd8216af8b0bc29' move completed: OK
I20240502 01:54:15.058986 20682 rebalancer_tool.cc:1136] tablet 4aefc61cb95f4e949e7a23a1c677749e: '3fceccefba274b1dbbd8216af8b0bc29' -> 'dfa1152987644bd781b46c7e42a7a685' move completed: OK
I20240502 01:54:15.193725 20682 rebalancer_tool.cc:1074] tablet 71cbc012efe14fdb937302796a9f84e5: 'dfa1152987644bd781b46c7e42a7a685' -> '2f306549407b4bb281ccf2994035e4f1' move scheduled
I20240502 01:54:15.356683 20682 rebalancer_tool.cc:1074] tablet 8eed067f639542d487de971761e91931: '6a36c952c51542c0bf7ea3ec2857a9f9' -> 'dfa1152987644bd781b46c7e42a7a685' move scheduled
I20240502 01:54:15.384065 20682 rebalancer_tool.cc:1074] tablet 61ab80ce377a44ac95a9c574e9a615f2: 'fdc2ac372492454c85387589a91f63af' -> 'dfa1152987644bd781b46c7e42a7a685' move scheduled
I20240502 01:54:15.507345 20682 rebalancer_tool.cc:1136] tablet 1b14ac3413004127b72904b631ec1b94: 'f19cc6cb1e76460aa6711e49a5775091' -> 'dfa1152987644bd781b46c7e42a7a685' move completed: OK
I20240502 01:54:15.517870 20682 rebalancer_tool.cc:1136] tablet cf0237a4c65b495ba892871dd850bed9: 'f19cc6cb1e76460aa6711e49a5775091' -> 'bb31bceba76d466db24b9341c04900c3' move completed: OK
I20240502 01:54:15.542402 20682 rebalancer_tool.cc:1074] tablet 0618761d30f34cfe9650d13c9275cdcb: 'dfa1152987644bd781b46c7e42a7a685' -> 'f19cc6cb1e76460aa6711e49a5775091' move scheduled
I20240502 01:54:16.494515 20682 rebalancer_tool.cc:1136] tablet 1fdd104f1a28446cbc25d4ca16d03d7d: 'dfa1152987644bd781b46c7e42a7a685' -> '2f306549407b4bb281ccf2994035e4f1' move completed: OK
I20240502 01:54:16.504969 20682 rebalancer_tool.cc:1136] tablet ee418fc4028a422eabac02ed63ea9047: '6a36c952c51542c0bf7ea3ec2857a9f9' -> 'bb31bceba76d466db24b9341c04900c3' move completed: OK
I20240502 01:54:16.528328 20682 rebalancer_tool.cc:1074] tablet 88e68acb7de346f0ad165cadabd077be: '6a36c952c51542c0bf7ea3ec2857a9f9' -> 'dfa1152987644bd781b46c7e42a7a685' move scheduled
I20240502 01:54:16.603428 20682 rebalancer_tool.cc:1136] tablet 29d8cb76dc514e7b9002bd2291d8ee8d: '6a36c952c51542c0bf7ea3ec2857a9f9' -> 'fdc2ac372492454c85387589a91f63af' move completed: OK
I20240502 01:54:17.479820 20682 rebalancer_tool.cc:1136] tablet f3e4c8f98b5e48e3ba6affc2df08943a: 'fdc2ac372492454c85387589a91f63af' -> 'f19cc6cb1e76460aa6711e49a5775091' move completed: OK
I20240502 01:54:17.493253 20682 rebalancer_tool.cc:1136] tablet 0618761d30f34cfe9650d13c9275cdcb: 'dfa1152987644bd781b46c7e42a7a685' -> 'f19cc6cb1e76460aa6711e49a5775091' move completed: OK
I20240502 01:54:17.668349 20682 rebalancer_tool.cc:1074] tablet 03bd47783e884a70bfea1dfab4f2e563: 'dfa1152987644bd781b46c7e42a7a685' -> '6a36c952c51542c0bf7ea3ec2857a9f9' move scheduled
I20240502 01:54:18.103227 20682 rebalancer_tool.cc:1136] tablet 88e68acb7de346f0ad165cadabd077be: '6a36c952c51542c0bf7ea3ec2857a9f9' -> 'dfa1152987644bd781b46c7e42a7a685' move completed: OK
I20240502 01:54:18.837579 20682 rebalancer_tool.cc:1136] tablet 61ab80ce377a44ac95a9c574e9a615f2: 'fdc2ac372492454c85387589a91f63af' -> 'dfa1152987644bd781b46c7e42a7a685' move completed: OK
I20240502 01:54:18.859977 20682 rebalancer_tool.cc:1136] tablet 8eed067f639542d487de971761e91931: '6a36c952c51542c0bf7ea3ec2857a9f9' -> 'dfa1152987644bd781b46c7e42a7a685' move completed: OK
I20240502 01:54:18.905297 20682 rebalancer_tool.cc:1136] tablet 71cbc012efe14fdb937302796a9f84e5: 'dfa1152987644bd781b46c7e42a7a685' -> '2f306549407b4bb281ccf2994035e4f1' move completed: OK
I20240502 01:54:19.197681 20682 rebalancer_tool.cc:1136] tablet 03bd47783e884a70bfea1dfab4f2e563: 'dfa1152987644bd781b46c7e42a7a685' -> '6a36c952c51542c0bf7ea3ec2857a9f9' move completed: OK
W20240502 01:54:19.869901 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a95d02d60f11435a8aafb0c271c0aff0: could not find any suitable replica to move from server dfa1152987644bd781b46c7e42a7a685 to server fdc2ac372492454c85387589a91f63af
W20240502 01:54:19.870069 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 26c9a097a1a4464a9d467639ce3fe653: could not find any suitable replica to move from server 6a36c952c51542c0bf7ea3ec2857a9f9 to server bb31bceba76d466db24b9341c04900c3
W20240502 01:54:19.870308 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 516d492205794b72a52193ce4c29c429: could not find any suitable replica to move from server 2f306549407b4bb281ccf2994035e4f1 to server f19cc6cb1e76460aa6711e49a5775091
W20240502 01:54:19.870468 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b994be1897424e729195614225fa1ffb: could not find any suitable replica to move from server f19cc6cb1e76460aa6711e49a5775091 to server 2f306549407b4bb281ccf2994035e4f1
W20240502 01:54:19.870718 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bc3f32f6efe0449a852946df8151762f: could not find any suitable replica to move from server 3fceccefba274b1dbbd8216af8b0bc29 to server 2f306549407b4bb281ccf2994035e4f1
W20240502 01:54:19.870857 20682 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 04407f9191884214866d42214a1a1beb: could not find any suitable replica to move from server 2f306549407b4bb281ccf2994035e4f1 to server bb31bceba76d466db24b9341c04900c3
I20240502 01:54:19.901384 20682 rebalancer_tool.cc:1074] tablet ee28f4aa31504dbfac977c4178aefdb1: '2f306549407b4bb281ccf2994035e4f1' -> 'bb31bceba76d466db24b9341c04900c3' move scheduled
I20240502 01:54:19.929309 20682 rebalancer_tool.cc:1074] tablet fa4d38dc9b674a9b90d4793dad2687ec: '6a36c952c51542c0bf7ea3ec2857a9f9' -> 'dfa1152987644bd781b46c7e42a7a685' move scheduled
I20240502 01:54:19.969170 20682 rebalancer_tool.cc:1074] tablet f6cac9478db94a21a9799b80d24c8dc9: 'fdc2ac372492454c85387589a91f63af' -> '4da690c0fa13407a85c0689ce8de9ade' move scheduled
I20240502 01:54:20.017187 20682 rebalancer_tool.cc:1074] tablet 29fb2b31cd9c4c8fb5923bb849a097d9: 'f19cc6cb1e76460aa6711e49a5775091' -> '3fceccefba274b1dbbd8216af8b0bc29' move scheduled
I20240502 01:54:20.053857 20682 rebalancer_tool.cc:1074] tablet bbf05e3b3ad44354be7600c3c07756fd: '6a36c952c51542c0bf7ea3ec2857a9f9' -> '2f306549407b4bb281ccf2994035e4f1' move scheduled
I20240502 01:54:20.092455 20682 rebalancer_tool.cc:1074] tablet 0615786e80314766892988f3cff6a74d: 'bb31bceba76d466db24b9341c04900c3' -> 'fdc2ac372492454c85387589a91f63af' move scheduled
I20240502 01:54:20.117256 20682 rebalancer_tool.cc:1074] tablet a86548104dca43bd9e1b3ec4d23ec0e4: 'dfa1152987644bd781b46c7e42a7a685' -> '4da690c0fa13407a85c0689ce8de9ade' move scheduled
I20240502 01:54:20.162439 20682 rebalancer_tool.cc:1074] tablet 495700f8ed5e41228f2e597465dad126: 'f19cc6cb1e76460aa6711e49a5775091' -> 'bb31bceba76d466db24b9341c04900c3' move scheduled
I20240502 01:54:20.206621 20682 rebalancer_tool.cc:1074] tablet 6ded7434afab44999620dfa3213c7a82: '3fceccefba274b1dbbd8216af8b0bc29' -> '6a36c952c51542c0bf7ea3ec2857a9f9' move scheduled
I20240502 01:54:20.244225 20682 rebalancer_tool.cc:1074] tablet 38bb8dd915f743f49b26ee2572871375: '2f306549407b4bb281ccf2994035e4f1' -> 'fdc2ac372492454c85387589a91f63af' move scheduled
I20240502 01:54:20.299003 20682 rebalancer_tool.cc:1074] tablet 0399c9d1d7b2408292dc78c185605ee8: '3fceccefba274b1dbbd8216af8b0bc29' -> 'dfa1152987644bd781b46c7e42a7a685' move scheduled
I20240502 01:54:20.348814 20682 rebalancer_tool.cc:1074] tablet 25ad701a0fec4e80b37df5dec598796a: '4da690c0fa13407a85c0689ce8de9ade' -> 'f19cc6cb1e76460aa6711e49a5775091' move scheduled
I20240502 01:54:20.398757 20682 rebalancer_tool.cc:1074] tablet e61094fdf9324469afa7ecdc45fd55de: 'bb31bceba76d466db24b9341c04900c3' -> '6a36c952c51542c0bf7ea3ec2857a9f9' move scheduled
I20240502 01:54:20.449481 20682 rebalancer_tool.cc:1074] tablet 3dcd890bdd2d413e95401897e9f1bcab: '2f306549407b4bb281ccf2994035e4f1' -> 'dfa1152987644bd781b46c7e42a7a685' move scheduled
I20240502 01:54:20.494933 20682 rebalancer_tool.cc:1074] tablet 7b2eae408b9843acaa61e698a3e44cff: '3fceccefba274b1dbbd8216af8b0bc29' -> 'fdc2ac372492454c85387589a91f63af' move scheduled
I20240502 01:54:20.535266 20682 rebalancer_tool.cc:1074] tablet a390a222562f425eabb11d7f52cbe9a6: '4da690c0fa13407a85c0689ce8de9ade' -> 'f19cc6cb1e76460aa6711e49a5775091' move scheduled
I20240502 01:54:20.657068 20682 rebalancer_tool.cc:1074] tablet 28463d45bfde44429d56179c1eec56b0: 'bb31bceba76d466db24b9341c04900c3' -> 'fdc2ac372492454c85387589a91f63af' move scheduled
I20240502 01:54:20.808385 20682 rebalancer_tool.cc:1074] tablet f3e4c8f98b5e48e3ba6affc2df08943a: '6a36c952c51542c0bf7ea3ec2857a9f9' -> '3fceccefba274b1dbbd8216af8b0bc29' move scheduled
I20240502 01:54:20.991300 20682 rebalancer_tool.cc:1074] tablet 2544ba74e9d146ef963fd92a1a6f8a98: '2f306549407b4bb281ccf2994035e4f1' -> 'dfa1152987644bd781b46c7e42a7a685' move scheduled
I20240502 01:54:21.130491 20682 rebalancer_tool.cc:1074] tablet adc2b57af23c41aebd1258ff8b60d074: 'f19cc6cb1e76460aa6711e49a5775091' -> '4da690c0fa13407a85c0689ce8de9ade' move scheduled
W20240502 01:54:21.594341 25784 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 01:54:21.322082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.42.69:34339 (local address 127.0.0.1:43200)
0502 01:54:21.326699 (+ 4617us) negotiation.cc:107] Waiting for socket to connect
0502 01:54:21.326717 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0502 01:54:21.326846 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 01:54:21.592996 (+266150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":4510,"thread_start_us":4981,"threads_started":1}
I20240502 01:54:21.595023 20682 rebalancer_tool.cc:1123] tablet 2544ba74e9d146ef963fd92a1a6f8a98: '2f306549407b4bb281ccf2994035e4f1' -> 'dfa1152987644bd781b46c7e42a7a685' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:54:21.793628 25828 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 01:54:21.609488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.42.72:42411 (local address 127.0.0.1:39072)
0502 01:54:21.609991 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0502 01:54:21.610007 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0502 01:54:21.610117 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 01:54:21.793403 (+183286us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":268,"threads_started":1}
I20240502 01:54:21.794395 20682 rebalancer_tool.cc:1123] tablet f3e4c8f98b5e48e3ba6affc2df08943a: '6a36c952c51542c0bf7ea3ec2857a9f9' -> '3fceccefba274b1dbbd8216af8b0bc29' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:54:21.808552 20682 rebalancer_tool.cc:1123] tablet a390a222562f425eabb11d7f52cbe9a6: '4da690c0fa13407a85c0689ce8de9ade' -> 'f19cc6cb1e76460aa6711e49a5775091' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
I20240502 01:54:21.864404 20682 rebalancer_tool.cc:1123] tablet 25ad701a0fec4e80b37df5dec598796a: '4da690c0fa13407a85c0689ce8de9ade' -> 'f19cc6cb1e76460aa6711e49a5775091' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
I20240502 01:54:21.871371 20682 rebalancer_tool.cc:1123] tablet 38bb8dd915f743f49b26ee2572871375: '2f306549407b4bb281ccf2994035e4f1' -> 'fdc2ac372492454c85387589a91f63af' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
I20240502 01:54:22.116014 20682 rebalancer_tool.cc:1136] tablet 495700f8ed5e41228f2e597465dad126: 'f19cc6cb1e76460aa6711e49a5775091' -> 'bb31bceba76d466db24b9341c04900c3' move completed: OK
I20240502 01:54:22.129199 20682 rebalancer_tool.cc:1136] tablet a86548104dca43bd9e1b3ec4d23ec0e4: 'dfa1152987644bd781b46c7e42a7a685' -> '4da690c0fa13407a85c0689ce8de9ade' move completed: OK
I20240502 01:54:22.172873 20682 rebalancer_tool.cc:1123] tablet bbf05e3b3ad44354be7600c3c07756fd: '6a36c952c51542c0bf7ea3ec2857a9f9' -> '2f306549407b4bb281ccf2994035e4f1' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
I20240502 01:54:22.194267 20682 rebalancer_tool.cc:1136] tablet 29fb2b31cd9c4c8fb5923bb849a097d9: 'f19cc6cb1e76460aa6711e49a5775091' -> '3fceccefba274b1dbbd8216af8b0bc29' move completed: OK
I20240502 01:54:22.198951 20682 rebalancer_tool.cc:1123] tablet f6cac9478db94a21a9799b80d24c8dc9: 'fdc2ac372492454c85387589a91f63af' -> '4da690c0fa13407a85c0689ce8de9ade' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
I20240502 01:54:22.212273 20682 rebalancer_tool.cc:1123] tablet ee28f4aa31504dbfac977c4178aefdb1: '2f306549407b4bb281ccf2994035e4f1' -> 'bb31bceba76d466db24b9341c04900c3' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
I20240502 01:54:22.217947 20682 rebalancer_tool.cc:1084] tablet 074a08b2b64640eb9784b6a23921ce8d: 'f19cc6cb1e76460aa6711e49a5775091' -> 'bb31bceba76d466db24b9341c04900c3' move ignored: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
I20240502 01:54:22.218096 20682 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): unacceptable health status UNAVAILABLE
I20240502 01:54:23.790644 11433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu pid 17394
W20240502 01:54:23.799950 17542 consensus_peers.cc:462] T 3dcd890bdd2d413e95401897e9f1bcab P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
I20240502 01:54:23.814816 17515 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 01:54:23.821249 20303 consensus_peers.cc:485] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:54:23.838461 17802 consensus_peers.cc:485] T 7c697210a14d4982a939ea7965cb769d P bb31bceba76d466db24b9341c04900c3 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:54:23.864027 20303 consensus_peers.cc:485] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:54:23.869271 17543 consensus_peers.cc:485] T 30f43088bd0f48ee86907df16194b9be P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:54:23.880347 17802 consensus_peers.cc:485] T 65bdab2c9fba48a6828a864278319cc0 P bb31bceba76d466db24b9341c04900c3 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:54:23.882903 17802 consensus_peers.cc:485] T f0b3862e504d4b43a60477e5efef29b1 P bb31bceba76d466db24b9341c04900c3 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:54:23.894902 20433 consensus_peers.cc:485] T 246cec994d274786aad7672f2a30e5d8 P 2f306549407b4bb281ccf2994035e4f1 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:54:23.895074 20303 consensus_peers.cc:485] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:54:23.900918 17543 consensus_peers.cc:485] T 049f5bd8759c488789b7c396985be3f9 P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:54:23.904006 20302 consensus_peers.cc:485] T aebde87a7f114537abd9c1d8b7e97004 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:54:23.909368 20433 consensus_peers.cc:485] T 8e21dfeaf732445098f5b17a0615bb8b P 2f306549407b4bb281ccf2994035e4f1 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:54:23.915627 25604 raft_consensus.cc:482] T 62887e332443457f8a5b41a982354dc4 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdc2ac372492454c85387589a91f63af)
I20240502 01:54:23.915812 25604 raft_consensus.cc:504] T 62887e332443457f8a5b41a982354dc4 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } }
W20240502 01:54:23.918272 17802 leader_election.cc:334] T 62887e332443457f8a5b41a982354dc4 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
I20240502 01:54:23.918447 25604 leader_election.cc:288] T 62887e332443457f8a5b41a982354dc4 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
W20240502 01:54:23.922955 17803 leader_election.cc:334] T 62887e332443457f8a5b41a982354dc4 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
I20240502 01:54:23.923063 17803 leader_election.cc:302] T 62887e332443457f8a5b41a982354dc4 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb31bceba76d466db24b9341c04900c3; no voters: f19cc6cb1e76460aa6711e49a5775091, fdc2ac372492454c85387589a91f63af
I20240502 01:54:23.923442 25604 raft_consensus.cc:2726] T 62887e332443457f8a5b41a982354dc4 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:54:23.936089 20434 consensus_peers.cc:485] T cb7d51b2b9f84d608d67e771cd958231 P 2f306549407b4bb281ccf2994035e4f1 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:54:23.941733 20302 consensus_peers.cc:485] T 61ab80ce377a44ac95a9c574e9a615f2 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:54:23.976693 20303 consensus_peers.cc:462] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
I20240502 01:54:23.984704 25604 raft_consensus.cc:482] T 8c129f9289024ec9898fbbc06b2fb2cd P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdc2ac372492454c85387589a91f63af)
W20240502 01:54:23.984882 20303 consensus_peers.cc:485] T f537e32b31264def980094906bab350c P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:54:23.984884 25604 raft_consensus.cc:504] T 8c129f9289024ec9898fbbc06b2fb2cd P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:23.985519 25604 leader_election.cc:288] T 8c129f9289024ec9898fbbc06b2fb2cd P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
W20240502 01:54:23.987529 17802 leader_election.cc:334] T 8c129f9289024ec9898fbbc06b2fb2cd P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
W20240502 01:54:23.987776 17803 consensus_peers.cc:485] T ed1522ec150a46f28fd553f9b563651f P bb31bceba76d466db24b9341c04900c3 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:54:23.987890 17803 leader_election.cc:334] T 8c129f9289024ec9898fbbc06b2fb2cd P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
I20240502 01:54:23.988001 17803 leader_election.cc:302] T 8c129f9289024ec9898fbbc06b2fb2cd P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb31bceba76d466db24b9341c04900c3; no voters: dfa1152987644bd781b46c7e42a7a685, fdc2ac372492454c85387589a91f63af
I20240502 01:54:23.988248 25604 raft_consensus.cc:2726] T 8c129f9289024ec9898fbbc06b2fb2cd P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:54:24.006222 17321 catalog_manager.cc:4573] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet adc2b57af23c41aebd1258ff8b60d074: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
W20240502 01:54:24.040591 17803 consensus_peers.cc:485] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:54:24.051473 17542 consensus_peers.cc:485] T 049f5bd8759c488789b7c396985be3f9 P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:54:24.063166 20433 consensus_peers.cc:485] T acbd240a2783424e96bd589f0232daf6 P 2f306549407b4bb281ccf2994035e4f1 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:54:24.083644 20433 consensus_peers.cc:485] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:54:24.109253 20434 consensus_peers.cc:485] T d10d76ac7b094fe9b230b1afcbed9ca6 P 2f306549407b4bb281ccf2994035e4f1 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:54:24.117653 20303 consensus_peers.cc:485] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:54:24.147739 17542 consensus_peers.cc:485] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:54:24.152685 17802 consensus_peers.cc:485] T 0618761d30f34cfe9650d13c9275cdcb P bb31bceba76d466db24b9341c04900c3 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:54:24.167368 20303 consensus_peers.cc:485] T 52b83e43cae646a29bbf3b826fbbcfb6 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:54:24.180903 26066 raft_consensus.cc:482] T 88e68acb7de346f0ad165cadabd077be P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdc2ac372492454c85387589a91f63af)
I20240502 01:54:24.181071 26066 raft_consensus.cc:504] T 88e68acb7de346f0ad165cadabd077be P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:24.181628 26066 leader_election.cc:288] T 88e68acb7de346f0ad165cadabd077be P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
W20240502 01:54:24.183099 20434 leader_election.cc:334] T 88e68acb7de346f0ad165cadabd077be P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
W20240502 01:54:24.185240 20433 leader_election.cc:334] T 88e68acb7de346f0ad165cadabd077be P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
I20240502 01:54:24.185338 20433 leader_election.cc:302] T 88e68acb7de346f0ad165cadabd077be P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1; no voters: dfa1152987644bd781b46c7e42a7a685, fdc2ac372492454c85387589a91f63af
I20240502 01:54:24.185534 26066 raft_consensus.cc:2726] T 88e68acb7de346f0ad165cadabd077be P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:54:24.189250 17542 consensus_peers.cc:462] T 3dcd890bdd2d413e95401897e9f1bcab P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
W20240502 01:54:24.192575 20302 consensus_peers.cc:485] T ebb9bca9f1bd41b0857939909b1110e4 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:54:24.220846 20433 consensus_peers.cc:485] T 4bb42883b1a0475b863d008bd01fa70a P 2f306549407b4bb281ccf2994035e4f1 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:54:24.222231 17542 consensus_peers.cc:485] T 6aacf7c4f19546f581c9a54ad171916b P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:54:24.225606 11433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu with pid 17394
W20240502 01:54:24.227456 20302 consensus_peers.cc:485] T 5824db6154734b0f8f9094ae46633771 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:54:24.268325 20433 consensus_peers.cc:485] T 6ded7434afab44999620dfa3213c7a82 P 2f306549407b4bb281ccf2994035e4f1 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:54:24.281494 17802 connection.cc:535] client connection to 127.11.42.65:38321 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:54:24.281842 20302 connection.cc:535] client connection to 127.11.42.65:38321 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:54:24.282231 11433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu pid 17525
W20240502 01:54:24.284904 20302 consensus_peers.cc:485] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Couldn't send request to peer 4da690c0fa13407a85c0689ce8de9ade. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.65:38321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:24.285070 20302 consensus_peers.cc:485] T 71cbc012efe14fdb937302796a9f84e5 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Couldn't send request to peer 4da690c0fa13407a85c0689ce8de9ade. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.65:38321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:24.285177 20302 consensus_peers.cc:485] T 472d7827b3074236a81a3ede098f6c0a P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Couldn't send request to peer 4da690c0fa13407a85c0689ce8de9ade. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.65:38321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:24.285248 20302 consensus_peers.cc:485] T c491a39ca87847d9ace688346c0baf57 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Couldn't send request to peer 4da690c0fa13407a85c0689ce8de9ade. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.65:38321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:24.285324 20302 consensus_peers.cc:485] T 1d8e2e0f6ed34e4cb004e1c5309c0f1f P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Couldn't send request to peer 4da690c0fa13407a85c0689ce8de9ade. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.65:38321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:24.287550 17802 consensus_peers.cc:485] T 2152c07e448b4582abff22b034363c70 P bb31bceba76d466db24b9341c04900c3 -> Peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Couldn't send request to peer 4da690c0fa13407a85c0689ce8de9ade. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.65:38321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:24.291738 17542 consensus_peers.cc:485] T 30f43088bd0f48ee86907df16194b9be P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Couldn't send request to peer 4da690c0fa13407a85c0689ce8de9ade. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.65:38321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:24.291846 17542 consensus_peers.cc:485] T 5152c002eadd4d54ab10a093bce45f17 P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Couldn't send request to peer 4da690c0fa13407a85c0689ce8de9ade. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.65:38321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:24.291910 17542 consensus_peers.cc:485] T 10ec78a08ab04678897a6892988e515f P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Couldn't send request to peer 4da690c0fa13407a85c0689ce8de9ade. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.65:38321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:24.291985 17542 consensus_peers.cc:485] T fa4d38dc9b674a9b90d4793dad2687ec P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Couldn't send request to peer 4da690c0fa13407a85c0689ce8de9ade. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.65:38321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:24.292043 17542 consensus_peers.cc:485] T 975f253ff03447ff96fe84169c05097e P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Couldn't send request to peer 4da690c0fa13407a85c0689ce8de9ade. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.65:38321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:24.292095 17542 consensus_peers.cc:485] T bffb870715a243d992107583d7c17f1b P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Couldn't send request to peer 4da690c0fa13407a85c0689ce8de9ade. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.65:38321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:24.292156 17542 consensus_peers.cc:485] T a86548104dca43bd9e1b3ec4d23ec0e4 P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Couldn't send request to peer 4da690c0fa13407a85c0689ce8de9ade. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.65:38321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:24.292668 17802 consensus_peers.cc:485] T 062b9d90fb0444f4be37bf09770c9caf P bb31bceba76d466db24b9341c04900c3 -> Peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Couldn't send request to peer 4da690c0fa13407a85c0689ce8de9ade. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.65:38321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:24.292793 17647 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 01:54:24.292878 20433 connection.cc:535] client connection to 127.11.42.65:38321 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:54:24.295622 20433 consensus_peers.cc:485] T 25ad701a0fec4e80b37df5dec598796a P 2f306549407b4bb281ccf2994035e4f1 -> Peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Couldn't send request to peer 4da690c0fa13407a85c0689ce8de9ade. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.65:38321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:24.295727 20433 consensus_peers.cc:485] T adc2b57af23c41aebd1258ff8b60d074 P 2f306549407b4bb281ccf2994035e4f1 -> Peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Couldn't send request to peer 4da690c0fa13407a85c0689ce8de9ade. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.65:38321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:24.295790 20433 consensus_peers.cc:485] T ee418fc4028a422eabac02ed63ea9047 P 2f306549407b4bb281ccf2994035e4f1 -> Peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Couldn't send request to peer 4da690c0fa13407a85c0689ce8de9ade. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.65:38321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:24.295850 20433 consensus_peers.cc:485] T 1423ea912bfc4472acb2b6798b3a99e9 P 2f306549407b4bb281ccf2994035e4f1 -> Peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Couldn't send request to peer 4da690c0fa13407a85c0689ce8de9ade. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.65:38321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:24.296001 20433 consensus_peers.cc:485] T b29218a879e74895bfe00021ad358aa4 P 2f306549407b4bb281ccf2994035e4f1 -> Peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Couldn't send request to peer 4da690c0fa13407a85c0689ce8de9ade. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.65:38321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:24.296073 20433 consensus_peers.cc:485] T 7b2eae408b9843acaa61e698a3e44cff P 2f306549407b4bb281ccf2994035e4f1 -> Peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Couldn't send request to peer 4da690c0fa13407a85c0689ce8de9ade. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.65:38321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:24.296136 20433 consensus_peers.cc:485] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1 -> Peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Couldn't send request to peer 4da690c0fa13407a85c0689ce8de9ade. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.65:38321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:24.296196 20433 consensus_peers.cc:485] T fac0ba26fabd4643bc38b7559faa471c P 2f306549407b4bb281ccf2994035e4f1 -> Peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Couldn't send request to peer 4da690c0fa13407a85c0689ce8de9ade. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.65:38321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:24.296257 20433 consensus_peers.cc:485] T 2ea539ebb37f42f68c166242a06f8d2e P 2f306549407b4bb281ccf2994035e4f1 -> Peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Couldn't send request to peer 4da690c0fa13407a85c0689ce8de9ade. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.65:38321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:24.307977 20433 consensus_peers.cc:485] T 139304da92ab4f48b8a08fa24ed60c95 P 2f306549407b4bb281ccf2994035e4f1 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:54:24.309332 25604 raft_consensus.cc:482] T 4aefc61cb95f4e949e7a23a1c677749e P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f19cc6cb1e76460aa6711e49a5775091)
I20240502 01:54:24.309500 25604 raft_consensus.cc:504] T 4aefc61cb95f4e949e7a23a1c677749e P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: false } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:24.310101 25604 leader_election.cc:288] T 4aefc61cb95f4e949e7a23a1c677749e P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
W20240502 01:54:24.311537 17802 leader_election.cc:334] T 4aefc61cb95f4e949e7a23a1c677749e P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
W20240502 01:54:24.311722 17802 leader_election.cc:334] T 4aefc61cb95f4e949e7a23a1c677749e P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
I20240502 01:54:24.311841 17802 leader_election.cc:302] T 4aefc61cb95f4e949e7a23a1c677749e P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb31bceba76d466db24b9341c04900c3; no voters: dfa1152987644bd781b46c7e42a7a685, f19cc6cb1e76460aa6711e49a5775091
I20240502 01:54:24.312085 25604 raft_consensus.cc:2726] T 4aefc61cb95f4e949e7a23a1c677749e P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:54:24.324945 20433 consensus_peers.cc:485] T 617443200e2c47b69d63ef056294f40c P 2f306549407b4bb281ccf2994035e4f1 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:54:24.374162 20302 consensus_peers.cc:485] T 59be89ca2c0b4275844f1e73cd20595b P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:54:24.390107 25604 raft_consensus.cc:482] T ee28f4aa31504dbfac977c4178aefdb1 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f19cc6cb1e76460aa6711e49a5775091)
I20240502 01:54:24.390297 25604 raft_consensus.cc:504] T ee28f4aa31504dbfac977c4178aefdb1 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } }
I20240502 01:54:24.390905 25604 leader_election.cc:288] T ee28f4aa31504dbfac977c4178aefdb1 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
W20240502 01:54:24.392740 17802 leader_election.cc:334] T ee28f4aa31504dbfac977c4178aefdb1 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
W20240502 01:54:24.393487 17802 leader_election.cc:334] T ee28f4aa31504dbfac977c4178aefdb1 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
I20240502 01:54:24.393595 17802 leader_election.cc:302] T ee28f4aa31504dbfac977c4178aefdb1 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb31bceba76d466db24b9341c04900c3; no voters: dfa1152987644bd781b46c7e42a7a685, f19cc6cb1e76460aa6711e49a5775091
I20240502 01:54:24.393890 25604 raft_consensus.cc:2726] T ee28f4aa31504dbfac977c4178aefdb1 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:54:24.425650 20302 consensus_peers.cc:485] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:54:24.452646 17802 consensus_peers.cc:485] T 2152c07e448b4582abff22b034363c70 P bb31bceba76d466db24b9341c04900c3 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:54:24.467372 20303 consensus_peers.cc:462] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
I20240502 01:54:24.560741 26066 raft_consensus.cc:482] T 29d8cb76dc514e7b9002bd2291d8ee8d P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dfa1152987644bd781b46c7e42a7a685)
I20240502 01:54:24.560904 26066 raft_consensus.cc:504] T 29d8cb76dc514e7b9002bd2291d8ee8d P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } }
I20240502 01:54:24.561569 26066 leader_election.cc:288] T 29d8cb76dc514e7b9002bd2291d8ee8d P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
W20240502 01:54:24.562728 20433 leader_election.cc:334] T 29d8cb76dc514e7b9002bd2291d8ee8d P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
W20240502 01:54:24.563117 20434 leader_election.cc:334] T 29d8cb76dc514e7b9002bd2291d8ee8d P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
I20240502 01:54:24.563227 20434 leader_election.cc:302] T 29d8cb76dc514e7b9002bd2291d8ee8d P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1; no voters: dfa1152987644bd781b46c7e42a7a685, fdc2ac372492454c85387589a91f63af
I20240502 01:54:24.563441 26066 raft_consensus.cc:2726] T 29d8cb76dc514e7b9002bd2291d8ee8d P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:54:24.570632 20302 consensus_peers.cc:485] T c491a39ca87847d9ace688346c0baf57 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:54:24.590206 26153 raft_consensus.cc:482] T 8e7f88b6c509477fb0900024e7de8f00 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dfa1152987644bd781b46c7e42a7a685)
I20240502 01:54:24.590353 26153 raft_consensus.cc:504] T 8e7f88b6c509477fb0900024e7de8f00 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } }
I20240502 01:54:24.591228 26153 leader_election.cc:288] T 8e7f88b6c509477fb0900024e7de8f00 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
W20240502 01:54:24.592687 20302 leader_election.cc:334] T 8e7f88b6c509477fb0900024e7de8f00 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
W20240502 01:54:24.592830 20303 leader_election.cc:334] T 8e7f88b6c509477fb0900024e7de8f00 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
I20240502 01:54:24.592929 20303 leader_election.cc:302] T 8e7f88b6c509477fb0900024e7de8f00 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9; no voters: dfa1152987644bd781b46c7e42a7a685, fdc2ac372492454c85387589a91f63af
I20240502 01:54:24.593159 26153 raft_consensus.cc:2726] T 8e7f88b6c509477fb0900024e7de8f00 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:24.602044 26151 raft_consensus.cc:482] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f19cc6cb1e76460aa6711e49a5775091)
I20240502 01:54:24.602175 26151 raft_consensus.cc:504] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:24.602676 26151 leader_election.cc:288] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
W20240502 01:54:24.603938 17802 leader_election.cc:334] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
W20240502 01:54:24.604104 17802 leader_election.cc:334] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
I20240502 01:54:24.604183 17802 leader_election.cc:302] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb31bceba76d466db24b9341c04900c3; no voters: dfa1152987644bd781b46c7e42a7a685, f19cc6cb1e76460aa6711e49a5775091
I20240502 01:54:24.604393 26151 raft_consensus.cc:2726] T bfd7b1d1002c4b57a14d43ed9c6d9f50 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:24.611451 26066 raft_consensus.cc:482] T 38bb8dd915f743f49b26ee2572871375 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f19cc6cb1e76460aa6711e49a5775091)
I20240502 01:54:24.611600 26066 raft_consensus.cc:504] T 38bb8dd915f743f49b26ee2572871375 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: NON_VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { promote: true } }
I20240502 01:54:24.612242 26066 leader_election.cc:288] T 38bb8dd915f743f49b26ee2572871375 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
W20240502 01:54:24.613737 20433 leader_election.cc:334] T 38bb8dd915f743f49b26ee2572871375 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
W20240502 01:54:24.613968 20433 leader_election.cc:334] T 38bb8dd915f743f49b26ee2572871375 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
I20240502 01:54:24.614095 20433 leader_election.cc:302] T 38bb8dd915f743f49b26ee2572871375 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1; no voters: dfa1152987644bd781b46c7e42a7a685, f19cc6cb1e76460aa6711e49a5775091
I20240502 01:54:24.614375 26066 raft_consensus.cc:2726] T 38bb8dd915f743f49b26ee2572871375 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:54:24.716027 17542 consensus_peers.cc:485] T 43572b8f8fba433ab00b046c769d886d P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:54:24.716174 17542 consensus_peers.cc:485] T e61094fdf9324469afa7ecdc45fd55de P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:54:24.718534 26153 raft_consensus.cc:482] T b9d82510030b4503ac892175a8bb1bb7 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f19cc6cb1e76460aa6711e49a5775091)
I20240502 01:54:24.718673 26153 raft_consensus.cc:504] T b9d82510030b4503ac892175a8bb1bb7 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } }
I20240502 01:54:24.719259 26153 leader_election.cc:288] T b9d82510030b4503ac892175a8bb1bb7 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
I20240502 01:54:24.724205 26155 raft_consensus.cc:482] T f6cac9478db94a21a9799b80d24c8dc9 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdc2ac372492454c85387589a91f63af)
I20240502 01:54:24.724349 26155 raft_consensus.cc:504] T f6cac9478db94a21a9799b80d24c8dc9 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } attrs { replace: true } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: NON_VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: true } }
I20240502 01:54:24.724944 26155 leader_election.cc:288] T f6cac9478db94a21a9799b80d24c8dc9 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411)
W20240502 01:54:24.738922 17543 leader_election.cc:334] T f6cac9478db94a21a9799b80d24c8dc9 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
W20240502 01:54:24.741914 17542 consensus_peers.cc:485] T 1f01949cf70440b38e59a5020fd48c44 P 3fceccefba274b1dbbd8216af8b0bc29 -> Peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Couldn't send request to peer f19cc6cb1e76460aa6711e49a5775091. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:54:24.742239 17542 leader_election.cc:334] T f6cac9478db94a21a9799b80d24c8dc9 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
I20240502 01:54:24.742352 17542 leader_election.cc:302] T f6cac9478db94a21a9799b80d24c8dc9 P 3fceccefba274b1dbbd8216af8b0bc29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fceccefba274b1dbbd8216af8b0bc29; no voters: f19cc6cb1e76460aa6711e49a5775091, fdc2ac372492454c85387589a91f63af
I20240502 01:54:24.742772 26158 raft_consensus.cc:2726] T f6cac9478db94a21a9799b80d24c8dc9 P 3fceccefba274b1dbbd8216af8b0bc29 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:24.746675 11433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu with pid 17525
W20240502 01:54:24.764598 20302 leader_election.cc:334] T b9d82510030b4503ac892175a8bb1bb7 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
W20240502 01:54:24.764775 20302 leader_election.cc:334] T b9d82510030b4503ac892175a8bb1bb7 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
I20240502 01:54:24.764842 20302 leader_election.cc:302] T b9d82510030b4503ac892175a8bb1bb7 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9; no voters: dfa1152987644bd781b46c7e42a7a685, f19cc6cb1e76460aa6711e49a5775091
I20240502 01:54:24.765194 25358 raft_consensus.cc:2726] T b9d82510030b4503ac892175a8bb1bb7 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:24.775336 25604 raft_consensus.cc:482] T 1a837e7dce89462596b2b6049069ec8e P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f19cc6cb1e76460aa6711e49a5775091)
I20240502 01:54:24.775473 25604 raft_consensus.cc:504] T 1a837e7dce89462596b2b6049069ec8e P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:24.775983 25604 leader_election.cc:288] T 1a837e7dce89462596b2b6049069ec8e P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
W20240502 01:54:24.777257 17802 leader_election.cc:334] T 1a837e7dce89462596b2b6049069ec8e P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
W20240502 01:54:24.777529 17803 leader_election.cc:334] T 1a837e7dce89462596b2b6049069ec8e P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
I20240502 01:54:24.777623 17803 leader_election.cc:302] T 1a837e7dce89462596b2b6049069ec8e P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb31bceba76d466db24b9341c04900c3; no voters: f19cc6cb1e76460aa6711e49a5775091, fdc2ac372492454c85387589a91f63af
I20240502 01:54:24.777896 25604 raft_consensus.cc:2726] T 1a837e7dce89462596b2b6049069ec8e P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:24.779343 25604 raft_consensus.cc:482] T d25f2ab1a7544cec893dd02c7c89285c P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4da690c0fa13407a85c0689ce8de9ade)
I20240502 01:54:24.779484 25604 raft_consensus.cc:504] T d25f2ab1a7544cec893dd02c7c89285c P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:24.779973 25604 leader_election.cc:288] T d25f2ab1a7544cec893dd02c7c89285c P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
W20240502 01:54:24.781620 17802 leader_election.cc:334] T d25f2ab1a7544cec893dd02c7c89285c P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
W20240502 01:54:24.781783 17802 leader_election.cc:334] T d25f2ab1a7544cec893dd02c7c89285c P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Network error: Client connection negotiation failed: client connection to 127.11.42.65:38321: connect: Connection refused (error 111)
I20240502 01:54:24.781854 17802 leader_election.cc:302] T d25f2ab1a7544cec893dd02c7c89285c P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb31bceba76d466db24b9341c04900c3; no voters: 4da690c0fa13407a85c0689ce8de9ade, f19cc6cb1e76460aa6711e49a5775091
I20240502 01:54:24.782064 25604 raft_consensus.cc:2726] T d25f2ab1a7544cec893dd02c7c89285c P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:54:24.811182 20431 connection.cc:535] client connection to 127.11.42.66:38531 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:54:24.811501 17319 connection.cc:535] server connection from 127.11.42.66:45283 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:54:24.811270 20300 connection.cc:535] client connection to 127.11.42.66:38531 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:54:24.811889 20302 connection.cc:535] server connection from 127.11.42.66:36601 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:54:24.812019 11433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu pid 17785
W20240502 01:54:24.812106 17800 connection.cc:535] client connection to 127.11.42.66:38531 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:54:24.812862 20431 consensus_peers.cc:485] T b671a9ceacea4a819f750fd37130be0f P 2f306549407b4bb281ccf2994035e4f1 -> Peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): Couldn't send request to peer 3fceccefba274b1dbbd8216af8b0bc29. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.66:38531: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:24.813210 20300 consensus_peers.cc:485] T ba1e82a39c2e439889da60007372a99b P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): Couldn't send request to peer 3fceccefba274b1dbbd8216af8b0bc29. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.66:38531: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:24.813400 20300 consensus_peers.cc:485] T 2714919062f5465c8cde1a3c4720978a P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): Couldn't send request to peer 3fceccefba274b1dbbd8216af8b0bc29. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.66:38531: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:24.813504 17800 consensus_peers.cc:485] T 57ab576948e346cbbe6715bf53f7c665 P bb31bceba76d466db24b9341c04900c3 -> Peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): Couldn't send request to peer 3fceccefba274b1dbbd8216af8b0bc29. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.66:38531: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:24.822508 17907 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 01:54:24.868952 20431 consensus_peers.cc:485] T 4bb42883b1a0475b863d008bd01fa70a P 2f306549407b4bb281ccf2994035e4f1 -> Peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): Couldn't send request to peer 3fceccefba274b1dbbd8216af8b0bc29. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.66:38531: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:24.880379 20431 consensus_peers.cc:485] T 7e9be84ef6a043818f166f6b47898855 P 2f306549407b4bb281ccf2994035e4f1 -> Peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): Couldn't send request to peer 3fceccefba274b1dbbd8216af8b0bc29. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.66:38531: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:24.882269 20300 consensus_peers.cc:485] T 074a08b2b64640eb9784b6a23921ce8d P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): Couldn't send request to peer 3fceccefba274b1dbbd8216af8b0bc29. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.66:38531: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:24.896070 20300 consensus_peers.cc:485] T 31a4ca0cc44e492fbd13a70ce38add35 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): Couldn't send request to peer 3fceccefba274b1dbbd8216af8b0bc29. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.66:38531: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:24.898888 20431 consensus_peers.cc:485] T 7b2eae408b9843acaa61e698a3e44cff P 2f306549407b4bb281ccf2994035e4f1 -> Peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): Couldn't send request to peer 3fceccefba274b1dbbd8216af8b0bc29. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.66:38531: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:24.913949 20300 consensus_peers.cc:485] T 5824db6154734b0f8f9094ae46633771 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): Couldn't send request to peer 3fceccefba274b1dbbd8216af8b0bc29. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.66:38531: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:24.930176 26162 raft_consensus.cc:482] T 5e8e78bd417d46cdb166087dbfa86446 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4da690c0fa13407a85c0689ce8de9ade)
I20240502 01:54:24.930330 26162 raft_consensus.cc:504] T 5e8e78bd417d46cdb166087dbfa86446 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } }
I20240502 01:54:24.930857 26162 leader_election.cc:288] T 5e8e78bd417d46cdb166087dbfa86446 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
W20240502 01:54:24.932096 20433 leader_election.cc:334] T 5e8e78bd417d46cdb166087dbfa86446 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Network error: Client connection negotiation failed: client connection to 127.11.42.65:38321: connect: Connection refused (error 111)
W20240502 01:54:24.932572 20431 leader_election.cc:334] T 5e8e78bd417d46cdb166087dbfa86446 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): Network error: Client connection negotiation failed: client connection to 127.11.42.66:38531: connect: Connection refused (error 111)
I20240502 01:54:24.932685 20431 leader_election.cc:302] T 5e8e78bd417d46cdb166087dbfa86446 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1; no voters: 3fceccefba274b1dbbd8216af8b0bc29, 4da690c0fa13407a85c0689ce8de9ade
I20240502 01:54:24.932919 26162 raft_consensus.cc:2726] T 5e8e78bd417d46cdb166087dbfa86446 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:54:24.961925 20300 consensus_peers.cc:485] T 472d7827b3074236a81a3ede098f6c0a P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): Couldn't send request to peer 3fceccefba274b1dbbd8216af8b0bc29. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.66:38531: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:25.020381 26162 raft_consensus.cc:482] T da06571e020948059fb06f65f26c831d P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f19cc6cb1e76460aa6711e49a5775091)
I20240502 01:54:25.020551 26162 raft_consensus.cc:504] T da06571e020948059fb06f65f26c831d P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } }
I20240502 01:54:25.021111 26162 leader_election.cc:288] T da06571e020948059fb06f65f26c831d P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
W20240502 01:54:25.022375 20433 leader_election.cc:334] T da06571e020948059fb06f65f26c831d P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
W20240502 01:54:25.022538 20433 leader_election.cc:334] T da06571e020948059fb06f65f26c831d P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
I20240502 01:54:25.022624 20433 leader_election.cc:302] T da06571e020948059fb06f65f26c831d P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1; no voters: dfa1152987644bd781b46c7e42a7a685, f19cc6cb1e76460aa6711e49a5775091
I20240502 01:54:25.022843 26162 raft_consensus.cc:2726] T da06571e020948059fb06f65f26c831d P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:54:25.046556 20303 consensus_peers.cc:462] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
W20240502 01:54:25.050360 17321 catalog_manager.cc:4573] TS f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet adc2b57af23c41aebd1258ff8b60d074: Network error: Client connection negotiation failed: client connection to 127.11.42.72:42411: connect: Connection refused (error 111)
I20240502 01:54:25.057127 25358 raft_consensus.cc:482] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4da690c0fa13407a85c0689ce8de9ade)
I20240502 01:54:25.057265 25358 raft_consensus.cc:504] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } attrs { promote: false } }
I20240502 01:54:25.057790 25358 leader_election.cc:288] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
W20240502 01:54:25.058980 20302 leader_election.cc:334] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Network error: Client connection negotiation failed: client connection to 127.11.42.65:38321: connect: Connection refused (error 111)
W20240502 01:54:25.059129 20302 leader_election.cc:334] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
I20240502 01:54:25.059199 20302 leader_election.cc:302] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9; no voters: 4da690c0fa13407a85c0689ce8de9ade, dfa1152987644bd781b46c7e42a7a685
I20240502 01:54:25.059425 25358 raft_consensus.cc:2726] T 648b1cc4d8b54c5eb0948ca8c1984a1f P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:54:25.075997 20431 consensus_peers.cc:485] T 6ded7434afab44999620dfa3213c7a82 P 2f306549407b4bb281ccf2994035e4f1 -> Peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): Couldn't send request to peer 3fceccefba274b1dbbd8216af8b0bc29. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.66:38531: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:25.080099 20431 consensus_peers.cc:485] T cb7d51b2b9f84d608d67e771cd958231 P 2f306549407b4bb281ccf2994035e4f1 -> Peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): Couldn't send request to peer 3fceccefba274b1dbbd8216af8b0bc29. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.66:38531: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:25.086838 20300 consensus_peers.cc:485] T 0399c9d1d7b2408292dc78c185605ee8 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): Couldn't send request to peer 3fceccefba274b1dbbd8216af8b0bc29. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.66:38531: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:25.100407 26167 raft_consensus.cc:482] T 5b092dd1cc004c26a33e17c28af4d3ba P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4da690c0fa13407a85c0689ce8de9ade)
I20240502 01:54:25.100545 26167 raft_consensus.cc:504] T 5b092dd1cc004c26a33e17c28af4d3ba P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } }
I20240502 01:54:25.101095 26167 leader_election.cc:288] T 5b092dd1cc004c26a33e17c28af4d3ba P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
W20240502 01:54:25.102246 20302 leader_election.cc:334] T 5b092dd1cc004c26a33e17c28af4d3ba P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Network error: Client connection negotiation failed: client connection to 127.11.42.65:38321: connect: Connection refused (error 111)
W20240502 01:54:25.102356 20300 leader_election.cc:334] T 5b092dd1cc004c26a33e17c28af4d3ba P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): Network error: Client connection negotiation failed: client connection to 127.11.42.66:38531: connect: Connection refused (error 111)
I20240502 01:54:25.102442 20300 leader_election.cc:302] T 5b092dd1cc004c26a33e17c28af4d3ba P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9; no voters: 3fceccefba274b1dbbd8216af8b0bc29, 4da690c0fa13407a85c0689ce8de9ade
I20240502 01:54:25.102651 26167 raft_consensus.cc:2726] T 5b092dd1cc004c26a33e17c28af4d3ba P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:54:25.106217 20431 consensus_peers.cc:485] T f3e4c8f98b5e48e3ba6affc2df08943a P 2f306549407b4bb281ccf2994035e4f1 -> Peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): Couldn't send request to peer 3fceccefba274b1dbbd8216af8b0bc29. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.66:38531: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:25.144143 26162 raft_consensus.cc:482] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4da690c0fa13407a85c0689ce8de9ade)
I20240502 01:54:25.144280 26162 raft_consensus.cc:504] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "3fceccefba274b1dbbd8216af8b0bc29" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 38531 } attrs { promote: false } }
I20240502 01:54:25.146564 26162 leader_election.cc:288] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531)
W20240502 01:54:25.147471 20433 leader_election.cc:334] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Network error: Client connection negotiation failed: client connection to 127.11.42.65:38321: connect: Connection refused (error 111)
I20240502 01:54:25.147979 26152 raft_consensus.cc:482] T 7efd29396c7c4fb597be2251cdcdd48f P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4da690c0fa13407a85c0689ce8de9ade)
I20240502 01:54:25.148105 26152 raft_consensus.cc:504] T 7efd29396c7c4fb597be2251cdcdd48f P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } }
I20240502 01:54:25.148646 26152 leader_election.cc:288] T 7efd29396c7c4fb597be2251cdcdd48f P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:54:25.148768 26168 raft_consensus.cc:482] T 277c6ae5c8dc4f8eae261bf06890d793 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4da690c0fa13407a85c0689ce8de9ade)
I20240502 01:54:25.148878 26168 raft_consensus.cc:504] T 277c6ae5c8dc4f8eae261bf06890d793 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } }
I20240502 01:54:25.149428 26168 leader_election.cc:288] T 277c6ae5c8dc4f8eae261bf06890d793 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), fdc2ac372492454c85387589a91f63af (127.11.42.69:34339)
I20240502 01:54:25.151916 26152 raft_consensus.cc:482] T 75bb4e19cf57405fabcb532b55a29ced P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4da690c0fa13407a85c0689ce8de9ade)
I20240502 01:54:25.152037 26152 raft_consensus.cc:504] T 75bb4e19cf57405fabcb532b55a29ced P bb31bceba76d466db24b9341c04900c3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: VOTER last_known_addr { host: "127.11.42.72" port: 42411 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { promote: false } }
W20240502 01:54:25.152092 20431 leader_election.cc:334] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): Network error: Client connection negotiation failed: client connection to 127.11.42.66:38531: connect: Connection refused (error 111)
I20240502 01:54:25.152192 20431 leader_election.cc:302] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1; no voters: 3fceccefba274b1dbbd8216af8b0bc29, 4da690c0fa13407a85c0689ce8de9ade
I20240502 01:54:25.152498 26152 leader_election.cc:288] T 75bb4e19cf57405fabcb532b55a29ced P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f19cc6cb1e76460aa6711e49a5775091 (127.11.42.72:42411), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
I20240502 01:54:25.152612 26164 raft_consensus.cc:2726] T 29fb2b31cd9c4c8fb5923bb849a097d9 P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:54:25.153004 17800 consensus_peers.cc:485] T de174ba4f7d848718ab8dbeb74369055 P bb31bceba76d466db24b9341c04900c3 -> Peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): Couldn't send request to peer 3fceccefba274b1dbbd8216af8b0bc29. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.66:38531: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:25.153121 17800 consensus_peers.cc:485] T 59c288e7279f4119afb6c3bf860e6dc3 P bb31bceba76d466db24b9341c04900c3 -> Peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): Couldn't send request to peer 3fceccefba274b1dbbd8216af8b0bc29. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.66:38531: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:25.154042 26152 raft_consensus.cc:482] T 495700f8ed5e41228f2e597465dad126 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4da690c0fa13407a85c0689ce8de9ade)
I20240502 01:54:25.154158 26152 raft_consensus.cc:504] T 495700f8ed5e41228f2e597465dad126 P bb31bceba76d466db24b9341c04900c3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } }
W20240502 01:54:25.154332 20433 consensus_peers.cc:485] T b29218a879e74895bfe00021ad358aa4 P 2f306549407b4bb281ccf2994035e4f1 -> Peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Couldn't send request to peer dfa1152987644bd781b46c7e42a7a685. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:54:25.154381 11433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu with pid 17785
I20240502 01:54:25.154668 26152 leader_election.cc:288] T 495700f8ed5e41228f2e597465dad126 P bb31bceba76d466db24b9341c04900c3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:54:25.183882 26066 raft_consensus.cc:482] T b39b88755cd74b0294d264215aea99cf P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4da690c0fa13407a85c0689ce8de9ade)
I20240502 01:54:25.184042 26066 raft_consensus.cc:504] T b39b88755cd74b0294d264215aea99cf P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } }
I20240502 01:54:25.184618 26066 leader_election.cc:288] T b39b88755cd74b0294d264215aea99cf P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
W20240502 01:54:25.186003 20433 leader_election.cc:334] T b39b88755cd74b0294d264215aea99cf P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
W20240502 01:54:25.186153 20433 leader_election.cc:334] T b39b88755cd74b0294d264215aea99cf P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Network error: Client connection negotiation failed: client connection to 127.11.42.65:38321: connect: Connection refused (error 111)
I20240502 01:54:25.186224 20433 leader_election.cc:302] T b39b88755cd74b0294d264215aea99cf P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1; no voters: 4da690c0fa13407a85c0689ce8de9ade, dfa1152987644bd781b46c7e42a7a685
I20240502 01:54:25.186417 26066 raft_consensus.cc:2726] T b39b88755cd74b0294d264215aea99cf P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:54:25.196429 20434 connection.cc:535] client connection to 127.11.42.68:44083 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:54:25.196520 20301 connection.cc:535] server connection from 127.11.42.68:34031 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:54:25.196748 17320 connection.cc:535] server connection from 127.11.42.68:60883 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:54:25.196954 20303 connection.cc:535] client connection to 127.11.42.68:44083 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:54:25.197350 11433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu pid 20282
W20240502 01:54:25.197930 20434 consensus_peers.cc:485] T 1423ea912bfc4472acb2b6798b3a99e9 P 2f306549407b4bb281ccf2994035e4f1 -> Peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): Couldn't send request to peer bb31bceba76d466db24b9341c04900c3. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.68:44083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:25.198110 20303 consensus_peers.cc:485] T 0399c9d1d7b2408292dc78c185605ee8 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): Couldn't send request to peer bb31bceba76d466db24b9341c04900c3. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.68:44083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:25.198201 20303 consensus_peers.cc:485] T 31aa6a26f64f49368ad43cc97c66f2f1 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): Couldn't send request to peer bb31bceba76d466db24b9341c04900c3. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.68:44083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:25.198264 20303 consensus_peers.cc:485] T e42126e16cda4d908b00e2e4a6807d6b P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): Couldn't send request to peer bb31bceba76d466db24b9341c04900c3. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.68:44083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:25.198359 20303 consensus_peers.cc:485] T 52b83e43cae646a29bbf3b826fbbcfb6 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): Couldn't send request to peer bb31bceba76d466db24b9341c04900c3. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.68:44083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:25.198426 20303 consensus_peers.cc:485] T aebde87a7f114537abd9c1d8b7e97004 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): Couldn't send request to peer bb31bceba76d466db24b9341c04900c3. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.68:44083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:25.199482 20434 consensus_peers.cc:485] T 139304da92ab4f48b8a08fa24ed60c95 P 2f306549407b4bb281ccf2994035e4f1 -> Peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): Couldn't send request to peer bb31bceba76d466db24b9341c04900c3. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.68:44083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:25.204474 20407 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 01:54:25.217655 20434 consensus_peers.cc:485] T 2ea539ebb37f42f68c166242a06f8d2e P 2f306549407b4bb281ccf2994035e4f1 -> Peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): Couldn't send request to peer bb31bceba76d466db24b9341c04900c3. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.68:44083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:25.225458 26066 raft_consensus.cc:482] T 5e3695e8ad0a44de9ba23ae3610aacec P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4da690c0fa13407a85c0689ce8de9ade)
I20240502 01:54:25.225601 26066 raft_consensus.cc:504] T 5e3695e8ad0a44de9ba23ae3610aacec P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc2ac372492454c85387589a91f63af" member_type: VOTER last_known_addr { host: "127.11.42.69" port: 34339 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } } peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } }
I20240502 01:54:25.226193 26066 leader_election.cc:288] T 5e3695e8ad0a44de9ba23ae3610aacec P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdc2ac372492454c85387589a91f63af (127.11.42.69:34339), 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321)
W20240502 01:54:25.227636 20433 leader_election.cc:334] T 5e3695e8ad0a44de9ba23ae3610aacec P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Network error: Client connection negotiation failed: client connection to 127.11.42.65:38321: connect: Connection refused (error 111)
W20240502 01:54:25.227766 20434 leader_election.cc:334] T 5e3695e8ad0a44de9ba23ae3610aacec P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111)
I20240502 01:54:25.227885 20434 leader_election.cc:302] T 5e3695e8ad0a44de9ba23ae3610aacec P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f306549407b4bb281ccf2994035e4f1; no voters: 4da690c0fa13407a85c0689ce8de9ade, fdc2ac372492454c85387589a91f63af
I20240502 01:54:25.228098 26066 raft_consensus.cc:2726] T 5e3695e8ad0a44de9ba23ae3610aacec P 2f306549407b4bb281ccf2994035e4f1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:54:25.261174 20431 consensus_peers.cc:485] T 68bc23294ac64c9486bffebbdb16b508 P 2f306549407b4bb281ccf2994035e4f1 -> Peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): Couldn't send request to peer 3fceccefba274b1dbbd8216af8b0bc29. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.66:38531: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:25.403283 20434 consensus_peers.cc:485] T 246cec994d274786aad7672f2a30e5d8 P 2f306549407b4bb281ccf2994035e4f1 -> Peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): Couldn't send request to peer bb31bceba76d466db24b9341c04900c3. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.68:44083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:25.501430 25358 raft_consensus.cc:482] T a390a222562f425eabb11d7f52cbe9a6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4da690c0fa13407a85c0689ce8de9ade)
I20240502 01:54:25.502254 25358 raft_consensus.cc:504] T a390a222562f425eabb11d7f52cbe9a6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } attrs { replace: true } } peers { permanent_uuid: "dfa1152987644bd781b46c7e42a7a685" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 36115 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "f19cc6cb1e76460aa6711e49a5775091" member_type: NON_VOTER last_known_addr { host: "127.11.42.72" port: 42411 } attrs { promote: true } }
I20240502 01:54:25.502874 25358 leader_election.cc:288] T a390a222562f425eabb11d7f52cbe9a6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115)
W20240502 01:54:25.505960 20302 leader_election.cc:334] T a390a222562f425eabb11d7f52cbe9a6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer dfa1152987644bd781b46c7e42a7a685 (127.11.42.67:36115): Network error: Client connection negotiation failed: client connection to 127.11.42.67:36115: connect: Connection refused (error 111)
W20240502 01:54:25.506165 20302 leader_election.cc:334] T a390a222562f425eabb11d7f52cbe9a6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321): Network error: Client connection negotiation failed: client connection to 127.11.42.65:38321: connect: Connection refused (error 111)
I20240502 01:54:25.506268 20302 leader_election.cc:302] T a390a222562f425eabb11d7f52cbe9a6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a36c952c51542c0bf7ea3ec2857a9f9; no voters: 4da690c0fa13407a85c0689ce8de9ade, dfa1152987644bd781b46c7e42a7a685
I20240502 01:54:25.506466 26181 raft_consensus.cc:2726] T a390a222562f425eabb11d7f52cbe9a6 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:54:25.508231 25358 raft_consensus.cc:482] T 495700f8ed5e41228f2e597465dad126 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4da690c0fa13407a85c0689ce8de9ade)
I20240502 01:54:25.508347 25358 raft_consensus.cc:504] T 495700f8ed5e41228f2e597465dad126 P 6a36c952c51542c0bf7ea3ec2857a9f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4da690c0fa13407a85c0689ce8de9ade" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 38321 } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } } peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } attrs { promote: false } }
I20240502 01:54:25.508870 25358 leader_election.cc:288] T 495700f8ed5e41228f2e597465dad126 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4da690c0fa13407a85c0689ce8de9ade (127.11.42.65:38321), bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083)
W20240502 01:54:25.511194 20300 consensus_peers.cc:485] T bbf05e3b3ad44354be7600c3c07756fd P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): Couldn't send request to peer 3fceccefba274b1dbbd8216af8b0bc29. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.66:38531: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:25.511286 20300 consensus_peers.cc:485] T 18549019e31b47d899d011cef0637221 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer 3fceccefba274b1dbbd8216af8b0bc29 (127.11.42.66:38531): Couldn't send request to peer 3fceccefba274b1dbbd8216af8b0bc29. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.66:38531: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:25.512986 20303 consensus_peers.cc:485] T 61ab80ce377a44ac95a9c574e9a615f2 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): Couldn't send request to peer bb31bceba76d466db24b9341c04900c3. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.68:44083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:25.513100 20303 consensus_peers.cc:485] T 28463d45bfde44429d56179c1eec56b0 P 6a36c952c51542c0bf7ea3ec2857a9f9 -> Peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): Couldn't send request to peer bb31bceba76d466db24b9341c04900c3. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.68:44083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:25.513172 20303 leader_election.cc:334] T 495700f8ed5e41228f2e597465dad126 P 6a36c952c51542c0bf7ea3ec2857a9f9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083): Network error: Client connection negotiation failed: client connection to 127.11.42.68:44083: connect: Connection refused (error 111)
I20240502 01:54:25.513769 11433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu with pid 20282
W20240502 01:54:25.550215 20434 consensus_peers.cc:485] T 7b2eae408b9843acaa61e698a3e44cff P 2f306549407b4bb281ccf2994035e4f1 -> Peer fdc2ac372492454c85387589a91f63af (127.11.42.69:34339): Couldn't send request to peer fdc2ac372492454c85387589a91f63af. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.11.42.69:34339: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:54:25.554893 20433 connection.cc:535] client connection to 127.11.42.70:39155 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:54:25.555176 17321 connection.cc:535] server connection from 127.11.42.70:59593 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:54:25.555408 11433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu pid 20415
W20240502 01:54:25.555786 20433 consensus_peers.cc:485] T acbd240a2783424e96bd589f0232daf6 P 2f306549407b4bb281ccf2994035e4f1 -> Peer 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155): Couldn't send request to peer 6a36c952c51542c0bf7ea3ec2857a9f9. Status: Network error: Client connection negotiation failed: client connection to 127.11.42.70:39155: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:25.560945 20538 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:54:25.850502 26190 raft_consensus.cc:482] T 377f8b554ad1448d981d2283b865f8fe P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb31bceba76d466db24b9341c04900c3)
I20240502 01:54:25.850656 26190 raft_consensus.cc:504] T 377f8b554ad1448d981d2283b865f8fe P 2f306549407b4bb281ccf2994035e4f1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bb31bceba76d466db24b9341c04900c3" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 44083 } } peers { permanent_uuid: "2f306549407b4bb281ccf2994035e4f1" member_type: VOTER last_known_addr { host: "127.11.42.71" port: 38623 } attrs { promote: false } } peers { permanent_uuid: "6a36c952c51542c0bf7ea3ec2857a9f9" member_type: VOTER last_known_addr { host: "127.11.42.70" port: 39155 } attrs { promote: false } }
I20240502 01:54:25.851320 26190 leader_election.cc:288] T 377f8b554ad1448d981d2283b865f8fe P 2f306549407b4bb281ccf2994035e4f1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb31bceba76d466db24b9341c04900c3 (127.11.42.68:44083), 6a36c952c51542c0bf7ea3ec2857a9f9 (127.11.42.70:39155)
I20240502 01:54:25.857723 11433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu with pid 20415
I20240502 01:54:25.896242 11433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu pid 17304
I20240502 01:54:25.902227 17365 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:54:26.195883 11433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu with pid 17304
2024-05-02T01:54:26Z chronyd exiting
I20240502 01:54:26.253595 11433 test_util.cc:175] -----------------------------------------------
I20240502 01:54:26.253670 11433 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714614746691041-11433-0
[ FAILED ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/1, where GetParam() = 4-byte object <01-00 00-00> (77522 ms)
[----------] 1 test from DDLDuringRebalancingTest (77522 ms total)
[----------] 1 test from RebalancingDuringElectionStormTest
[ RUN ] RebalancingDuringElectionStormTest.RoundRobin/1
I20240502 01:54:26.255741 11433 test_util.cc:268] Using random seed: -502816939
I20240502 01:54:26.257521 11433 ts_itest-base.cc:112] Starting cluster with:
I20240502 01:54:26.257594 11433 ts_itest-base.cc:113] --------------
I20240502 01:54:26.257630 11433 ts_itest-base.cc:114] 8 tablet servers
I20240502 01:54:26.257683 11433 ts_itest-base.cc:115] 3 replicas per TS
I20240502 01:54:26.257720 11433 ts_itest-base.cc:116] --------------
2024-05-02T01:54:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T01:54:26Z Disabled control of system clock
I20240502 01:54:26.284766 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.42.126:36275
--webserver_interface=127.11.42.126
--webserver_port=0
--builtin_ntp_servers=127.11.42.84:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.42.126:36275
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 01:54:26.601135 26205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:26.601435 26205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:26.601570 26205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:26.608882 26205 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:26.608995 26205 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 01:54:26.609117 26205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:26.609194 26205 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 01:54:26.609254 26205 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 01:54:26.618013 26205 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.11.42.126:36275
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.42.126:36275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.11.42.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:26.619222 26205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:26.620863 26205 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:26.631143 26213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:26.631498 26210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:26.631961 26211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:26.632412 26205 server_base.cc:1034] running on GCE node
I20240502 01:54:26.633112 26205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:26.634374 26205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:26.635574 26205 hybrid_clock.cc:648] HybridClock initialized: now 1714614866635531 us; error 31 us; skew 500 ppm
I20240502 01:54:26.635883 26205 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:26.641563 26205 webserver.cc:460] Webserver started at http://127.11.42.126:39893/ using document root <none> and password file <none>
I20240502 01:54:26.642220 26205 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:26.642336 26205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:26.642637 26205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:54:26.645078 26205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "5cde75c60c9e428aa02b0ed51d873cff"
format_stamp: "Formatted at 2024-05-02 01:54:26 on dist-test-slave-xvk5"
I20240502 01:54:26.645817 26205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "5cde75c60c9e428aa02b0ed51d873cff"
format_stamp: "Formatted at 2024-05-02 01:54:26 on dist-test-slave-xvk5"
I20240502 01:54:26.650554 26205 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 01:54:26.653623 26219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:26.654790 26205 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:54:26.654943 26205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/wal
uuid: "5cde75c60c9e428aa02b0ed51d873cff"
format_stamp: "Formatted at 2024-05-02 01:54:26 on dist-test-slave-xvk5"
I20240502 01:54:26.655118 26205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:26.681073 26205 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:54:26.681849 26205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:26.682118 26205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:26.699352 26205 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.126:36275
I20240502 01:54:26.699375 26271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.126:36275 every 8 connection(s)
I20240502 01:54:26.700675 26205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 01:54:26.704317 26272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:26.709512 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 26205
I20240502 01:54:26.709743 11433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 01:54:26.714284 26272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cde75c60c9e428aa02b0ed51d873cff: Bootstrap starting.
I20240502 01:54:26.717566 26272 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cde75c60c9e428aa02b0ed51d873cff: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:26.718693 26272 log.cc:824] T 00000000000000000000000000000000 P 5cde75c60c9e428aa02b0ed51d873cff: Log is configured to *not* fsync() on all Append() calls
I20240502 01:54:26.721223 26272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cde75c60c9e428aa02b0ed51d873cff: No bootstrap required, opened a new log
I20240502 01:54:26.728039 26272 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cde75c60c9e428aa02b0ed51d873cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cde75c60c9e428aa02b0ed51d873cff" member_type: VOTER last_known_addr { host: "127.11.42.126" port: 36275 } }
I20240502 01:54:26.728340 26272 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cde75c60c9e428aa02b0ed51d873cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:26.728440 26272 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cde75c60c9e428aa02b0ed51d873cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cde75c60c9e428aa02b0ed51d873cff, State: Initialized, Role: FOLLOWER
I20240502 01:54:26.728998 26272 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cde75c60c9e428aa02b0ed51d873cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cde75c60c9e428aa02b0ed51d873cff" member_type: VOTER last_known_addr { host: "127.11.42.126" port: 36275 } }
I20240502 01:54:26.729216 26272 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cde75c60c9e428aa02b0ed51d873cff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:54:26.729353 26272 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cde75c60c9e428aa02b0ed51d873cff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:54:26.729491 26272 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cde75c60c9e428aa02b0ed51d873cff [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:26.731391 26272 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cde75c60c9e428aa02b0ed51d873cff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cde75c60c9e428aa02b0ed51d873cff" member_type: VOTER last_known_addr { host: "127.11.42.126" port: 36275 } }
I20240502 01:54:26.731832 26272 leader_election.cc:302] T 00000000000000000000000000000000 P 5cde75c60c9e428aa02b0ed51d873cff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cde75c60c9e428aa02b0ed51d873cff; no voters:
I20240502 01:54:26.732419 26272 leader_election.cc:288] T 00000000000000000000000000000000 P 5cde75c60c9e428aa02b0ed51d873cff [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:54:26.732558 26277 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cde75c60c9e428aa02b0ed51d873cff [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:26.733196 26277 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cde75c60c9e428aa02b0ed51d873cff [term 1 LEADER]: Becoming Leader. State: Replica: 5cde75c60c9e428aa02b0ed51d873cff, State: Running, Role: LEADER
I20240502 01:54:26.733666 26277 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cde75c60c9e428aa02b0ed51d873cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cde75c60c9e428aa02b0ed51d873cff" member_type: VOTER last_known_addr { host: "127.11.42.126" port: 36275 } }
I20240502 01:54:26.734205 26272 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cde75c60c9e428aa02b0ed51d873cff [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:54:26.736166 26278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cde75c60c9e428aa02b0ed51d873cff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cde75c60c9e428aa02b0ed51d873cff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cde75c60c9e428aa02b0ed51d873cff" member_type: VOTER last_known_addr { host: "127.11.42.126" port: 36275 } } }
I20240502 01:54:26.736516 26278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cde75c60c9e428aa02b0ed51d873cff [sys.catalog]: This master's current role is: LEADER
I20240502 01:54:26.736439 26279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cde75c60c9e428aa02b0ed51d873cff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cde75c60c9e428aa02b0ed51d873cff. Latest consensus state: current_term: 1 leader_uuid: "5cde75c60c9e428aa02b0ed51d873cff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cde75c60c9e428aa02b0ed51d873cff" member_type: VOTER last_known_addr { host: "127.11.42.126" port: 36275 } } }
I20240502 01:54:26.737787 26279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cde75c60c9e428aa02b0ed51d873cff [sys.catalog]: This master's current role is: LEADER
I20240502 01:54:26.738067 26284 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:54:26.743403 26284 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:54:26.751024 26284 catalog_manager.cc:1347] Generated new cluster ID: 99d8c471ad544e15881003a2a50c75d0
I20240502 01:54:26.751121 26284 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:54:26.786490 26284 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:54:26.788172 26284 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:54:26.800179 26284 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 5cde75c60c9e428aa02b0ed51d873cff: Generated new TSK 0
I20240502 01:54:26.801184 26284 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:54:26.810027 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.65:0
--local_ip_for_outbound_sockets=127.11.42.65
--webserver_interface=127.11.42.65
--webserver_port=0
--tserver_master_addrs=127.11.42.126:36275
--builtin_ntp_servers=127.11.42.84:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 01:54:27.136368 26296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:27.136646 26296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:27.136756 26296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:27.144028 26296 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:27.144161 26296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:27.144328 26296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.65
I20240502 01:54:27.154633 26296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.11.42.65
--webserver_port=0
--tserver_master_addrs=127.11.42.126:36275
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.65
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:27.155861 26296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:27.157629 26296 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:27.169085 26301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:27.169567 26304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:27.169965 26302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:27.170493 26296 server_base.cc:1034] running on GCE node
I20240502 01:54:27.171044 26296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:27.171840 26296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:27.173090 26296 hybrid_clock.cc:648] HybridClock initialized: now 1714614867173026 us; error 45 us; skew 500 ppm
I20240502 01:54:27.173372 26296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:27.175412 26296 webserver.cc:460] Webserver started at http://127.11.42.65:35749/ using document root <none> and password file <none>
I20240502 01:54:27.176067 26296 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:27.176162 26296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:27.176438 26296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:54:27.178890 26296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "5e9d825f803c40e089b64e319beccf6e"
format_stamp: "Formatted at 2024-05-02 01:54:27 on dist-test-slave-xvk5"
I20240502 01:54:27.179595 26296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "5e9d825f803c40e089b64e319beccf6e"
format_stamp: "Formatted at 2024-05-02 01:54:27 on dist-test-slave-xvk5"
I20240502 01:54:27.184341 26296 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 01:54:27.187564 26310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:27.188768 26296 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:27.188935 26296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "5e9d825f803c40e089b64e319beccf6e"
format_stamp: "Formatted at 2024-05-02 01:54:27 on dist-test-slave-xvk5"
I20240502 01:54:27.189168 26296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:27.203935 26296 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:54:27.204694 26296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:27.204953 26296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:27.205878 26296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:27.207340 26296 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:27.207428 26296 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:27.207537 26296 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:27.207612 26296 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:27.237645 26296 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.65:34085
I20240502 01:54:27.237721 26424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.65:34085 every 8 connection(s)
I20240502 01:54:27.238929 26296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 01:54:27.242048 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 26296
I20240502 01:54:27.242223 11433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 01:54:27.246081 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.66:0
--local_ip_for_outbound_sockets=127.11.42.66
--webserver_interface=127.11.42.66
--webserver_port=0
--tserver_master_addrs=127.11.42.126:36275
--builtin_ntp_servers=127.11.42.84:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:54:27.252405 26425 heartbeater.cc:347] Connected to a master server at 127.11.42.126:36275
I20240502 01:54:27.252684 26425 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:27.253614 26425 heartbeater.cc:511] Master 127.11.42.126:36275 requested a full tablet report, sending...
I20240502 01:54:27.255910 26236 ts_manager.cc:194] Registered new tserver with Master: 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085)
I20240502 01:54:27.257937 26236 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.65:46659
W20240502 01:54:27.563742 26429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:27.563987 26429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:27.564078 26429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:27.571393 26429 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:27.571499 26429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:27.571619 26429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.66
I20240502 01:54:27.580302 26429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.11.42.66
--webserver_port=0
--tserver_master_addrs=127.11.42.126:36275
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.66
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:27.581511 26429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:27.583132 26429 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:27.594563 26434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:27.595016 26437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:27.595422 26429 server_base.cc:1034] running on GCE node
W20240502 01:54:27.595815 26435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:27.596441 26429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:27.597385 26429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:27.598590 26429 hybrid_clock.cc:648] HybridClock initialized: now 1714614867598545 us; error 37 us; skew 500 ppm
I20240502 01:54:27.598889 26429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:27.601058 26429 webserver.cc:460] Webserver started at http://127.11.42.66:35579/ using document root <none> and password file <none>
I20240502 01:54:27.601754 26429 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:27.601866 26429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:27.602198 26429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:54:27.604686 26429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "4bc1541b47df446297a369de05163012"
format_stamp: "Formatted at 2024-05-02 01:54:27 on dist-test-slave-xvk5"
I20240502 01:54:27.605449 26429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "4bc1541b47df446297a369de05163012"
format_stamp: "Formatted at 2024-05-02 01:54:27 on dist-test-slave-xvk5"
I20240502 01:54:27.610265 26429 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 01:54:27.613413 26443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:27.614576 26429 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20240502 01:54:27.614723 26429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "4bc1541b47df446297a369de05163012"
format_stamp: "Formatted at 2024-05-02 01:54:27 on dist-test-slave-xvk5"
I20240502 01:54:27.614907 26429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:27.633783 26429 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:54:27.634547 26429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:27.634790 26429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:27.635701 26429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:27.637145 26429 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:27.637240 26429 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:27.637348 26429 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:27.637418 26429 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:27.669405 26429 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.66:34003
I20240502 01:54:27.669450 26556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.66:34003 every 8 connection(s)
I20240502 01:54:27.670675 26429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 01:54:27.678165 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 26429
I20240502 01:54:27.678362 11433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 01:54:27.681901 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.67:0
--local_ip_for_outbound_sockets=127.11.42.67
--webserver_interface=127.11.42.67
--webserver_port=0
--tserver_master_addrs=127.11.42.126:36275
--builtin_ntp_servers=127.11.42.84:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:54:27.683215 26557 heartbeater.cc:347] Connected to a master server at 127.11.42.126:36275
I20240502 01:54:27.683528 26557 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:27.684470 26557 heartbeater.cc:511] Master 127.11.42.126:36275 requested a full tablet report, sending...
I20240502 01:54:27.685940 26236 ts_manager.cc:194] Registered new tserver with Master: 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:27.686733 26236 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.66:43531
W20240502 01:54:28.000363 26560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:28.000636 26560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:28.000763 26560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:28.008049 26560 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:28.008163 26560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:28.008317 26560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.67
I20240502 01:54:28.017004 26560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.11.42.67
--webserver_port=0
--tserver_master_addrs=127.11.42.126:36275
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.67
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:28.018257 26560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:28.020073 26560 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:28.031378 26566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:28.031849 26565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:28.032236 26568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:28.032723 26560 server_base.cc:1034] running on GCE node
I20240502 01:54:28.033284 26560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:28.034166 26560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:28.035372 26560 hybrid_clock.cc:648] HybridClock initialized: now 1714614868035319 us; error 43 us; skew 500 ppm
I20240502 01:54:28.035708 26560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:28.037911 26560 webserver.cc:460] Webserver started at http://127.11.42.67:43055/ using document root <none> and password file <none>
I20240502 01:54:28.038635 26560 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:28.038745 26560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:28.039062 26560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:54:28.041620 26560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
format_stamp: "Formatted at 2024-05-02 01:54:28 on dist-test-slave-xvk5"
I20240502 01:54:28.042335 26560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
format_stamp: "Formatted at 2024-05-02 01:54:28 on dist-test-slave-xvk5"
I20240502 01:54:28.047140 26560 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 01:54:28.050262 26574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:28.051445 26560 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.003s
I20240502 01:54:28.051587 26560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
format_stamp: "Formatted at 2024-05-02 01:54:28 on dist-test-slave-xvk5"
I20240502 01:54:28.051764 26560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:28.070545 26560 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:54:28.071256 26560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:28.071498 26560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:28.072366 26560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:28.073801 26560 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:28.073891 26560 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:28.073999 26560 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:28.074074 26560 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:28.104076 26560 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.67:43151
I20240502 01:54:28.104140 26687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.67:43151 every 8 connection(s)
I20240502 01:54:28.105424 26560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 01:54:28.112783 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 26560
I20240502 01:54:28.112968 11433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 01:54:28.116039 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.68:0
--local_ip_for_outbound_sockets=127.11.42.68
--webserver_interface=127.11.42.68
--webserver_port=0
--tserver_master_addrs=127.11.42.126:36275
--builtin_ntp_servers=127.11.42.84:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:54:28.118575 26688 heartbeater.cc:347] Connected to a master server at 127.11.42.126:36275
I20240502 01:54:28.118882 26688 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:28.119834 26688 heartbeater.cc:511] Master 127.11.42.126:36275 requested a full tablet report, sending...
I20240502 01:54:28.121245 26236 ts_manager.cc:194] Registered new tserver with Master: 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:28.122022 26236 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.67:47395
I20240502 01:54:28.260733 26425 heartbeater.cc:503] Master 127.11.42.126:36275 was elected leader, sending a full tablet report...
W20240502 01:54:28.435750 26691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:28.436064 26691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:28.436215 26691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:28.446259 26691 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:28.446416 26691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:28.446605 26691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.68
I20240502 01:54:28.460891 26691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.11.42.68
--webserver_port=0
--tserver_master_addrs=127.11.42.126:36275
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.68
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:28.462674 26691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:28.465097 26691 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:28.476790 26697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:28.477234 26699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:28.477584 26696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:28.478068 26691 server_base.cc:1034] running on GCE node
I20240502 01:54:28.478605 26691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:28.479458 26691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:28.480660 26691 hybrid_clock.cc:648] HybridClock initialized: now 1714614868480611 us; error 46 us; skew 500 ppm
I20240502 01:54:28.480957 26691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:28.483107 26691 webserver.cc:460] Webserver started at http://127.11.42.68:41753/ using document root <none> and password file <none>
I20240502 01:54:28.483773 26691 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:28.483876 26691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:28.484184 26691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:54:28.486660 26691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "401942254c4c43a1ba02c4ae68d17320"
format_stamp: "Formatted at 2024-05-02 01:54:28 on dist-test-slave-xvk5"
I20240502 01:54:28.487412 26691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "401942254c4c43a1ba02c4ae68d17320"
format_stamp: "Formatted at 2024-05-02 01:54:28 on dist-test-slave-xvk5"
I20240502 01:54:28.492131 26691 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20240502 01:54:28.495280 26705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:28.496413 26691 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:28.496556 26691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "401942254c4c43a1ba02c4ae68d17320"
format_stamp: "Formatted at 2024-05-02 01:54:28 on dist-test-slave-xvk5"
I20240502 01:54:28.496726 26691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:28.521160 26691 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:54:28.521881 26691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:28.522133 26691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:28.522975 26691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:28.524339 26691 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:28.524437 26691 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:28.524540 26691 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:28.524608 26691 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:28.553834 26691 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.68:37959
I20240502 01:54:28.553891 26818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.68:37959 every 8 connection(s)
I20240502 01:54:28.554994 26691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 01:54:28.556370 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 26691
I20240502 01:54:28.556582 11433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 01:54:28.559868 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.69:0
--local_ip_for_outbound_sockets=127.11.42.69
--webserver_interface=127.11.42.69
--webserver_port=0
--tserver_master_addrs=127.11.42.126:36275
--builtin_ntp_servers=127.11.42.84:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:54:28.568131 26819 heartbeater.cc:347] Connected to a master server at 127.11.42.126:36275
I20240502 01:54:28.568390 26819 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:28.569169 26819 heartbeater.cc:511] Master 127.11.42.126:36275 requested a full tablet report, sending...
I20240502 01:54:28.570497 26236 ts_manager.cc:194] Registered new tserver with Master: 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:28.571492 26236 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.68:52495
I20240502 01:54:28.689466 26557 heartbeater.cc:503] Master 127.11.42.126:36275 was elected leader, sending a full tablet report...
W20240502 01:54:28.876825 26822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:28.877099 26822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:28.877235 26822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:28.884403 26822 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:28.884507 26822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:28.884647 26822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.69
I20240502 01:54:28.893254 26822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.11.42.69
--webserver_port=0
--tserver_master_addrs=127.11.42.126:36275
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.69
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:28.894483 26822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:28.896112 26822 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:28.907428 26827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:28.907878 26828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:28.908272 26830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:28.908800 26822 server_base.cc:1034] running on GCE node
I20240502 01:54:28.909358 26822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:28.910251 26822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:28.911458 26822 hybrid_clock.cc:648] HybridClock initialized: now 1714614868911402 us; error 44 us; skew 500 ppm
I20240502 01:54:28.911747 26822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:28.913833 26822 webserver.cc:460] Webserver started at http://127.11.42.69:38851/ using document root <none> and password file <none>
I20240502 01:54:28.914511 26822 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:28.914621 26822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:28.914934 26822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:54:28.917434 26822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "807eabb8a90f448390ae330c2c07330c"
format_stamp: "Formatted at 2024-05-02 01:54:28 on dist-test-slave-xvk5"
I20240502 01:54:28.918164 26822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "807eabb8a90f448390ae330c2c07330c"
format_stamp: "Formatted at 2024-05-02 01:54:28 on dist-test-slave-xvk5"
I20240502 01:54:28.923030 26822 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 01:54:28.926172 26836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:28.927376 26822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:54:28.927516 26822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "807eabb8a90f448390ae330c2c07330c"
format_stamp: "Formatted at 2024-05-02 01:54:28 on dist-test-slave-xvk5"
I20240502 01:54:28.927695 26822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:28.948354 26822 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:54:28.949152 26822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:28.949410 26822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:28.950276 26822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:28.951731 26822 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:28.951826 26822 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:28.951929 26822 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:28.951997 26822 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:28.982450 26822 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.69:33779
I20240502 01:54:28.982510 26949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.69:33779 every 8 connection(s)
I20240502 01:54:28.983700 26822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:54:28.991524 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 26822
I20240502 01:54:28.991660 11433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 01:54:28.995173 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.70:0
--local_ip_for_outbound_sockets=127.11.42.70
--webserver_interface=127.11.42.70
--webserver_port=0
--tserver_master_addrs=127.11.42.126:36275
--builtin_ntp_servers=127.11.42.84:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:54:28.995817 26950 heartbeater.cc:347] Connected to a master server at 127.11.42.126:36275
I20240502 01:54:28.996089 26950 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:28.996876 26950 heartbeater.cc:511] Master 127.11.42.126:36275 requested a full tablet report, sending...
I20240502 01:54:28.998385 26236 ts_manager.cc:194] Registered new tserver with Master: 807eabb8a90f448390ae330c2c07330c (127.11.42.69:33779)
I20240502 01:54:28.999356 26236 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.69:46583
I20240502 01:54:29.124493 26688 heartbeater.cc:503] Master 127.11.42.126:36275 was elected leader, sending a full tablet report...
W20240502 01:54:29.311499 26953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:29.311745 26953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:29.311864 26953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:29.318914 26953 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:29.319025 26953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:29.319162 26953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.70
I20240502 01:54:29.327535 26953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.11.42.70
--webserver_port=0
--tserver_master_addrs=127.11.42.126:36275
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.70
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:29.328650 26953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:29.330243 26953 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:29.341326 26961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:29.341697 26958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:29.342093 26959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:29.342466 26953 server_base.cc:1034] running on GCE node
I20240502 01:54:29.342967 26953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:29.343770 26953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:29.344959 26953 hybrid_clock.cc:648] HybridClock initialized: now 1714614869344920 us; error 38 us; skew 500 ppm
I20240502 01:54:29.345256 26953 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:29.347172 26953 webserver.cc:460] Webserver started at http://127.11.42.70:35285/ using document root <none> and password file <none>
I20240502 01:54:29.347819 26953 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:29.347924 26953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:29.348217 26953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:54:29.350551 26953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "f62b051e38d3421087d97b594674043f"
format_stamp: "Formatted at 2024-05-02 01:54:29 on dist-test-slave-xvk5"
I20240502 01:54:29.351246 26953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "f62b051e38d3421087d97b594674043f"
format_stamp: "Formatted at 2024-05-02 01:54:29 on dist-test-slave-xvk5"
I20240502 01:54:29.355778 26953 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20240502 01:54:29.359062 26967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:29.360395 26953 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240502 01:54:29.360538 26953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "f62b051e38d3421087d97b594674043f"
format_stamp: "Formatted at 2024-05-02 01:54:29 on dist-test-slave-xvk5"
I20240502 01:54:29.360714 26953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:29.397073 26953 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:54:29.397840 26953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:29.398085 26953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:29.398957 26953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:29.400350 26953 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:29.400442 26953 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:29.400550 26953 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:29.400627 26953 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:29.433550 26953 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.70:39005
I20240502 01:54:29.433614 27080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.70:39005 every 8 connection(s)
I20240502 01:54:29.434814 26953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 01:54:29.445144 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 26953
I20240502 01:54:29.445348 11433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 01:54:29.446725 27081 heartbeater.cc:347] Connected to a master server at 127.11.42.126:36275
I20240502 01:54:29.446986 27081 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:29.447839 27081 heartbeater.cc:511] Master 127.11.42.126:36275 requested a full tablet report, sending...
I20240502 01:54:29.448235 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.71:0
--local_ip_for_outbound_sockets=127.11.42.71
--webserver_interface=127.11.42.71
--webserver_port=0
--tserver_master_addrs=127.11.42.126:36275
--builtin_ntp_servers=127.11.42.84:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:54:29.449231 26236 ts_manager.cc:194] Registered new tserver with Master: f62b051e38d3421087d97b594674043f (127.11.42.70:39005)
I20240502 01:54:29.450243 26236 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.70:50201
I20240502 01:54:29.573738 26819 heartbeater.cc:503] Master 127.11.42.126:36275 was elected leader, sending a full tablet report...
W20240502 01:54:29.766381 27084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:29.766629 27084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:29.766750 27084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:29.774016 27084 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:29.774122 27084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:29.774276 27084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.71
I20240502 01:54:29.782855 27084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.11.42.71
--webserver_port=0
--tserver_master_addrs=127.11.42.126:36275
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.71
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:29.783987 27084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:29.785655 27084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:29.796700 27089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:29.797164 27090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:29.797533 27092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:29.797987 27084 server_base.cc:1034] running on GCE node
I20240502 01:54:29.798498 27084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:29.799340 27084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:29.800542 27084 hybrid_clock.cc:648] HybridClock initialized: now 1714614869800495 us; error 32 us; skew 500 ppm
I20240502 01:54:29.800808 27084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:29.802827 27084 webserver.cc:460] Webserver started at http://127.11.42.71:40485/ using document root <none> and password file <none>
I20240502 01:54:29.803470 27084 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:29.803561 27084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:29.803824 27084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:54:29.806253 27084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "f2aea971554443a7afbf01e7e79a025d"
format_stamp: "Formatted at 2024-05-02 01:54:29 on dist-test-slave-xvk5"
I20240502 01:54:29.806973 27084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "f2aea971554443a7afbf01e7e79a025d"
format_stamp: "Formatted at 2024-05-02 01:54:29 on dist-test-slave-xvk5"
I20240502 01:54:29.811673 27084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 01:54:29.814764 27098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:29.815929 27084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 01:54:29.816076 27084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "f2aea971554443a7afbf01e7e79a025d"
format_stamp: "Formatted at 2024-05-02 01:54:29 on dist-test-slave-xvk5"
I20240502 01:54:29.816249 27084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:29.829231 27084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:54:29.829921 27084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:29.830152 27084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:29.830981 27084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:29.832340 27084 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:29.832437 27084 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:29.832538 27084 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:29.832629 27084 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:29.861385 27084 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.71:41219
I20240502 01:54:29.861444 27210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.71:41219 every 8 connection(s)
I20240502 01:54:29.862548 27084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 01:54:29.867273 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 27084
I20240502 01:54:29.867424 11433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 01:54:29.870476 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.72:0
--local_ip_for_outbound_sockets=127.11.42.72
--webserver_interface=127.11.42.72
--webserver_port=0
--tserver_master_addrs=127.11.42.126:36275
--builtin_ntp_servers=127.11.42.84:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:54:29.874572 27211 heartbeater.cc:347] Connected to a master server at 127.11.42.126:36275
I20240502 01:54:29.874820 27211 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:29.875545 27211 heartbeater.cc:511] Master 127.11.42.126:36275 requested a full tablet report, sending...
I20240502 01:54:29.876905 26236 ts_manager.cc:194] Registered new tserver with Master: f2aea971554443a7afbf01e7e79a025d (127.11.42.71:41219)
I20240502 01:54:29.877698 26236 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.71:36095
I20240502 01:54:30.001787 26950 heartbeater.cc:503] Master 127.11.42.126:36275 was elected leader, sending a full tablet report...
W20240502 01:54:30.188115 27214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:30.188410 27214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:30.188548 27214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:30.196082 27214 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:30.196189 27214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:30.196348 27214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.72
I20240502 01:54:30.205173 27214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.11.42.72
--webserver_port=0
--tserver_master_addrs=127.11.42.126:36275
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.72
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:30.206384 27214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:30.208118 27214 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:30.219337 27219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:30.219789 27220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:30.220181 27222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:30.220588 27214 server_base.cc:1034] running on GCE node
I20240502 01:54:30.221174 27214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:30.222029 27214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:30.223222 27214 hybrid_clock.cc:648] HybridClock initialized: now 1714614870223180 us; error 37 us; skew 500 ppm
I20240502 01:54:30.223522 27214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:30.225639 27214 webserver.cc:460] Webserver started at http://127.11.42.72:41651/ using document root <none> and password file <none>
I20240502 01:54:30.226286 27214 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:30.226403 27214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:30.226712 27214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:54:30.229152 27214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "0bd788251a5d4622a1117f2000224a2a"
format_stamp: "Formatted at 2024-05-02 01:54:30 on dist-test-slave-xvk5"
I20240502 01:54:30.229879 27214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "0bd788251a5d4622a1117f2000224a2a"
format_stamp: "Formatted at 2024-05-02 01:54:30 on dist-test-slave-xvk5"
I20240502 01:54:30.234745 27214 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240502 01:54:30.237977 27228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:30.239375 27214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240502 01:54:30.239526 27214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "0bd788251a5d4622a1117f2000224a2a"
format_stamp: "Formatted at 2024-05-02 01:54:30 on dist-test-slave-xvk5"
I20240502 01:54:30.239719 27214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:30.264046 27214 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:54:30.264755 27214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:30.265044 27214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:30.265998 27214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:30.267411 27214 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:30.267505 27214 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:30.267607 27214 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:30.267676 27214 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:30.297170 27214 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.72:33965
I20240502 01:54:30.297228 27343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.72:33965 every 8 connection(s)
I20240502 01:54:30.298362 27214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 01:54:30.300154 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 27214
I20240502 01:54:30.300367 11433 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 01:54:30.310395 27344 heartbeater.cc:347] Connected to a master server at 127.11.42.126:36275
I20240502 01:54:30.310621 27344 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:30.311285 27344 heartbeater.cc:511] Master 127.11.42.126:36275 requested a full tablet report, sending...
I20240502 01:54:30.312517 26236 ts_manager.cc:194] Registered new tserver with Master: 0bd788251a5d4622a1117f2000224a2a (127.11.42.72:33965)
I20240502 01:54:30.313268 26236 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.72:34049
I20240502 01:54:30.316195 11433 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 01:54:30.332113 11433 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 01:54:30.332227 11433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu pid 26822
I20240502 01:54:30.339896 26945 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:54:30.420217 11433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu with pid 26822
I20240502 01:54:30.435204 11433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu pid 26953
I20240502 01:54:30.443369 27076 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:54:30.521802 27081 heartbeater.cc:503] Master 127.11.42.126:36275 was elected leader, sending a full tablet report...
I20240502 01:54:30.522902 11433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu with pid 26953
W20240502 01:54:30.538146 26221 connection.cc:535] server connection from 127.11.42.70:50201 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:54:30.538400 11433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu pid 27084
I20240502 01:54:30.546409 27206 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:54:30.625001 11433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu with pid 27084
I20240502 01:54:30.640633 11433 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu pid 27214
I20240502 01:54:30.648995 27339 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:54:30.728477 11433 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu with pid 27214
I20240502 01:54:34.495261 26236 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36868:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:34.533041 26359 tablet_service.cc:1467] Processing CreateTablet for tablet 53b7377c0d144e07a4501c2d626f6494 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6642b46a4ab442f5aae5ef2db67ba1a6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:34.534912 26359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53b7377c0d144e07a4501c2d626f6494. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:34.539394 26358 tablet_service.cc:1467] Processing CreateTablet for tablet 9ead21d62e564a37a8c3d5a452621d13 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6642b46a4ab442f5aae5ef2db67ba1a6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:34.539469 26357 tablet_service.cc:1467] Processing CreateTablet for tablet 4f9ed0162eb74c4ea422f122da01dce3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6642b46a4ab442f5aae5ef2db67ba1a6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:34.539945 26358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ead21d62e564a37a8c3d5a452621d13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:34.540184 26357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f9ed0162eb74c4ea422f122da01dce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:34.539680 26490 tablet_service.cc:1467] Processing CreateTablet for tablet 4f9ed0162eb74c4ea422f122da01dce3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6642b46a4ab442f5aae5ef2db67ba1a6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:34.542464 26753 tablet_service.cc:1467] Processing CreateTablet for tablet 4f9ed0162eb74c4ea422f122da01dce3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6642b46a4ab442f5aae5ef2db67ba1a6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:34.543030 26491 tablet_service.cc:1467] Processing CreateTablet for tablet 9ead21d62e564a37a8c3d5a452621d13 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6642b46a4ab442f5aae5ef2db67ba1a6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:34.543437 26490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f9ed0162eb74c4ea422f122da01dce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:34.543788 26753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f9ed0162eb74c4ea422f122da01dce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:34.545151 26491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ead21d62e564a37a8c3d5a452621d13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:34.544939 26623 tablet_service.cc:1467] Processing CreateTablet for tablet 53b7377c0d144e07a4501c2d626f6494 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6642b46a4ab442f5aae5ef2db67ba1a6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:34.546228 26623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53b7377c0d144e07a4501c2d626f6494. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:34.549193 26754 tablet_service.cc:1467] Processing CreateTablet for tablet 53b7377c0d144e07a4501c2d626f6494 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6642b46a4ab442f5aae5ef2db67ba1a6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:34.549363 26622 tablet_service.cc:1467] Processing CreateTablet for tablet 9ead21d62e564a37a8c3d5a452621d13 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6642b46a4ab442f5aae5ef2db67ba1a6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:34.549680 26754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53b7377c0d144e07a4501c2d626f6494. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:34.549813 26622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ead21d62e564a37a8c3d5a452621d13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:34.572922 27373 tablet_bootstrap.cc:492] T 4f9ed0162eb74c4ea422f122da01dce3 P 5e9d825f803c40e089b64e319beccf6e: Bootstrap starting.
I20240502 01:54:34.573035 27372 tablet_bootstrap.cc:492] T 4f9ed0162eb74c4ea422f122da01dce3 P 401942254c4c43a1ba02c4ae68d17320: Bootstrap starting.
I20240502 01:54:34.574326 27375 tablet_bootstrap.cc:492] T 4f9ed0162eb74c4ea422f122da01dce3 P 4bc1541b47df446297a369de05163012: Bootstrap starting.
I20240502 01:54:34.575897 27372 tablet_bootstrap.cc:654] T 4f9ed0162eb74c4ea422f122da01dce3 P 401942254c4c43a1ba02c4ae68d17320: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:34.576303 27373 tablet_bootstrap.cc:654] T 4f9ed0162eb74c4ea422f122da01dce3 P 5e9d825f803c40e089b64e319beccf6e: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:34.577080 27372 log.cc:824] T 4f9ed0162eb74c4ea422f122da01dce3 P 401942254c4c43a1ba02c4ae68d17320: Log is configured to *not* fsync() on all Append() calls
I20240502 01:54:34.577534 27373 log.cc:824] T 4f9ed0162eb74c4ea422f122da01dce3 P 5e9d825f803c40e089b64e319beccf6e: Log is configured to *not* fsync() on all Append() calls
I20240502 01:54:34.578327 27374 tablet_bootstrap.cc:492] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2: Bootstrap starting.
I20240502 01:54:34.582032 27375 tablet_bootstrap.cc:654] T 4f9ed0162eb74c4ea422f122da01dce3 P 4bc1541b47df446297a369de05163012: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:34.582230 27374 tablet_bootstrap.cc:654] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:34.583333 27375 log.cc:824] T 4f9ed0162eb74c4ea422f122da01dce3 P 4bc1541b47df446297a369de05163012: Log is configured to *not* fsync() on all Append() calls
I20240502 01:54:34.583523 27374 log.cc:824] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2: Log is configured to *not* fsync() on all Append() calls
I20240502 01:54:34.586426 27372 tablet_bootstrap.cc:492] T 4f9ed0162eb74c4ea422f122da01dce3 P 401942254c4c43a1ba02c4ae68d17320: No bootstrap required, opened a new log
I20240502 01:54:34.586706 27372 ts_tablet_manager.cc:1367] T 4f9ed0162eb74c4ea422f122da01dce3 P 401942254c4c43a1ba02c4ae68d17320: Time spent bootstrapping tablet: real 0.014s user 0.013s sys 0.000s
I20240502 01:54:34.588245 27373 tablet_bootstrap.cc:492] T 4f9ed0162eb74c4ea422f122da01dce3 P 5e9d825f803c40e089b64e319beccf6e: No bootstrap required, opened a new log
I20240502 01:54:34.588495 27374 tablet_bootstrap.cc:492] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2: No bootstrap required, opened a new log
I20240502 01:54:34.588553 27373 ts_tablet_manager.cc:1367] T 4f9ed0162eb74c4ea422f122da01dce3 P 5e9d825f803c40e089b64e319beccf6e: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.010s
I20240502 01:54:34.588577 27375 tablet_bootstrap.cc:492] T 4f9ed0162eb74c4ea422f122da01dce3 P 4bc1541b47df446297a369de05163012: No bootstrap required, opened a new log
I20240502 01:54:34.588807 27374 ts_tablet_manager.cc:1367] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20240502 01:54:34.588819 27375 ts_tablet_manager.cc:1367] T 4f9ed0162eb74c4ea422f122da01dce3 P 4bc1541b47df446297a369de05163012: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20240502 01:54:34.595646 27372 raft_consensus.cc:348] T 4f9ed0162eb74c4ea422f122da01dce3 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:34.596113 27372 raft_consensus.cc:374] T 4f9ed0162eb74c4ea422f122da01dce3 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:34.596240 27372 raft_consensus.cc:729] T 4f9ed0162eb74c4ea422f122da01dce3 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Initialized, Role: FOLLOWER
I20240502 01:54:34.596952 27372 consensus_queue.cc:260] T 4f9ed0162eb74c4ea422f122da01dce3 P 401942254c4c43a1ba02c4ae68d17320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:34.597448 27374 raft_consensus.cc:348] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:34.597932 27374 raft_consensus.cc:374] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:34.597828 27375 raft_consensus.cc:348] T 4f9ed0162eb74c4ea422f122da01dce3 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:34.598070 27374 raft_consensus.cc:729] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Initialized, Role: FOLLOWER
I20240502 01:54:34.598341 27375 raft_consensus.cc:374] T 4f9ed0162eb74c4ea422f122da01dce3 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:34.598116 27373 raft_consensus.cc:348] T 4f9ed0162eb74c4ea422f122da01dce3 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:34.598484 27373 raft_consensus.cc:374] T 4f9ed0162eb74c4ea422f122da01dce3 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:34.598507 27375 raft_consensus.cc:729] T 4f9ed0162eb74c4ea422f122da01dce3 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bc1541b47df446297a369de05163012, State: Initialized, Role: FOLLOWER
I20240502 01:54:34.598615 27373 raft_consensus.cc:729] T 4f9ed0162eb74c4ea422f122da01dce3 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Initialized, Role: FOLLOWER
I20240502 01:54:34.598738 27374 consensus_queue.cc:260] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:34.599400 27373 consensus_queue.cc:260] T 4f9ed0162eb74c4ea422f122da01dce3 P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:34.599390 27375 consensus_queue.cc:260] T 4f9ed0162eb74c4ea422f122da01dce3 P 4bc1541b47df446297a369de05163012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:34.604497 27375 ts_tablet_manager.cc:1398] T 4f9ed0162eb74c4ea422f122da01dce3 P 4bc1541b47df446297a369de05163012: Time spent starting tablet: real 0.016s user 0.009s sys 0.006s
I20240502 01:54:34.606570 27373 ts_tablet_manager.cc:1398] T 4f9ed0162eb74c4ea422f122da01dce3 P 5e9d825f803c40e089b64e319beccf6e: Time spent starting tablet: real 0.018s user 0.010s sys 0.003s
I20240502 01:54:34.607055 27373 tablet_bootstrap.cc:492] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e: Bootstrap starting.
I20240502 01:54:34.608772 27373 tablet_bootstrap.cc:654] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:34.611011 27375 tablet_bootstrap.cc:492] T 9ead21d62e564a37a8c3d5a452621d13 P 4bc1541b47df446297a369de05163012: Bootstrap starting.
I20240502 01:54:34.612493 27374 ts_tablet_manager.cc:1398] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent starting tablet: real 0.024s user 0.011s sys 0.010s
I20240502 01:54:34.613052 27375 tablet_bootstrap.cc:654] T 9ead21d62e564a37a8c3d5a452621d13 P 4bc1541b47df446297a369de05163012: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:34.613274 27374 tablet_bootstrap.cc:492] T 53b7377c0d144e07a4501c2d626f6494 P 83c50029f7a94cfca6d1ee68e0728aa2: Bootstrap starting.
I20240502 01:54:34.615032 27374 tablet_bootstrap.cc:654] T 53b7377c0d144e07a4501c2d626f6494 P 83c50029f7a94cfca6d1ee68e0728aa2: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:34.615157 27372 ts_tablet_manager.cc:1398] T 4f9ed0162eb74c4ea422f122da01dce3 P 401942254c4c43a1ba02c4ae68d17320: Time spent starting tablet: real 0.028s user 0.017s sys 0.007s
I20240502 01:54:34.615600 27372 tablet_bootstrap.cc:492] T 53b7377c0d144e07a4501c2d626f6494 P 401942254c4c43a1ba02c4ae68d17320: Bootstrap starting.
I20240502 01:54:34.617431 27372 tablet_bootstrap.cc:654] T 53b7377c0d144e07a4501c2d626f6494 P 401942254c4c43a1ba02c4ae68d17320: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:34.621318 27375 tablet_bootstrap.cc:492] T 9ead21d62e564a37a8c3d5a452621d13 P 4bc1541b47df446297a369de05163012: No bootstrap required, opened a new log
I20240502 01:54:34.621575 27375 ts_tablet_manager.cc:1367] T 9ead21d62e564a37a8c3d5a452621d13 P 4bc1541b47df446297a369de05163012: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.004s
I20240502 01:54:34.622611 27375 raft_consensus.cc:348] T 9ead21d62e564a37a8c3d5a452621d13 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:34.622890 27375 raft_consensus.cc:374] T 9ead21d62e564a37a8c3d5a452621d13 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:34.623028 27375 raft_consensus.cc:729] T 9ead21d62e564a37a8c3d5a452621d13 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bc1541b47df446297a369de05163012, State: Initialized, Role: FOLLOWER
I20240502 01:54:34.623049 27374 tablet_bootstrap.cc:492] T 53b7377c0d144e07a4501c2d626f6494 P 83c50029f7a94cfca6d1ee68e0728aa2: No bootstrap required, opened a new log
I20240502 01:54:34.623180 27374 ts_tablet_manager.cc:1367] T 53b7377c0d144e07a4501c2d626f6494 P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20240502 01:54:34.623245 27373 tablet_bootstrap.cc:492] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e: No bootstrap required, opened a new log
I20240502 01:54:34.623327 27375 consensus_queue.cc:260] T 9ead21d62e564a37a8c3d5a452621d13 P 4bc1541b47df446297a369de05163012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:34.623456 27373 ts_tablet_manager.cc:1367] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.005s
I20240502 01:54:34.624222 27374 raft_consensus.cc:348] T 53b7377c0d144e07a4501c2d626f6494 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:34.624473 27374 raft_consensus.cc:374] T 53b7377c0d144e07a4501c2d626f6494 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:34.624604 27374 raft_consensus.cc:729] T 53b7377c0d144e07a4501c2d626f6494 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Initialized, Role: FOLLOWER
I20240502 01:54:34.624800 27375 ts_tablet_manager.cc:1398] T 9ead21d62e564a37a8c3d5a452621d13 P 4bc1541b47df446297a369de05163012: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:34.624858 27374 consensus_queue.cc:260] T 53b7377c0d144e07a4501c2d626f6494 P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:34.625509 27374 ts_tablet_manager.cc:1398] T 53b7377c0d144e07a4501c2d626f6494 P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:54:34.625536 27372 tablet_bootstrap.cc:492] T 53b7377c0d144e07a4501c2d626f6494 P 401942254c4c43a1ba02c4ae68d17320: No bootstrap required, opened a new log
I20240502 01:54:34.625661 27372 ts_tablet_manager.cc:1367] T 53b7377c0d144e07a4501c2d626f6494 P 401942254c4c43a1ba02c4ae68d17320: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240502 01:54:34.624383 27373 raft_consensus.cc:348] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:34.625958 27373 raft_consensus.cc:374] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:34.626092 27373 raft_consensus.cc:729] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Initialized, Role: FOLLOWER
I20240502 01:54:34.626366 27373 consensus_queue.cc:260] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:34.626649 27372 raft_consensus.cc:348] T 53b7377c0d144e07a4501c2d626f6494 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:34.626964 27373 ts_tablet_manager.cc:1398] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:54:34.627315 27373 tablet_bootstrap.cc:492] T 53b7377c0d144e07a4501c2d626f6494 P 5e9d825f803c40e089b64e319beccf6e: Bootstrap starting.
I20240502 01:54:34.627514 27372 raft_consensus.cc:374] T 53b7377c0d144e07a4501c2d626f6494 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:34.627660 27372 raft_consensus.cc:729] T 53b7377c0d144e07a4501c2d626f6494 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Initialized, Role: FOLLOWER
I20240502 01:54:34.627911 27372 consensus_queue.cc:260] T 53b7377c0d144e07a4501c2d626f6494 P 401942254c4c43a1ba02c4ae68d17320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:34.628437 27372 ts_tablet_manager.cc:1398] T 53b7377c0d144e07a4501c2d626f6494 P 401942254c4c43a1ba02c4ae68d17320: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:34.628979 27373 tablet_bootstrap.cc:654] T 53b7377c0d144e07a4501c2d626f6494 P 5e9d825f803c40e089b64e319beccf6e: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:34.631772 27373 tablet_bootstrap.cc:492] T 53b7377c0d144e07a4501c2d626f6494 P 5e9d825f803c40e089b64e319beccf6e: No bootstrap required, opened a new log
I20240502 01:54:34.631911 27373 ts_tablet_manager.cc:1367] T 53b7377c0d144e07a4501c2d626f6494 P 5e9d825f803c40e089b64e319beccf6e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240502 01:54:34.632393 27373 raft_consensus.cc:348] T 53b7377c0d144e07a4501c2d626f6494 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:34.632562 27373 raft_consensus.cc:374] T 53b7377c0d144e07a4501c2d626f6494 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:34.632629 27373 raft_consensus.cc:729] T 53b7377c0d144e07a4501c2d626f6494 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Initialized, Role: FOLLOWER
I20240502 01:54:34.632833 27373 consensus_queue.cc:260] T 53b7377c0d144e07a4501c2d626f6494 P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:34.633316 27373 ts_tablet_manager.cc:1398] T 53b7377c0d144e07a4501c2d626f6494 P 5e9d825f803c40e089b64e319beccf6e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 01:54:34.646901 27381 raft_consensus.cc:482] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:34.647092 27381 raft_consensus.cc:504] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:34.648670 27381 leader_election.cc:288] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085)
I20240502 01:54:34.655501 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ead21d62e564a37a8c3d5a452621d13" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9d825f803c40e089b64e319beccf6e" is_pre_election: true
I20240502 01:54:34.655514 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ead21d62e564a37a8c3d5a452621d13" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc1541b47df446297a369de05163012" is_pre_election: true
I20240502 01:54:34.655953 26511 raft_consensus.cc:2454] T 9ead21d62e564a37a8c3d5a452621d13 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 0.
I20240502 01:54:34.655972 26379 raft_consensus.cc:2454] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 0.
I20240502 01:54:34.656504 26578 leader_election.cc:302] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:34.656836 27381 raft_consensus.cc:2781] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:34.656977 27381 raft_consensus.cc:482] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:34.657090 27381 raft_consensus.cc:3037] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:34.659121 27381 raft_consensus.cc:504] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:34.659664 27381 leader_election.cc:288] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 1 election: Requested vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085)
I20240502 01:54:34.659993 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ead21d62e564a37a8c3d5a452621d13" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:34.660120 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ead21d62e564a37a8c3d5a452621d13" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:34.660143 26511 raft_consensus.cc:3037] T 9ead21d62e564a37a8c3d5a452621d13 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:34.660319 26379 raft_consensus.cc:3037] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:34.662155 26511 raft_consensus.cc:2454] T 9ead21d62e564a37a8c3d5a452621d13 P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 1.
I20240502 01:54:34.662307 26379 raft_consensus.cc:2454] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 1.
I20240502 01:54:34.662475 26578 leader_election.cc:302] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:34.662726 27381 raft_consensus.cc:2781] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:34.663146 27381 raft_consensus.cc:686] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 LEADER]: Becoming Leader. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:34.663604 27381 consensus_queue.cc:237] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:34.667631 26235 catalog_manager.cc:5484] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67). New cstate: current_term: 1 leader_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: HEALTHY } } }
W20240502 01:54:34.676584 26558 tablet.cc:2369] T 4f9ed0162eb74c4ea422f122da01dce3 P 4bc1541b47df446297a369de05163012: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:54:34.729638 27380 raft_consensus.cc:482] T 4f9ed0162eb74c4ea422f122da01dce3 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:34.729912 27380 raft_consensus.cc:504] T 4f9ed0162eb74c4ea422f122da01dce3 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:34.731894 27380 leader_election.cc:288] T 4f9ed0162eb74c4ea422f122da01dce3 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085)
I20240502 01:54:34.738618 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9ed0162eb74c4ea422f122da01dce3" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9d825f803c40e089b64e319beccf6e" is_pre_election: true
I20240502 01:54:34.738703 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9ed0162eb74c4ea422f122da01dce3" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc1541b47df446297a369de05163012" is_pre_election: true
I20240502 01:54:34.738866 26379 raft_consensus.cc:2454] T 4f9ed0162eb74c4ea422f122da01dce3 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 401942254c4c43a1ba02c4ae68d17320 in term 0.
I20240502 01:54:34.738935 26511 raft_consensus.cc:2454] T 4f9ed0162eb74c4ea422f122da01dce3 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 401942254c4c43a1ba02c4ae68d17320 in term 0.
I20240502 01:54:34.739370 26709 leader_election.cc:302] T 4f9ed0162eb74c4ea422f122da01dce3 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 4bc1541b47df446297a369de05163012; no voters:
I20240502 01:54:34.739754 27380 raft_consensus.cc:2781] T 4f9ed0162eb74c4ea422f122da01dce3 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:34.739907 27380 raft_consensus.cc:482] T 4f9ed0162eb74c4ea422f122da01dce3 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:34.740010 27380 raft_consensus.cc:3037] T 4f9ed0162eb74c4ea422f122da01dce3 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:34.740872 27381 raft_consensus.cc:482] T 53b7377c0d144e07a4501c2d626f6494 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:34.741088 27381 raft_consensus.cc:504] T 53b7377c0d144e07a4501c2d626f6494 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:34.741727 27381 leader_election.cc:288] T 53b7377c0d144e07a4501c2d626f6494 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:34.742166 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b7377c0d144e07a4501c2d626f6494" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9d825f803c40e089b64e319beccf6e" is_pre_election: true
I20240502 01:54:34.742378 26379 raft_consensus.cc:2454] T 53b7377c0d144e07a4501c2d626f6494 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 0.
I20240502 01:54:34.742381 27380 raft_consensus.cc:504] T 4f9ed0162eb74c4ea422f122da01dce3 P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:34.742803 26577 leader_election.cc:302] T 53b7377c0d144e07a4501c2d626f6494 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e9d825f803c40e089b64e319beccf6e, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:34.743059 27380 leader_election.cc:288] T 4f9ed0162eb74c4ea422f122da01dce3 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 1 election: Requested vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085)
I20240502 01:54:34.743367 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9ed0162eb74c4ea422f122da01dce3" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:34.743597 26511 raft_consensus.cc:3037] T 4f9ed0162eb74c4ea422f122da01dce3 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:34.743611 27381 raft_consensus.cc:2781] T 53b7377c0d144e07a4501c2d626f6494 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:34.743647 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9ed0162eb74c4ea422f122da01dce3" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:34.743703 27381 raft_consensus.cc:482] T 53b7377c0d144e07a4501c2d626f6494 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:34.743795 27381 raft_consensus.cc:3037] T 53b7377c0d144e07a4501c2d626f6494 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:34.743821 26379 raft_consensus.cc:3037] T 4f9ed0162eb74c4ea422f122da01dce3 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:54:34.745252 26426 tablet.cc:2369] T 4f9ed0162eb74c4ea422f122da01dce3 P 5e9d825f803c40e089b64e319beccf6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:54:34.746124 26511 raft_consensus.cc:2454] T 4f9ed0162eb74c4ea422f122da01dce3 P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 401942254c4c43a1ba02c4ae68d17320 in term 1.
I20240502 01:54:34.746493 26379 raft_consensus.cc:2454] T 4f9ed0162eb74c4ea422f122da01dce3 P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 401942254c4c43a1ba02c4ae68d17320 in term 1.
I20240502 01:54:34.746557 26709 leader_election.cc:302] T 4f9ed0162eb74c4ea422f122da01dce3 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 4bc1541b47df446297a369de05163012; no voters:
I20240502 01:54:34.746812 27380 raft_consensus.cc:2781] T 4f9ed0162eb74c4ea422f122da01dce3 P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:34.747114 27381 raft_consensus.cc:504] T 53b7377c0d144e07a4501c2d626f6494 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:34.747352 27380 raft_consensus.cc:686] T 4f9ed0162eb74c4ea422f122da01dce3 P 401942254c4c43a1ba02c4ae68d17320 [term 1 LEADER]: Becoming Leader. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Running, Role: LEADER
I20240502 01:54:34.747936 27380 consensus_queue.cc:237] T 4f9ed0162eb74c4ea422f122da01dce3 P 401942254c4c43a1ba02c4ae68d17320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:34.748183 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b7377c0d144e07a4501c2d626f6494" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:34.748361 26379 raft_consensus.cc:3037] T 53b7377c0d144e07a4501c2d626f6494 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:34.750238 26774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b7377c0d144e07a4501c2d626f6494" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "401942254c4c43a1ba02c4ae68d17320" is_pre_election: true
I20240502 01:54:34.750612 26774 raft_consensus.cc:2454] T 53b7377c0d144e07a4501c2d626f6494 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 0.
I20240502 01:54:34.750618 26379 raft_consensus.cc:2454] T 53b7377c0d144e07a4501c2d626f6494 P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 1.
I20240502 01:54:34.747835 27381 leader_election.cc:288] T 53b7377c0d144e07a4501c2d626f6494 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 1 election: Requested vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:34.750990 26577 leader_election.cc:302] T 53b7377c0d144e07a4501c2d626f6494 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e9d825f803c40e089b64e319beccf6e, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:34.750998 26773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b7377c0d144e07a4501c2d626f6494" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:34.751212 27381 raft_consensus.cc:2781] T 53b7377c0d144e07a4501c2d626f6494 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:34.751240 26773 raft_consensus.cc:3037] T 53b7377c0d144e07a4501c2d626f6494 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:34.751408 27381 raft_consensus.cc:686] T 53b7377c0d144e07a4501c2d626f6494 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 LEADER]: Becoming Leader. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:34.751886 27381 consensus_queue.cc:237] T 53b7377c0d144e07a4501c2d626f6494 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:34.753626 26773 raft_consensus.cc:2454] T 53b7377c0d144e07a4501c2d626f6494 P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 1.
I20240502 01:54:34.754667 26235 catalog_manager.cc:5484] T 53b7377c0d144e07a4501c2d626f6494 P 83c50029f7a94cfca6d1ee68e0728aa2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67). New cstate: current_term: 1 leader_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:34.755254 26236 catalog_manager.cc:5484] T 4f9ed0162eb74c4ea422f122da01dce3 P 401942254c4c43a1ba02c4ae68d17320 reported cstate change: term changed from 0 to 1, leader changed from <none> to 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68). New cstate: current_term: 1 leader_uuid: "401942254c4c43a1ba02c4ae68d17320" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } }
W20240502 01:54:34.810701 26820 tablet.cc:2369] T 4f9ed0162eb74c4ea422f122da01dce3 P 401942254c4c43a1ba02c4ae68d17320: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:54:34.861341 26689 tablet.cc:2369] T 53b7377c0d144e07a4501c2d626f6494 P 83c50029f7a94cfca6d1ee68e0728aa2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:54:35.079003 27399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:35.079294 27399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:35.086495 27399 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 01:54:35.152838 27380 consensus_queue.cc:1035] T 4f9ed0162eb74c4ea422f122da01dce3 P 401942254c4c43a1ba02c4ae68d17320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:35.176893 27380 consensus_queue.cc:1035] T 4f9ed0162eb74c4ea422f122da01dce3 P 401942254c4c43a1ba02c4ae68d17320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:35.176927 27391 consensus_queue.cc:1035] T 53b7377c0d144e07a4501c2d626f6494 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:35.178944 27391 consensus_queue.cc:1035] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:35.222226 27381 consensus_queue.cc:1035] T 53b7377c0d144e07a4501c2d626f6494 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:35.243086 27381 consensus_queue.cc:1035] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 274.802 ms
time per row: 0.137401 ms
I20240502 01:54:35.506727 26228 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36868:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:35.515085 26490 tablet_service.cc:1467] Processing CreateTablet for tablet d9dd99b54b7c4d19a80c229bf99d7976 (DEFAULT_TABLE table=rebalance_test_table_1 [id=b6483f8123af416ea04904ea75d90215]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:35.515286 26622 tablet_service.cc:1467] Processing CreateTablet for tablet d9dd99b54b7c4d19a80c229bf99d7976 (DEFAULT_TABLE table=rebalance_test_table_1 [id=b6483f8123af416ea04904ea75d90215]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:35.515561 26490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9dd99b54b7c4d19a80c229bf99d7976. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:35.515700 26622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9dd99b54b7c4d19a80c229bf99d7976. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:35.515647 26753 tablet_service.cc:1467] Processing CreateTablet for tablet d9dd99b54b7c4d19a80c229bf99d7976 (DEFAULT_TABLE table=rebalance_test_table_1 [id=b6483f8123af416ea04904ea75d90215]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:35.516119 26753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9dd99b54b7c4d19a80c229bf99d7976. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:35.517045 26623 tablet_service.cc:1467] Processing CreateTablet for tablet 1c47a74a6dc54d5eb0b47818d816b7fd (DEFAULT_TABLE table=rebalance_test_table_1 [id=b6483f8123af416ea04904ea75d90215]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:35.517433 26623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c47a74a6dc54d5eb0b47818d816b7fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:35.518215 26621 tablet_service.cc:1467] Processing CreateTablet for tablet d2287fdff0db43df90ff5580c0430b8c (DEFAULT_TABLE table=rebalance_test_table_1 [id=b6483f8123af416ea04904ea75d90215]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:35.518702 26621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2287fdff0db43df90ff5580c0430b8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:35.521468 26357 tablet_service.cc:1467] Processing CreateTablet for tablet d2287fdff0db43df90ff5580c0430b8c (DEFAULT_TABLE table=rebalance_test_table_1 [id=b6483f8123af416ea04904ea75d90215]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:35.521901 26357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2287fdff0db43df90ff5580c0430b8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:35.525978 26491 tablet_service.cc:1467] Processing CreateTablet for tablet 1c47a74a6dc54d5eb0b47818d816b7fd (DEFAULT_TABLE table=rebalance_test_table_1 [id=b6483f8123af416ea04904ea75d90215]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:35.525996 26754 tablet_service.cc:1467] Processing CreateTablet for tablet 1c47a74a6dc54d5eb0b47818d816b7fd (DEFAULT_TABLE table=rebalance_test_table_1 [id=b6483f8123af416ea04904ea75d90215]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:35.526250 26489 tablet_service.cc:1467] Processing CreateTablet for tablet d2287fdff0db43df90ff5580c0430b8c (DEFAULT_TABLE table=rebalance_test_table_1 [id=b6483f8123af416ea04904ea75d90215]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:35.526378 26754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c47a74a6dc54d5eb0b47818d816b7fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:35.526373 26491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c47a74a6dc54d5eb0b47818d816b7fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:35.526666 26489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2287fdff0db43df90ff5580c0430b8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:35.532009 27471 tablet_bootstrap.cc:492] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2: Bootstrap starting.
I20240502 01:54:35.533784 27471 tablet_bootstrap.cc:654] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:35.542187 27472 tablet_bootstrap.cc:492] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320: Bootstrap starting.
I20240502 01:54:35.542385 27471 tablet_bootstrap.cc:492] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2: No bootstrap required, opened a new log
I20240502 01:54:35.543167 27471 ts_tablet_manager.cc:1367] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20240502 01:54:35.544131 27472 tablet_bootstrap.cc:654] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:35.544098 27471 raft_consensus.cc:348] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:35.544412 27471 raft_consensus.cc:374] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:35.544570 27471 raft_consensus.cc:729] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Initialized, Role: FOLLOWER
I20240502 01:54:35.544986 27471 consensus_queue.cc:260] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:35.545845 27471 ts_tablet_manager.cc:1398] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:54:35.545902 27470 tablet_bootstrap.cc:492] T d9dd99b54b7c4d19a80c229bf99d7976 P 4bc1541b47df446297a369de05163012: Bootstrap starting.
I20240502 01:54:35.547745 27470 tablet_bootstrap.cc:654] T d9dd99b54b7c4d19a80c229bf99d7976 P 4bc1541b47df446297a369de05163012: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:35.547770 27434 raft_consensus.cc:482] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:35.547933 27434 raft_consensus.cc:504] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:35.548724 27471 tablet_bootstrap.cc:492] T 1c47a74a6dc54d5eb0b47818d816b7fd P 83c50029f7a94cfca6d1ee68e0728aa2: Bootstrap starting.
I20240502 01:54:35.548758 27434 leader_election.cc:288] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:35.549772 26773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9dd99b54b7c4d19a80c229bf99d7976" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "401942254c4c43a1ba02c4ae68d17320" is_pre_election: true
I20240502 01:54:35.549823 26510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9dd99b54b7c4d19a80c229bf99d7976" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc1541b47df446297a369de05163012" is_pre_election: true
I20240502 01:54:35.550285 27471 tablet_bootstrap.cc:654] T 1c47a74a6dc54d5eb0b47818d816b7fd P 83c50029f7a94cfca6d1ee68e0728aa2: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:35.550331 27472 tablet_bootstrap.cc:492] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320: No bootstrap required, opened a new log
I20240502 01:54:35.550498 27472 ts_tablet_manager.cc:1367] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
W20240502 01:54:35.550762 26578 leader_election.cc:341] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bc1541b47df446297a369de05163012 (127.11.42.66:34003): Illegal state: must be running to vote when last-logged opid is not known
W20240502 01:54:35.551055 26578 leader_election.cc:341] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:35.551190 26578 leader_election.cc:302] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83c50029f7a94cfca6d1ee68e0728aa2; no voters: 401942254c4c43a1ba02c4ae68d17320, 4bc1541b47df446297a369de05163012
I20240502 01:54:35.551321 27472 raft_consensus.cc:348] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:35.551501 27472 raft_consensus.cc:374] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:35.551581 27472 raft_consensus.cc:729] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Initialized, Role: FOLLOWER
I20240502 01:54:35.551820 27472 consensus_queue.cc:260] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:35.552466 27472 ts_tablet_manager.cc:1398] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:54:35.552786 27472 tablet_bootstrap.cc:492] T d9dd99b54b7c4d19a80c229bf99d7976 P 401942254c4c43a1ba02c4ae68d17320: Bootstrap starting.
I20240502 01:54:35.554456 27472 tablet_bootstrap.cc:654] T d9dd99b54b7c4d19a80c229bf99d7976 P 401942254c4c43a1ba02c4ae68d17320: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:35.555016 27434 raft_consensus.cc:2726] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:54:35.555205 27474 tablet_bootstrap.cc:492] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e: Bootstrap starting.
I20240502 01:54:35.558612 27474 tablet_bootstrap.cc:654] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:35.561318 27470 tablet_bootstrap.cc:492] T d9dd99b54b7c4d19a80c229bf99d7976 P 4bc1541b47df446297a369de05163012: No bootstrap required, opened a new log
I20240502 01:54:35.561470 27470 ts_tablet_manager.cc:1367] T d9dd99b54b7c4d19a80c229bf99d7976 P 4bc1541b47df446297a369de05163012: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.000s
I20240502 01:54:35.561882 27472 tablet_bootstrap.cc:492] T d9dd99b54b7c4d19a80c229bf99d7976 P 401942254c4c43a1ba02c4ae68d17320: No bootstrap required, opened a new log
I20240502 01:54:35.562034 27472 ts_tablet_manager.cc:1367] T d9dd99b54b7c4d19a80c229bf99d7976 P 401942254c4c43a1ba02c4ae68d17320: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20240502 01:54:35.562731 27472 raft_consensus.cc:348] T d9dd99b54b7c4d19a80c229bf99d7976 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:35.562927 27472 raft_consensus.cc:374] T d9dd99b54b7c4d19a80c229bf99d7976 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:35.563004 27472 raft_consensus.cc:729] T d9dd99b54b7c4d19a80c229bf99d7976 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Initialized, Role: FOLLOWER
I20240502 01:54:35.563226 27472 consensus_queue.cc:260] T d9dd99b54b7c4d19a80c229bf99d7976 P 401942254c4c43a1ba02c4ae68d17320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:35.563355 27471 tablet_bootstrap.cc:492] T 1c47a74a6dc54d5eb0b47818d816b7fd P 83c50029f7a94cfca6d1ee68e0728aa2: No bootstrap required, opened a new log
I20240502 01:54:35.563760 27472 ts_tablet_manager.cc:1398] T d9dd99b54b7c4d19a80c229bf99d7976 P 401942254c4c43a1ba02c4ae68d17320: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:54:35.564083 27470 raft_consensus.cc:348] T d9dd99b54b7c4d19a80c229bf99d7976 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:35.564317 27470 raft_consensus.cc:374] T d9dd99b54b7c4d19a80c229bf99d7976 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:35.564432 27470 raft_consensus.cc:729] T d9dd99b54b7c4d19a80c229bf99d7976 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bc1541b47df446297a369de05163012, State: Initialized, Role: FOLLOWER
I20240502 01:54:35.564742 27470 consensus_queue.cc:260] T d9dd99b54b7c4d19a80c229bf99d7976 P 4bc1541b47df446297a369de05163012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:35.565435 27470 ts_tablet_manager.cc:1398] T d9dd99b54b7c4d19a80c229bf99d7976 P 4bc1541b47df446297a369de05163012: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:54:35.565718 27470 tablet_bootstrap.cc:492] T 1c47a74a6dc54d5eb0b47818d816b7fd P 4bc1541b47df446297a369de05163012: Bootstrap starting.
I20240502 01:54:35.567127 27470 tablet_bootstrap.cc:654] T 1c47a74a6dc54d5eb0b47818d816b7fd P 4bc1541b47df446297a369de05163012: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:35.569470 27474 tablet_bootstrap.cc:492] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e: No bootstrap required, opened a new log
I20240502 01:54:35.569670 27474 ts_tablet_manager.cc:1367] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.000s
I20240502 01:54:35.563508 27471 ts_tablet_manager.cc:1367] T 1c47a74a6dc54d5eb0b47818d816b7fd P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.005s
I20240502 01:54:35.570891 27474 raft_consensus.cc:348] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:35.571161 27474 raft_consensus.cc:374] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:35.571162 27471 raft_consensus.cc:348] T 1c47a74a6dc54d5eb0b47818d816b7fd P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:35.571300 27474 raft_consensus.cc:729] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Initialized, Role: FOLLOWER
I20240502 01:54:35.571396 27471 raft_consensus.cc:374] T 1c47a74a6dc54d5eb0b47818d816b7fd P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:35.571481 27471 raft_consensus.cc:729] T 1c47a74a6dc54d5eb0b47818d816b7fd P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Initialized, Role: FOLLOWER
I20240502 01:54:35.571759 27471 consensus_queue.cc:260] T 1c47a74a6dc54d5eb0b47818d816b7fd P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:35.571754 27474 consensus_queue.cc:260] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:35.572453 27474 ts_tablet_manager.cc:1398] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:35.573179 27471 ts_tablet_manager.cc:1398] T 1c47a74a6dc54d5eb0b47818d816b7fd P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:54:35.573531 27471 tablet_bootstrap.cc:492] T d2287fdff0db43df90ff5580c0430b8c P 83c50029f7a94cfca6d1ee68e0728aa2: Bootstrap starting.
I20240502 01:54:35.574136 27422 raft_consensus.cc:482] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:35.574298 27422 raft_consensus.cc:504] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:35.576318 27422 leader_election.cc:288] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:35.586066 27470 tablet_bootstrap.cc:492] T 1c47a74a6dc54d5eb0b47818d816b7fd P 4bc1541b47df446297a369de05163012: No bootstrap required, opened a new log
I20240502 01:54:35.586287 27470 ts_tablet_manager.cc:1367] T 1c47a74a6dc54d5eb0b47818d816b7fd P 4bc1541b47df446297a369de05163012: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.013s
I20240502 01:54:35.586390 27471 tablet_bootstrap.cc:654] T d2287fdff0db43df90ff5580c0430b8c P 83c50029f7a94cfca6d1ee68e0728aa2: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:35.587137 27470 raft_consensus.cc:348] T 1c47a74a6dc54d5eb0b47818d816b7fd P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:35.587441 27470 raft_consensus.cc:374] T 1c47a74a6dc54d5eb0b47818d816b7fd P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:35.587559 27470 raft_consensus.cc:729] T 1c47a74a6dc54d5eb0b47818d816b7fd P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bc1541b47df446297a369de05163012, State: Initialized, Role: FOLLOWER
I20240502 01:54:35.587903 27470 consensus_queue.cc:260] T 1c47a74a6dc54d5eb0b47818d816b7fd P 4bc1541b47df446297a369de05163012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:35.588609 27470 ts_tablet_manager.cc:1398] T 1c47a74a6dc54d5eb0b47818d816b7fd P 4bc1541b47df446297a369de05163012: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:54:35.588973 27470 tablet_bootstrap.cc:492] T d2287fdff0db43df90ff5580c0430b8c P 4bc1541b47df446297a369de05163012: Bootstrap starting.
I20240502 01:54:35.590595 27470 tablet_bootstrap.cc:654] T d2287fdff0db43df90ff5580c0430b8c P 4bc1541b47df446297a369de05163012: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:35.595165 27470 tablet_bootstrap.cc:492] T d2287fdff0db43df90ff5580c0430b8c P 4bc1541b47df446297a369de05163012: No bootstrap required, opened a new log
I20240502 01:54:35.595342 27470 ts_tablet_manager.cc:1367] T d2287fdff0db43df90ff5580c0430b8c P 4bc1541b47df446297a369de05163012: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240502 01:54:35.595539 27471 tablet_bootstrap.cc:492] T d2287fdff0db43df90ff5580c0430b8c P 83c50029f7a94cfca6d1ee68e0728aa2: No bootstrap required, opened a new log
I20240502 01:54:35.595750 27471 ts_tablet_manager.cc:1367] T d2287fdff0db43df90ff5580c0430b8c P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.000s
I20240502 01:54:35.596029 27470 raft_consensus.cc:348] T d2287fdff0db43df90ff5580c0430b8c P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:35.596240 27470 raft_consensus.cc:374] T d2287fdff0db43df90ff5580c0430b8c P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:35.596354 27470 raft_consensus.cc:729] T d2287fdff0db43df90ff5580c0430b8c P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bc1541b47df446297a369de05163012, State: Initialized, Role: FOLLOWER
I20240502 01:54:35.596596 27470 consensus_queue.cc:260] T d2287fdff0db43df90ff5580c0430b8c P 4bc1541b47df446297a369de05163012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:35.596622 27471 raft_consensus.cc:348] T d2287fdff0db43df90ff5580c0430b8c P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:35.596901 27471 raft_consensus.cc:374] T d2287fdff0db43df90ff5580c0430b8c P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:35.597052 27471 raft_consensus.cc:729] T d2287fdff0db43df90ff5580c0430b8c P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Initialized, Role: FOLLOWER
I20240502 01:54:35.597136 27470 ts_tablet_manager.cc:1398] T d2287fdff0db43df90ff5580c0430b8c P 4bc1541b47df446297a369de05163012: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:54:35.597345 27471 consensus_queue.cc:260] T d2287fdff0db43df90ff5580c0430b8c P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:35.597970 27471 ts_tablet_manager.cc:1398] T d2287fdff0db43df90ff5580c0430b8c P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:54:35.599624 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2287fdff0db43df90ff5580c0430b8c" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" is_pre_election: true
I20240502 01:54:35.600021 26643 raft_consensus.cc:2454] T d2287fdff0db43df90ff5580c0430b8c P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 0.
I20240502 01:54:35.600335 27395 raft_consensus.cc:482] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:35.600494 27395 raft_consensus.cc:504] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:35.601006 26313 leader_election.cc:302] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e9d825f803c40e089b64e319beccf6e, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:35.601446 27395 leader_election.cc:288] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:35.601574 27422 raft_consensus.cc:2781] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:35.601701 27422 raft_consensus.cc:482] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:35.601790 27422 raft_consensus.cc:3037] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:35.602428 26510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c47a74a6dc54d5eb0b47818d816b7fd" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc1541b47df446297a369de05163012" is_pre_election: true
I20240502 01:54:35.602623 26510 raft_consensus.cc:2454] T 1c47a74a6dc54d5eb0b47818d816b7fd P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 401942254c4c43a1ba02c4ae68d17320 in term 0.
I20240502 01:54:35.603003 26709 leader_election.cc:302] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 4bc1541b47df446297a369de05163012; no voters:
I20240502 01:54:35.603274 27395 raft_consensus.cc:2781] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:35.603387 27395 raft_consensus.cc:482] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:35.603466 27395 raft_consensus.cc:3037] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:35.604319 27422 raft_consensus.cc:504] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:35.605243 27422 leader_election.cc:288] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 1 election: Requested vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:35.605916 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2287fdff0db43df90ff5580c0430b8c" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:35.606091 26643 raft_consensus.cc:3037] T d2287fdff0db43df90ff5580c0430b8c P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:35.606233 27395 raft_consensus.cc:504] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:35.606886 27395 leader_election.cc:288] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 1 election: Requested vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:35.607349 26510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c47a74a6dc54d5eb0b47818d816b7fd" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:35.607549 26510 raft_consensus.cc:3037] T 1c47a74a6dc54d5eb0b47818d816b7fd P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:35.608865 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2287fdff0db43df90ff5580c0430b8c" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc1541b47df446297a369de05163012" is_pre_election: true
I20240502 01:54:35.609061 26511 raft_consensus.cc:2454] T d2287fdff0db43df90ff5580c0430b8c P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 0.
I20240502 01:54:35.608987 26509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2287fdff0db43df90ff5580c0430b8c" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:35.609282 26509 raft_consensus.cc:3037] T d2287fdff0db43df90ff5580c0430b8c P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:35.609347 26643 raft_consensus.cc:2454] T d2287fdff0db43df90ff5580c0430b8c P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 1.
I20240502 01:54:35.609738 26313 leader_election.cc:302] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e9d825f803c40e089b64e319beccf6e, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:35.610014 27422 raft_consensus.cc:2781] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:35.610481 27422 raft_consensus.cc:686] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [term 1 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:35.611016 27422 consensus_queue.cc:237] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:35.611591 26510 raft_consensus.cc:2454] T 1c47a74a6dc54d5eb0b47818d816b7fd P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 401942254c4c43a1ba02c4ae68d17320 in term 1.
I20240502 01:54:35.611668 26509 raft_consensus.cc:2454] T d2287fdff0db43df90ff5580c0430b8c P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 1.
I20240502 01:54:35.612037 26709 leader_election.cc:302] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 4bc1541b47df446297a369de05163012; no voters:
I20240502 01:54:35.612368 27395 raft_consensus.cc:2781] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:35.612516 27395 raft_consensus.cc:686] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320 [term 1 LEADER]: Becoming Leader. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Running, Role: LEADER
I20240502 01:54:35.612802 27395 consensus_queue.cc:237] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:35.615329 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c47a74a6dc54d5eb0b47818d816b7fd" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" is_pre_election: true
I20240502 01:54:35.615542 26643 raft_consensus.cc:2454] T 1c47a74a6dc54d5eb0b47818d816b7fd P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 401942254c4c43a1ba02c4ae68d17320 in term 0.
I20240502 01:54:35.615871 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c47a74a6dc54d5eb0b47818d816b7fd" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:35.616130 26642 raft_consensus.cc:3037] T 1c47a74a6dc54d5eb0b47818d816b7fd P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:35.617086 26232 catalog_manager.cc:5484] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320 reported cstate change: term changed from 0 to 1, leader changed from <none> to 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68). New cstate: current_term: 1 leader_uuid: "401942254c4c43a1ba02c4ae68d17320" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:35.617715 26228 catalog_manager.cc:5484] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 1 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:35.618759 26642 raft_consensus.cc:2454] T 1c47a74a6dc54d5eb0b47818d816b7fd P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 401942254c4c43a1ba02c4ae68d17320 in term 1.
I20240502 01:54:35.918130 27395 raft_consensus.cc:482] T d9dd99b54b7c4d19a80c229bf99d7976 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:35.918315 27395 raft_consensus.cc:504] T d9dd99b54b7c4d19a80c229bf99d7976 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:35.918918 27395 leader_election.cc:288] T d9dd99b54b7c4d19a80c229bf99d7976 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:35.919458 26509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9dd99b54b7c4d19a80c229bf99d7976" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc1541b47df446297a369de05163012" is_pre_election: true
I20240502 01:54:35.919533 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9dd99b54b7c4d19a80c229bf99d7976" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" is_pre_election: true
I20240502 01:54:35.919732 26509 raft_consensus.cc:2454] T d9dd99b54b7c4d19a80c229bf99d7976 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 401942254c4c43a1ba02c4ae68d17320 in term 0.
I20240502 01:54:35.919764 26642 raft_consensus.cc:2454] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 401942254c4c43a1ba02c4ae68d17320 in term 0.
I20240502 01:54:35.920148 26708 leader_election.cc:302] T d9dd99b54b7c4d19a80c229bf99d7976 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:35.920439 27395 raft_consensus.cc:2781] T d9dd99b54b7c4d19a80c229bf99d7976 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:35.920567 27395 raft_consensus.cc:482] T d9dd99b54b7c4d19a80c229bf99d7976 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:35.920666 27395 raft_consensus.cc:3037] T d9dd99b54b7c4d19a80c229bf99d7976 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:35.922508 27395 raft_consensus.cc:504] T d9dd99b54b7c4d19a80c229bf99d7976 P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:35.923014 27395 leader_election.cc:288] T d9dd99b54b7c4d19a80c229bf99d7976 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 1 election: Requested vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:35.923419 26509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9dd99b54b7c4d19a80c229bf99d7976" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:35.923447 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9dd99b54b7c4d19a80c229bf99d7976" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:35.923581 26509 raft_consensus.cc:3037] T d9dd99b54b7c4d19a80c229bf99d7976 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:35.923630 26642 raft_consensus.cc:3037] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:35.925370 26509 raft_consensus.cc:2454] T d9dd99b54b7c4d19a80c229bf99d7976 P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 401942254c4c43a1ba02c4ae68d17320 in term 1.
I20240502 01:54:35.925408 26642 raft_consensus.cc:2454] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 401942254c4c43a1ba02c4ae68d17320 in term 1.
I20240502 01:54:35.925773 26709 leader_election.cc:302] T d9dd99b54b7c4d19a80c229bf99d7976 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 4bc1541b47df446297a369de05163012, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:35.926121 27395 raft_consensus.cc:2781] T d9dd99b54b7c4d19a80c229bf99d7976 P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:35.926260 27395 raft_consensus.cc:686] T d9dd99b54b7c4d19a80c229bf99d7976 P 401942254c4c43a1ba02c4ae68d17320 [term 1 LEADER]: Becoming Leader. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Running, Role: LEADER
I20240502 01:54:35.926492 27395 consensus_queue.cc:237] T d9dd99b54b7c4d19a80c229bf99d7976 P 401942254c4c43a1ba02c4ae68d17320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:35.928784 26228 catalog_manager.cc:5484] T d9dd99b54b7c4d19a80c229bf99d7976 P 401942254c4c43a1ba02c4ae68d17320 reported cstate change: term changed from 0 to 1, leader changed from <none> to 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68). New cstate: current_term: 1 leader_uuid: "401942254c4c43a1ba02c4ae68d17320" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:35.996672 27422 consensus_queue.cc:1035] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:36.003180 27487 consensus_queue.cc:1035] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:36.062672 27395 consensus_queue.cc:1035] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:36.068092 27395 consensus_queue.cc:1035] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:54:36.322198 27492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:36.322494 27492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:36.329526 27492 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 01:54:36.363904 27395 consensus_queue.cc:1035] T d9dd99b54b7c4d19a80c229bf99d7976 P 401942254c4c43a1ba02c4ae68d17320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:36.368901 27455 consensus_queue.cc:1035] T d9dd99b54b7c4d19a80c229bf99d7976 P 401942254c4c43a1ba02c4ae68d17320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 183.79 ms
time per row: 0.091895 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 01:54:37.505127 26230 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36868:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:37.514019 26490 tablet_service.cc:1467] Processing CreateTablet for tablet 37e818dc0b834eb185e27fcb3cb972e1 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ee49b69299dd42b7a77396e7444ab46c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:37.514505 26490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37e818dc0b834eb185e27fcb3cb972e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:37.514655 26357 tablet_service.cc:1467] Processing CreateTablet for tablet 37e818dc0b834eb185e27fcb3cb972e1 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ee49b69299dd42b7a77396e7444ab46c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:37.515058 26357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37e818dc0b834eb185e27fcb3cb972e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:37.515523 26754 tablet_service.cc:1467] Processing CreateTablet for tablet 37e818dc0b834eb185e27fcb3cb972e1 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ee49b69299dd42b7a77396e7444ab46c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:37.515935 26754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37e818dc0b834eb185e27fcb3cb972e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:37.519327 26621 tablet_service.cc:1467] Processing CreateTablet for tablet 3d751cdd58eb4cef8039fd4c50855e99 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ee49b69299dd42b7a77396e7444ab46c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:37.519769 26621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d751cdd58eb4cef8039fd4c50855e99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:37.520017 26752 tablet_service.cc:1467] Processing CreateTablet for tablet 245ca06c39774fb5b0f522550a63ac5b (DEFAULT_TABLE table=rebalance_test_table_2 [id=ee49b69299dd42b7a77396e7444ab46c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:37.520370 26752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 245ca06c39774fb5b0f522550a63ac5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:37.520649 26489 tablet_service.cc:1467] Processing CreateTablet for tablet 245ca06c39774fb5b0f522550a63ac5b (DEFAULT_TABLE table=rebalance_test_table_2 [id=ee49b69299dd42b7a77396e7444ab46c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:37.520792 26359 tablet_service.cc:1467] Processing CreateTablet for tablet 3d751cdd58eb4cef8039fd4c50855e99 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ee49b69299dd42b7a77396e7444ab46c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:37.521216 26359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d751cdd58eb4cef8039fd4c50855e99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:37.521214 26489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 245ca06c39774fb5b0f522550a63ac5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:37.520040 26753 tablet_service.cc:1467] Processing CreateTablet for tablet 3d751cdd58eb4cef8039fd4c50855e99 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ee49b69299dd42b7a77396e7444ab46c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:37.525449 26753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d751cdd58eb4cef8039fd4c50855e99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:37.525362 26358 tablet_service.cc:1467] Processing CreateTablet for tablet 245ca06c39774fb5b0f522550a63ac5b (DEFAULT_TABLE table=rebalance_test_table_2 [id=ee49b69299dd42b7a77396e7444ab46c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:37.525717 26358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 245ca06c39774fb5b0f522550a63ac5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:37.540055 27560 tablet_bootstrap.cc:492] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2: Bootstrap starting.
I20240502 01:54:37.540693 27559 tablet_bootstrap.cc:492] T 37e818dc0b834eb185e27fcb3cb972e1 P 5e9d825f803c40e089b64e319beccf6e: Bootstrap starting.
I20240502 01:54:37.539791 27561 tablet_bootstrap.cc:492] T 3d751cdd58eb4cef8039fd4c50855e99 P 401942254c4c43a1ba02c4ae68d17320: Bootstrap starting.
I20240502 01:54:37.541461 27562 tablet_bootstrap.cc:492] T 37e818dc0b834eb185e27fcb3cb972e1 P 4bc1541b47df446297a369de05163012: Bootstrap starting.
I20240502 01:54:37.541719 27560 tablet_bootstrap.cc:654] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:37.542320 27559 tablet_bootstrap.cc:654] T 37e818dc0b834eb185e27fcb3cb972e1 P 5e9d825f803c40e089b64e319beccf6e: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:37.542768 27561 tablet_bootstrap.cc:654] T 3d751cdd58eb4cef8039fd4c50855e99 P 401942254c4c43a1ba02c4ae68d17320: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:37.543038 27562 tablet_bootstrap.cc:654] T 37e818dc0b834eb185e27fcb3cb972e1 P 4bc1541b47df446297a369de05163012: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:37.547567 27559 tablet_bootstrap.cc:492] T 37e818dc0b834eb185e27fcb3cb972e1 P 5e9d825f803c40e089b64e319beccf6e: No bootstrap required, opened a new log
I20240502 01:54:37.547781 27559 ts_tablet_manager.cc:1367] T 37e818dc0b834eb185e27fcb3cb972e1 P 5e9d825f803c40e089b64e319beccf6e: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240502 01:54:37.548629 27559 raft_consensus.cc:348] T 37e818dc0b834eb185e27fcb3cb972e1 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:37.548980 27559 raft_consensus.cc:374] T 37e818dc0b834eb185e27fcb3cb972e1 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:37.549142 27559 raft_consensus.cc:729] T 37e818dc0b834eb185e27fcb3cb972e1 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Initialized, Role: FOLLOWER
I20240502 01:54:37.549455 27559 consensus_queue.cc:260] T 37e818dc0b834eb185e27fcb3cb972e1 P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:37.550216 27559 ts_tablet_manager.cc:1398] T 37e818dc0b834eb185e27fcb3cb972e1 P 5e9d825f803c40e089b64e319beccf6e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:37.550537 27559 tablet_bootstrap.cc:492] T 3d751cdd58eb4cef8039fd4c50855e99 P 5e9d825f803c40e089b64e319beccf6e: Bootstrap starting.
I20240502 01:54:37.550922 27562 tablet_bootstrap.cc:492] T 37e818dc0b834eb185e27fcb3cb972e1 P 4bc1541b47df446297a369de05163012: No bootstrap required, opened a new log
I20240502 01:54:37.551115 27562 ts_tablet_manager.cc:1367] T 37e818dc0b834eb185e27fcb3cb972e1 P 4bc1541b47df446297a369de05163012: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20240502 01:54:37.552006 27559 tablet_bootstrap.cc:654] T 3d751cdd58eb4cef8039fd4c50855e99 P 5e9d825f803c40e089b64e319beccf6e: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:37.551895 27562 raft_consensus.cc:348] T 37e818dc0b834eb185e27fcb3cb972e1 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:37.552089 27562 raft_consensus.cc:374] T 37e818dc0b834eb185e27fcb3cb972e1 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:37.552211 27562 raft_consensus.cc:729] T 37e818dc0b834eb185e27fcb3cb972e1 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bc1541b47df446297a369de05163012, State: Initialized, Role: FOLLOWER
I20240502 01:54:37.552525 27562 consensus_queue.cc:260] T 37e818dc0b834eb185e27fcb3cb972e1 P 4bc1541b47df446297a369de05163012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:37.553367 27561 tablet_bootstrap.cc:492] T 3d751cdd58eb4cef8039fd4c50855e99 P 401942254c4c43a1ba02c4ae68d17320: No bootstrap required, opened a new log
I20240502 01:54:37.553510 27561 ts_tablet_manager.cc:1367] T 3d751cdd58eb4cef8039fd4c50855e99 P 401942254c4c43a1ba02c4ae68d17320: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.007s
I20240502 01:54:37.554299 27561 raft_consensus.cc:348] T 3d751cdd58eb4cef8039fd4c50855e99 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:37.554562 27561 raft_consensus.cc:374] T 3d751cdd58eb4cef8039fd4c50855e99 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:37.554726 27561 raft_consensus.cc:729] T 3d751cdd58eb4cef8039fd4c50855e99 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Initialized, Role: FOLLOWER
I20240502 01:54:37.555040 27561 consensus_queue.cc:260] T 3d751cdd58eb4cef8039fd4c50855e99 P 401942254c4c43a1ba02c4ae68d17320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:37.555658 27561 ts_tablet_manager.cc:1398] T 3d751cdd58eb4cef8039fd4c50855e99 P 401942254c4c43a1ba02c4ae68d17320: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:54:37.556003 27561 tablet_bootstrap.cc:492] T 37e818dc0b834eb185e27fcb3cb972e1 P 401942254c4c43a1ba02c4ae68d17320: Bootstrap starting.
I20240502 01:54:37.557313 27560 tablet_bootstrap.cc:492] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2: No bootstrap required, opened a new log
I20240502 01:54:37.557511 27560 ts_tablet_manager.cc:1367] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.010s
I20240502 01:54:37.557631 27561 tablet_bootstrap.cc:654] T 37e818dc0b834eb185e27fcb3cb972e1 P 401942254c4c43a1ba02c4ae68d17320: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:37.558286 27560 raft_consensus.cc:348] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:37.558557 27560 raft_consensus.cc:374] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:37.558744 27560 raft_consensus.cc:729] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Initialized, Role: FOLLOWER
I20240502 01:54:37.559021 27560 consensus_queue.cc:260] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:37.560273 27560 ts_tablet_manager.cc:1398] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:37.560997 27562 ts_tablet_manager.cc:1398] T 37e818dc0b834eb185e27fcb3cb972e1 P 4bc1541b47df446297a369de05163012: Time spent starting tablet: real 0.010s user 0.005s sys 0.004s
I20240502 01:54:37.562220 27559 tablet_bootstrap.cc:492] T 3d751cdd58eb4cef8039fd4c50855e99 P 5e9d825f803c40e089b64e319beccf6e: No bootstrap required, opened a new log
I20240502 01:54:37.562391 27559 ts_tablet_manager.cc:1367] T 3d751cdd58eb4cef8039fd4c50855e99 P 5e9d825f803c40e089b64e319beccf6e: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.001s
I20240502 01:54:37.561352 27562 tablet_bootstrap.cc:492] T 245ca06c39774fb5b0f522550a63ac5b P 4bc1541b47df446297a369de05163012: Bootstrap starting.
I20240502 01:54:37.563083 27559 raft_consensus.cc:348] T 3d751cdd58eb4cef8039fd4c50855e99 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:37.563539 27559 raft_consensus.cc:374] T 3d751cdd58eb4cef8039fd4c50855e99 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:37.563623 27559 raft_consensus.cc:729] T 3d751cdd58eb4cef8039fd4c50855e99 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Initialized, Role: FOLLOWER
I20240502 01:54:37.563854 27559 consensus_queue.cc:260] T 3d751cdd58eb4cef8039fd4c50855e99 P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:37.564381 27559 ts_tablet_manager.cc:1398] T 3d751cdd58eb4cef8039fd4c50855e99 P 5e9d825f803c40e089b64e319beccf6e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:54:37.565299 27559 tablet_bootstrap.cc:492] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e: Bootstrap starting.
I20240502 01:54:37.566619 27562 tablet_bootstrap.cc:654] T 245ca06c39774fb5b0f522550a63ac5b P 4bc1541b47df446297a369de05163012: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:37.566783 27559 tablet_bootstrap.cc:654] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:37.573582 27561 tablet_bootstrap.cc:492] T 37e818dc0b834eb185e27fcb3cb972e1 P 401942254c4c43a1ba02c4ae68d17320: No bootstrap required, opened a new log
I20240502 01:54:37.573748 27561 ts_tablet_manager.cc:1367] T 37e818dc0b834eb185e27fcb3cb972e1 P 401942254c4c43a1ba02c4ae68d17320: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.000s
I20240502 01:54:37.574028 27559 tablet_bootstrap.cc:492] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e: No bootstrap required, opened a new log
I20240502 01:54:37.574210 27559 ts_tablet_manager.cc:1367] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20240502 01:54:37.574234 27562 tablet_bootstrap.cc:492] T 245ca06c39774fb5b0f522550a63ac5b P 4bc1541b47df446297a369de05163012: No bootstrap required, opened a new log
I20240502 01:54:37.574374 27562 ts_tablet_manager.cc:1367] T 245ca06c39774fb5b0f522550a63ac5b P 4bc1541b47df446297a369de05163012: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20240502 01:54:37.574529 27561 raft_consensus.cc:348] T 37e818dc0b834eb185e27fcb3cb972e1 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:37.574787 27561 raft_consensus.cc:374] T 37e818dc0b834eb185e27fcb3cb972e1 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:37.574898 27561 raft_consensus.cc:729] T 37e818dc0b834eb185e27fcb3cb972e1 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Initialized, Role: FOLLOWER
I20240502 01:54:37.574946 27559 raft_consensus.cc:348] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:37.575179 27559 raft_consensus.cc:374] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:37.575085 27562 raft_consensus.cc:348] T 245ca06c39774fb5b0f522550a63ac5b P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:37.575289 27562 raft_consensus.cc:374] T 245ca06c39774fb5b0f522550a63ac5b P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:37.575304 27559 raft_consensus.cc:729] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Initialized, Role: FOLLOWER
I20240502 01:54:37.575212 27561 consensus_queue.cc:260] T 37e818dc0b834eb185e27fcb3cb972e1 P 401942254c4c43a1ba02c4ae68d17320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:37.575405 27562 raft_consensus.cc:729] T 245ca06c39774fb5b0f522550a63ac5b P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bc1541b47df446297a369de05163012, State: Initialized, Role: FOLLOWER
I20240502 01:54:37.575593 27559 consensus_queue.cc:260] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:37.575683 27562 consensus_queue.cc:260] T 245ca06c39774fb5b0f522550a63ac5b P 4bc1541b47df446297a369de05163012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:37.575911 27561 ts_tablet_manager.cc:1398] T 37e818dc0b834eb185e27fcb3cb972e1 P 401942254c4c43a1ba02c4ae68d17320: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:54:37.576319 27561 tablet_bootstrap.cc:492] T 245ca06c39774fb5b0f522550a63ac5b P 401942254c4c43a1ba02c4ae68d17320: Bootstrap starting.
I20240502 01:54:37.576370 27559 ts_tablet_manager.cc:1398] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:54:37.577778 27561 tablet_bootstrap.cc:654] T 245ca06c39774fb5b0f522550a63ac5b P 401942254c4c43a1ba02c4ae68d17320: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:37.581611 27562 ts_tablet_manager.cc:1398] T 245ca06c39774fb5b0f522550a63ac5b P 4bc1541b47df446297a369de05163012: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240502 01:54:37.583603 27561 tablet_bootstrap.cc:492] T 245ca06c39774fb5b0f522550a63ac5b P 401942254c4c43a1ba02c4ae68d17320: No bootstrap required, opened a new log
I20240502 01:54:37.583768 27561 ts_tablet_manager.cc:1367] T 245ca06c39774fb5b0f522550a63ac5b P 401942254c4c43a1ba02c4ae68d17320: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20240502 01:54:37.584475 27561 raft_consensus.cc:348] T 245ca06c39774fb5b0f522550a63ac5b P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:37.584650 27561 raft_consensus.cc:374] T 245ca06c39774fb5b0f522550a63ac5b P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:37.584748 27561 raft_consensus.cc:729] T 245ca06c39774fb5b0f522550a63ac5b P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Initialized, Role: FOLLOWER
I20240502 01:54:37.584944 27561 consensus_queue.cc:260] T 245ca06c39774fb5b0f522550a63ac5b P 401942254c4c43a1ba02c4ae68d17320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:37.585429 27561 ts_tablet_manager.cc:1398] T 245ca06c39774fb5b0f522550a63ac5b P 401942254c4c43a1ba02c4ae68d17320: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:54:37.693142 27422 raft_consensus.cc:482] T 37e818dc0b834eb185e27fcb3cb972e1 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:37.693324 27422 raft_consensus.cc:504] T 37e818dc0b834eb185e27fcb3cb972e1 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:37.693984 27422 leader_election.cc:288] T 37e818dc0b834eb185e27fcb3cb972e1 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:37.694504 26510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e818dc0b834eb185e27fcb3cb972e1" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc1541b47df446297a369de05163012" is_pre_election: true
I20240502 01:54:37.694754 26510 raft_consensus.cc:2454] T 37e818dc0b834eb185e27fcb3cb972e1 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 0.
I20240502 01:54:37.695247 26314 leader_election.cc:302] T 37e818dc0b834eb185e27fcb3cb972e1 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:37.695667 27422 raft_consensus.cc:2781] T 37e818dc0b834eb185e27fcb3cb972e1 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:37.695822 27422 raft_consensus.cc:482] T 37e818dc0b834eb185e27fcb3cb972e1 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:37.695941 27422 raft_consensus.cc:3037] T 37e818dc0b834eb185e27fcb3cb972e1 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:37.698204 27422 raft_consensus.cc:504] T 37e818dc0b834eb185e27fcb3cb972e1 P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:37.698925 27422 leader_election.cc:288] T 37e818dc0b834eb185e27fcb3cb972e1 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 1 election: Requested vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:37.699368 26510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e818dc0b834eb185e27fcb3cb972e1" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:37.699571 26510 raft_consensus.cc:3037] T 37e818dc0b834eb185e27fcb3cb972e1 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:37.701005 26773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e818dc0b834eb185e27fcb3cb972e1" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "401942254c4c43a1ba02c4ae68d17320" is_pre_election: true
I20240502 01:54:37.701117 26774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e818dc0b834eb185e27fcb3cb972e1" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:37.701265 26773 raft_consensus.cc:2454] T 37e818dc0b834eb185e27fcb3cb972e1 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 0.
I20240502 01:54:37.701896 26510 raft_consensus.cc:2454] T 37e818dc0b834eb185e27fcb3cb972e1 P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 1.
I20240502 01:54:37.702214 26314 leader_election.cc:302] T 37e818dc0b834eb185e27fcb3cb972e1 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4bc1541b47df446297a369de05163012, 5e9d825f803c40e089b64e319beccf6e; no voters: 401942254c4c43a1ba02c4ae68d17320
I20240502 01:54:37.702495 27422 raft_consensus.cc:2781] T 37e818dc0b834eb185e27fcb3cb972e1 P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:37.702862 27422 raft_consensus.cc:686] T 37e818dc0b834eb185e27fcb3cb972e1 P 5e9d825f803c40e089b64e319beccf6e [term 1 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:37.703166 27422 consensus_queue.cc:237] T 37e818dc0b834eb185e27fcb3cb972e1 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:37.705946 26235 catalog_manager.cc:5484] T 37e818dc0b834eb185e27fcb3cb972e1 P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 1 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:37.807300 27434 raft_consensus.cc:482] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:37.807493 27434 raft_consensus.cc:504] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:37.808136 27434 leader_election.cc:288] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:37.808622 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d751cdd58eb4cef8039fd4c50855e99" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9d825f803c40e089b64e319beccf6e" is_pre_election: true
I20240502 01:54:37.808653 26773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d751cdd58eb4cef8039fd4c50855e99" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "401942254c4c43a1ba02c4ae68d17320" is_pre_election: true
I20240502 01:54:37.808871 26773 raft_consensus.cc:2454] T 3d751cdd58eb4cef8039fd4c50855e99 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 0.
I20240502 01:54:37.808900 26379 raft_consensus.cc:2454] T 3d751cdd58eb4cef8039fd4c50855e99 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 0.
I20240502 01:54:37.809278 26578 leader_election.cc:302] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:37.809551 27434 raft_consensus.cc:2781] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:37.809657 27434 raft_consensus.cc:482] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:37.809744 27434 raft_consensus.cc:3037] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:37.811637 27434 raft_consensus.cc:504] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:37.812196 27434 leader_election.cc:288] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 1 election: Requested vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:37.812534 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d751cdd58eb4cef8039fd4c50855e99" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:37.812633 26773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d751cdd58eb4cef8039fd4c50855e99" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:37.812722 26379 raft_consensus.cc:3037] T 3d751cdd58eb4cef8039fd4c50855e99 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:37.812819 26773 raft_consensus.cc:3037] T 3d751cdd58eb4cef8039fd4c50855e99 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:37.814601 26379 raft_consensus.cc:2454] T 3d751cdd58eb4cef8039fd4c50855e99 P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 1.
I20240502 01:54:37.814642 26773 raft_consensus.cc:2454] T 3d751cdd58eb4cef8039fd4c50855e99 P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 1.
I20240502 01:54:37.814963 26577 leader_election.cc:302] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e9d825f803c40e089b64e319beccf6e, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:37.815218 27434 raft_consensus.cc:2781] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:37.815541 27434 raft_consensus.cc:686] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 LEADER]: Becoming Leader. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:37.815796 27434 consensus_queue.cc:237] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:37.818413 26235 catalog_manager.cc:5484] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67). New cstate: current_term: 1 leader_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:37.958660 27422 raft_consensus.cc:482] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:37.958925 27422 raft_consensus.cc:504] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:37.959609 27422 leader_election.cc:288] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:37.960170 26773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "245ca06c39774fb5b0f522550a63ac5b" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "401942254c4c43a1ba02c4ae68d17320" is_pre_election: true
I20240502 01:54:37.960215 26510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "245ca06c39774fb5b0f522550a63ac5b" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc1541b47df446297a369de05163012" is_pre_election: true
I20240502 01:54:37.960430 26773 raft_consensus.cc:2454] T 245ca06c39774fb5b0f522550a63ac5b P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 0.
I20240502 01:54:37.960449 26510 raft_consensus.cc:2454] T 245ca06c39774fb5b0f522550a63ac5b P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 0.
I20240502 01:54:37.960860 26314 leader_election.cc:302] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:37.961182 27422 raft_consensus.cc:2781] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:37.961290 27422 raft_consensus.cc:482] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:37.961403 27422 raft_consensus.cc:3037] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:37.963210 27422 raft_consensus.cc:504] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:37.963758 27422 leader_election.cc:288] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 1 election: Requested vote from peers 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:37.964186 26773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "245ca06c39774fb5b0f522550a63ac5b" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:37.964349 26510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "245ca06c39774fb5b0f522550a63ac5b" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:37.964429 26773 raft_consensus.cc:3037] T 245ca06c39774fb5b0f522550a63ac5b P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:37.964572 26510 raft_consensus.cc:3037] T 245ca06c39774fb5b0f522550a63ac5b P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:37.967192 26773 raft_consensus.cc:2454] T 245ca06c39774fb5b0f522550a63ac5b P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 1.
I20240502 01:54:37.967238 26510 raft_consensus.cc:2454] T 245ca06c39774fb5b0f522550a63ac5b P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 1.
I20240502 01:54:37.967659 26314 leader_election.cc:302] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:37.967927 27422 raft_consensus.cc:2781] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:37.968091 27422 raft_consensus.cc:686] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e [term 1 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:37.968351 27422 consensus_queue.cc:237] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:37.971402 26235 catalog_manager.cc:5484] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 1 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:38.114583 27422 consensus_queue.cc:1035] T 37e818dc0b834eb185e27fcb3cb972e1 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:38.120844 26773 raft_consensus.cc:3037] T 37e818dc0b834eb185e27fcb3cb972e1 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:38.124431 27422 consensus_queue.cc:1035] T 37e818dc0b834eb185e27fcb3cb972e1 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:38.197867 27434 consensus_queue.cc:1035] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:38.204355 27434 consensus_queue.cc:1035] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:54:38.312855 27580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:38.313310 27580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:38.320993 27580 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 01:54:38.358913 27422 consensus_queue.cc:1035] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:38.364802 27575 consensus_queue.cc:1035] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 237.126 ms
time per row: 0.118563 ms
I20240502 01:54:38.733160 26235 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36868:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:38.742478 26621 tablet_service.cc:1467] Processing CreateTablet for tablet 99a4b7d022e1461a8233952ab47ef5d3 (DEFAULT_TABLE table=rebalance_test_table_3 [id=eaef0d67903b4d29ac4990e98661a561]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:38.742637 26357 tablet_service.cc:1467] Processing CreateTablet for tablet 99a4b7d022e1461a8233952ab47ef5d3 (DEFAULT_TABLE table=rebalance_test_table_3 [id=eaef0d67903b4d29ac4990e98661a561]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:38.742949 26621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99a4b7d022e1461a8233952ab47ef5d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:38.743089 26357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99a4b7d022e1461a8233952ab47ef5d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:38.743580 26490 tablet_service.cc:1467] Processing CreateTablet for tablet 99a4b7d022e1461a8233952ab47ef5d3 (DEFAULT_TABLE table=rebalance_test_table_3 [id=eaef0d67903b4d29ac4990e98661a561]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:38.744024 26490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99a4b7d022e1461a8233952ab47ef5d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:38.745819 26622 tablet_service.cc:1467] Processing CreateTablet for tablet dc82c30359e84bddb4bcfe1b3b13ced7 (DEFAULT_TABLE table=rebalance_test_table_3 [id=eaef0d67903b4d29ac4990e98661a561]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:38.746213 26622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc82c30359e84bddb4bcfe1b3b13ced7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:38.746127 26753 tablet_service.cc:1467] Processing CreateTablet for tablet dc82c30359e84bddb4bcfe1b3b13ced7 (DEFAULT_TABLE table=rebalance_test_table_3 [id=eaef0d67903b4d29ac4990e98661a561]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:38.746518 26753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc82c30359e84bddb4bcfe1b3b13ced7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:38.749397 26752 tablet_service.cc:1467] Processing CreateTablet for tablet 0ddd395eec1146bc8a20292d81be4868 (DEFAULT_TABLE table=rebalance_test_table_3 [id=eaef0d67903b4d29ac4990e98661a561]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:38.749748 26752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ddd395eec1146bc8a20292d81be4868. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:38.754222 26489 tablet_service.cc:1467] Processing CreateTablet for tablet dc82c30359e84bddb4bcfe1b3b13ced7 (DEFAULT_TABLE table=rebalance_test_table_3 [id=eaef0d67903b4d29ac4990e98661a561]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:38.754294 26358 tablet_service.cc:1467] Processing CreateTablet for tablet 0ddd395eec1146bc8a20292d81be4868 (DEFAULT_TABLE table=rebalance_test_table_3 [id=eaef0d67903b4d29ac4990e98661a561]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:38.754630 26489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc82c30359e84bddb4bcfe1b3b13ced7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:38.754724 26358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ddd395eec1146bc8a20292d81be4868. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:38.754853 27649 tablet_bootstrap.cc:492] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e: Bootstrap starting.
I20240502 01:54:38.757382 27649 tablet_bootstrap.cc:654] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:38.763711 26622 tablet_service.cc:1467] Processing CreateTablet for tablet 0ddd395eec1146bc8a20292d81be4868 (DEFAULT_TABLE table=rebalance_test_table_3 [id=eaef0d67903b4d29ac4990e98661a561]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:38.764233 26622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ddd395eec1146bc8a20292d81be4868. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:38.768229 27652 tablet_bootstrap.cc:492] T 0ddd395eec1146bc8a20292d81be4868 P 401942254c4c43a1ba02c4ae68d17320: Bootstrap starting.
I20240502 01:54:38.769925 27652 tablet_bootstrap.cc:654] T 0ddd395eec1146bc8a20292d81be4868 P 401942254c4c43a1ba02c4ae68d17320: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:38.770733 27648 tablet_bootstrap.cc:492] T 99a4b7d022e1461a8233952ab47ef5d3 P 83c50029f7a94cfca6d1ee68e0728aa2: Bootstrap starting.
I20240502 01:54:38.772086 27651 tablet_bootstrap.cc:492] T dc82c30359e84bddb4bcfe1b3b13ced7 P 4bc1541b47df446297a369de05163012: Bootstrap starting.
I20240502 01:54:38.772512 27648 tablet_bootstrap.cc:654] T 99a4b7d022e1461a8233952ab47ef5d3 P 83c50029f7a94cfca6d1ee68e0728aa2: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:38.772538 27649 tablet_bootstrap.cc:492] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e: No bootstrap required, opened a new log
I20240502 01:54:38.772679 27649 ts_tablet_manager.cc:1367] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.004s
I20240502 01:54:38.772740 27652 tablet_bootstrap.cc:492] T 0ddd395eec1146bc8a20292d81be4868 P 401942254c4c43a1ba02c4ae68d17320: No bootstrap required, opened a new log
I20240502 01:54:38.772886 27652 ts_tablet_manager.cc:1367] T 0ddd395eec1146bc8a20292d81be4868 P 401942254c4c43a1ba02c4ae68d17320: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 01:54:38.773540 27649 raft_consensus.cc:348] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:38.773757 27649 raft_consensus.cc:374] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:38.773694 27652 raft_consensus.cc:348] T 0ddd395eec1146bc8a20292d81be4868 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:38.773845 27649 raft_consensus.cc:729] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Initialized, Role: FOLLOWER
I20240502 01:54:38.773880 27652 raft_consensus.cc:374] T 0ddd395eec1146bc8a20292d81be4868 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:38.773957 27652 raft_consensus.cc:729] T 0ddd395eec1146bc8a20292d81be4868 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Initialized, Role: FOLLOWER
I20240502 01:54:38.774228 27651 tablet_bootstrap.cc:654] T dc82c30359e84bddb4bcfe1b3b13ced7 P 4bc1541b47df446297a369de05163012: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:38.774173 27649 consensus_queue.cc:260] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:38.774173 27652 consensus_queue.cc:260] T 0ddd395eec1146bc8a20292d81be4868 P 401942254c4c43a1ba02c4ae68d17320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:38.774868 27649 ts_tablet_manager.cc:1398] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:38.775141 27649 tablet_bootstrap.cc:492] T 0ddd395eec1146bc8a20292d81be4868 P 5e9d825f803c40e089b64e319beccf6e: Bootstrap starting.
I20240502 01:54:38.776113 27652 ts_tablet_manager.cc:1398] T 0ddd395eec1146bc8a20292d81be4868 P 401942254c4c43a1ba02c4ae68d17320: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:54:38.776388 27652 tablet_bootstrap.cc:492] T dc82c30359e84bddb4bcfe1b3b13ced7 P 401942254c4c43a1ba02c4ae68d17320: Bootstrap starting.
I20240502 01:54:38.777907 27652 tablet_bootstrap.cc:654] T dc82c30359e84bddb4bcfe1b3b13ced7 P 401942254c4c43a1ba02c4ae68d17320: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:38.781819 27649 tablet_bootstrap.cc:654] T 0ddd395eec1146bc8a20292d81be4868 P 5e9d825f803c40e089b64e319beccf6e: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:38.786190 27648 tablet_bootstrap.cc:492] T 99a4b7d022e1461a8233952ab47ef5d3 P 83c50029f7a94cfca6d1ee68e0728aa2: No bootstrap required, opened a new log
I20240502 01:54:38.786429 27648 ts_tablet_manager.cc:1367] T 99a4b7d022e1461a8233952ab47ef5d3 P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.003s
I20240502 01:54:38.787438 27651 tablet_bootstrap.cc:492] T dc82c30359e84bddb4bcfe1b3b13ced7 P 4bc1541b47df446297a369de05163012: No bootstrap required, opened a new log
I20240502 01:54:38.787334 27648 raft_consensus.cc:348] T 99a4b7d022e1461a8233952ab47ef5d3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:38.787595 27651 ts_tablet_manager.cc:1367] T dc82c30359e84bddb4bcfe1b3b13ced7 P 4bc1541b47df446297a369de05163012: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.009s
I20240502 01:54:38.787693 27648 raft_consensus.cc:374] T 99a4b7d022e1461a8233952ab47ef5d3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:38.787744 27649 tablet_bootstrap.cc:492] T 0ddd395eec1146bc8a20292d81be4868 P 5e9d825f803c40e089b64e319beccf6e: No bootstrap required, opened a new log
I20240502 01:54:38.787873 27648 raft_consensus.cc:729] T 99a4b7d022e1461a8233952ab47ef5d3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Initialized, Role: FOLLOWER
I20240502 01:54:38.787916 27649 ts_tablet_manager.cc:1367] T 0ddd395eec1146bc8a20292d81be4868 P 5e9d825f803c40e089b64e319beccf6e: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20240502 01:54:38.788178 27648 consensus_queue.cc:260] T 99a4b7d022e1461a8233952ab47ef5d3 P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:38.788372 27651 raft_consensus.cc:348] T dc82c30359e84bddb4bcfe1b3b13ced7 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:38.788524 27652 tablet_bootstrap.cc:492] T dc82c30359e84bddb4bcfe1b3b13ced7 P 401942254c4c43a1ba02c4ae68d17320: No bootstrap required, opened a new log
I20240502 01:54:38.788579 27651 raft_consensus.cc:374] T dc82c30359e84bddb4bcfe1b3b13ced7 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:38.788673 27651 raft_consensus.cc:729] T dc82c30359e84bddb4bcfe1b3b13ced7 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bc1541b47df446297a369de05163012, State: Initialized, Role: FOLLOWER
I20240502 01:54:38.788684 27652 ts_tablet_manager.cc:1367] T dc82c30359e84bddb4bcfe1b3b13ced7 P 401942254c4c43a1ba02c4ae68d17320: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20240502 01:54:38.789078 27648 ts_tablet_manager.cc:1398] T 99a4b7d022e1461a8233952ab47ef5d3 P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:54:38.788975 27651 consensus_queue.cc:260] T dc82c30359e84bddb4bcfe1b3b13ced7 P 4bc1541b47df446297a369de05163012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:38.789448 27648 tablet_bootstrap.cc:492] T dc82c30359e84bddb4bcfe1b3b13ced7 P 83c50029f7a94cfca6d1ee68e0728aa2: Bootstrap starting.
I20240502 01:54:38.789425 27652 raft_consensus.cc:348] T dc82c30359e84bddb4bcfe1b3b13ced7 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:38.789644 27652 raft_consensus.cc:374] T dc82c30359e84bddb4bcfe1b3b13ced7 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:38.789729 27652 raft_consensus.cc:729] T dc82c30359e84bddb4bcfe1b3b13ced7 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Initialized, Role: FOLLOWER
I20240502 01:54:38.790017 27652 consensus_queue.cc:260] T dc82c30359e84bddb4bcfe1b3b13ced7 P 401942254c4c43a1ba02c4ae68d17320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:38.790165 27651 ts_tablet_manager.cc:1398] T dc82c30359e84bddb4bcfe1b3b13ced7 P 4bc1541b47df446297a369de05163012: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:38.790450 27651 tablet_bootstrap.cc:492] T 99a4b7d022e1461a8233952ab47ef5d3 P 4bc1541b47df446297a369de05163012: Bootstrap starting.
I20240502 01:54:38.791136 27652 ts_tablet_manager.cc:1398] T dc82c30359e84bddb4bcfe1b3b13ced7 P 401942254c4c43a1ba02c4ae68d17320: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:38.791355 27648 tablet_bootstrap.cc:654] T dc82c30359e84bddb4bcfe1b3b13ced7 P 83c50029f7a94cfca6d1ee68e0728aa2: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:38.792091 27649 raft_consensus.cc:348] T 0ddd395eec1146bc8a20292d81be4868 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:38.792323 27649 raft_consensus.cc:374] T 0ddd395eec1146bc8a20292d81be4868 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:38.792410 27649 raft_consensus.cc:729] T 0ddd395eec1146bc8a20292d81be4868 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Initialized, Role: FOLLOWER
I20240502 01:54:38.792665 27651 tablet_bootstrap.cc:654] T 99a4b7d022e1461a8233952ab47ef5d3 P 4bc1541b47df446297a369de05163012: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:38.792656 27649 consensus_queue.cc:260] T 0ddd395eec1146bc8a20292d81be4868 P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:38.793292 27649 ts_tablet_manager.cc:1398] T 0ddd395eec1146bc8a20292d81be4868 P 5e9d825f803c40e089b64e319beccf6e: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:54:38.798602 27651 tablet_bootstrap.cc:492] T 99a4b7d022e1461a8233952ab47ef5d3 P 4bc1541b47df446297a369de05163012: No bootstrap required, opened a new log
I20240502 01:54:38.798849 27648 tablet_bootstrap.cc:492] T dc82c30359e84bddb4bcfe1b3b13ced7 P 83c50029f7a94cfca6d1ee68e0728aa2: No bootstrap required, opened a new log
I20240502 01:54:38.798844 27651 ts_tablet_manager.cc:1367] T 99a4b7d022e1461a8233952ab47ef5d3 P 4bc1541b47df446297a369de05163012: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20240502 01:54:38.799014 27648 ts_tablet_manager.cc:1367] T dc82c30359e84bddb4bcfe1b3b13ced7 P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20240502 01:54:38.799661 27651 raft_consensus.cc:348] T 99a4b7d022e1461a8233952ab47ef5d3 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:38.799708 27648 raft_consensus.cc:348] T dc82c30359e84bddb4bcfe1b3b13ced7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:38.799836 27651 raft_consensus.cc:374] T 99a4b7d022e1461a8233952ab47ef5d3 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:38.799888 27648 raft_consensus.cc:374] T dc82c30359e84bddb4bcfe1b3b13ced7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:38.799902 27651 raft_consensus.cc:729] T 99a4b7d022e1461a8233952ab47ef5d3 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bc1541b47df446297a369de05163012, State: Initialized, Role: FOLLOWER
I20240502 01:54:38.799979 27648 raft_consensus.cc:729] T dc82c30359e84bddb4bcfe1b3b13ced7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Initialized, Role: FOLLOWER
I20240502 01:54:38.800132 27651 consensus_queue.cc:260] T 99a4b7d022e1461a8233952ab47ef5d3 P 4bc1541b47df446297a369de05163012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:38.800223 27648 consensus_queue.cc:260] T dc82c30359e84bddb4bcfe1b3b13ced7 P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:38.800572 27651 ts_tablet_manager.cc:1398] T 99a4b7d022e1461a8233952ab47ef5d3 P 4bc1541b47df446297a369de05163012: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:54:38.800724 27648 ts_tablet_manager.cc:1398] T dc82c30359e84bddb4bcfe1b3b13ced7 P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:54:38.801067 27648 tablet_bootstrap.cc:492] T 0ddd395eec1146bc8a20292d81be4868 P 83c50029f7a94cfca6d1ee68e0728aa2: Bootstrap starting.
I20240502 01:54:38.802433 27648 tablet_bootstrap.cc:654] T 0ddd395eec1146bc8a20292d81be4868 P 83c50029f7a94cfca6d1ee68e0728aa2: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:38.804838 27648 tablet_bootstrap.cc:492] T 0ddd395eec1146bc8a20292d81be4868 P 83c50029f7a94cfca6d1ee68e0728aa2: No bootstrap required, opened a new log
I20240502 01:54:38.804947 27648 ts_tablet_manager.cc:1367] T 0ddd395eec1146bc8a20292d81be4868 P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 01:54:38.805526 27648 raft_consensus.cc:348] T 0ddd395eec1146bc8a20292d81be4868 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:38.805708 27648 raft_consensus.cc:374] T 0ddd395eec1146bc8a20292d81be4868 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:38.805790 27648 raft_consensus.cc:729] T 0ddd395eec1146bc8a20292d81be4868 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Initialized, Role: FOLLOWER
I20240502 01:54:38.806003 27648 consensus_queue.cc:260] T 0ddd395eec1146bc8a20292d81be4868 P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:38.806488 27648 ts_tablet_manager.cc:1398] T 0ddd395eec1146bc8a20292d81be4868 P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:54:38.819667 27586 raft_consensus.cc:482] T 0ddd395eec1146bc8a20292d81be4868 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:38.819818 27586 raft_consensus.cc:504] T 0ddd395eec1146bc8a20292d81be4868 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:38.820441 27586 leader_election.cc:288] T 0ddd395eec1146bc8a20292d81be4868 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:38.820901 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ddd395eec1146bc8a20292d81be4868" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" is_pre_election: true
I20240502 01:54:38.820910 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ddd395eec1146bc8a20292d81be4868" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9d825f803c40e089b64e319beccf6e" is_pre_election: true
I20240502 01:54:38.821121 26643 raft_consensus.cc:2454] T 0ddd395eec1146bc8a20292d81be4868 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 401942254c4c43a1ba02c4ae68d17320 in term 0.
I20240502 01:54:38.821139 26379 raft_consensus.cc:2454] T 0ddd395eec1146bc8a20292d81be4868 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 401942254c4c43a1ba02c4ae68d17320 in term 0.
I20240502 01:54:38.821518 26708 leader_election.cc:302] T 0ddd395eec1146bc8a20292d81be4868 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:38.821782 27586 raft_consensus.cc:2781] T 0ddd395eec1146bc8a20292d81be4868 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:38.821894 27586 raft_consensus.cc:482] T 0ddd395eec1146bc8a20292d81be4868 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:38.821978 27586 raft_consensus.cc:3037] T 0ddd395eec1146bc8a20292d81be4868 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:38.823776 27586 raft_consensus.cc:504] T 0ddd395eec1146bc8a20292d81be4868 P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:38.824290 27586 leader_election.cc:288] T 0ddd395eec1146bc8a20292d81be4868 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 1 election: Requested vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:38.824676 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ddd395eec1146bc8a20292d81be4868" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:38.824761 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ddd395eec1146bc8a20292d81be4868" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:38.824867 26379 raft_consensus.cc:3037] T 0ddd395eec1146bc8a20292d81be4868 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:38.824963 26643 raft_consensus.cc:3037] T 0ddd395eec1146bc8a20292d81be4868 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:38.827629 26643 raft_consensus.cc:2454] T 0ddd395eec1146bc8a20292d81be4868 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 401942254c4c43a1ba02c4ae68d17320 in term 1.
I20240502 01:54:38.827646 26379 raft_consensus.cc:2454] T 0ddd395eec1146bc8a20292d81be4868 P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 401942254c4c43a1ba02c4ae68d17320 in term 1.
I20240502 01:54:38.828007 26708 leader_election.cc:302] T 0ddd395eec1146bc8a20292d81be4868 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:38.828250 27586 raft_consensus.cc:2781] T 0ddd395eec1146bc8a20292d81be4868 P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:38.828662 27586 raft_consensus.cc:686] T 0ddd395eec1146bc8a20292d81be4868 P 401942254c4c43a1ba02c4ae68d17320 [term 1 LEADER]: Becoming Leader. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Running, Role: LEADER
I20240502 01:54:38.828946 27586 consensus_queue.cc:237] T 0ddd395eec1146bc8a20292d81be4868 P 401942254c4c43a1ba02c4ae68d17320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:38.831630 26227 catalog_manager.cc:5484] T 0ddd395eec1146bc8a20292d81be4868 P 401942254c4c43a1ba02c4ae68d17320 reported cstate change: term changed from 0 to 1, leader changed from <none> to 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68). New cstate: current_term: 1 leader_uuid: "401942254c4c43a1ba02c4ae68d17320" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:38.881130 27638 raft_consensus.cc:482] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:38.881296 27638 raft_consensus.cc:504] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:38.881970 27638 leader_election.cc:288] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:38.882462 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a4b7d022e1461a8233952ab47ef5d3" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" is_pre_election: true
I20240502 01:54:38.882467 26509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a4b7d022e1461a8233952ab47ef5d3" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc1541b47df446297a369de05163012" is_pre_election: true
I20240502 01:54:38.882665 26643 raft_consensus.cc:2454] T 99a4b7d022e1461a8233952ab47ef5d3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 0.
I20240502 01:54:38.882678 26509 raft_consensus.cc:2454] T 99a4b7d022e1461a8233952ab47ef5d3 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 0.
I20240502 01:54:38.883073 26313 leader_election.cc:302] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e9d825f803c40e089b64e319beccf6e, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:38.883381 27638 raft_consensus.cc:2781] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:38.883507 27638 raft_consensus.cc:482] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:38.883587 27638 raft_consensus.cc:3037] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:38.885561 27638 raft_consensus.cc:504] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:38.886133 27638 leader_election.cc:288] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 1 election: Requested vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:38.886545 26509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a4b7d022e1461a8233952ab47ef5d3" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:38.886602 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a4b7d022e1461a8233952ab47ef5d3" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:38.886708 26509 raft_consensus.cc:3037] T 99a4b7d022e1461a8233952ab47ef5d3 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:38.886773 26643 raft_consensus.cc:3037] T 99a4b7d022e1461a8233952ab47ef5d3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:38.888630 26643 raft_consensus.cc:2454] T 99a4b7d022e1461a8233952ab47ef5d3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 1.
I20240502 01:54:38.888640 26509 raft_consensus.cc:2454] T 99a4b7d022e1461a8233952ab47ef5d3 P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 1.
I20240502 01:54:38.889052 26314 leader_election.cc:302] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 5e9d825f803c40e089b64e319beccf6e, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:38.889371 27638 raft_consensus.cc:2781] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:38.889523 27638 raft_consensus.cc:686] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [term 1 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:38.889804 27638 consensus_queue.cc:237] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:38.892904 26227 catalog_manager.cc:5484] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 1 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:39.046846 27626 raft_consensus.cc:482] T dc82c30359e84bddb4bcfe1b3b13ced7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:39.047011 27626 raft_consensus.cc:504] T dc82c30359e84bddb4bcfe1b3b13ced7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:39.047673 27626 leader_election.cc:288] T dc82c30359e84bddb4bcfe1b3b13ced7 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:39.048072 26509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc82c30359e84bddb4bcfe1b3b13ced7" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc1541b47df446297a369de05163012" is_pre_election: true
I20240502 01:54:39.048223 26774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc82c30359e84bddb4bcfe1b3b13ced7" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "401942254c4c43a1ba02c4ae68d17320" is_pre_election: true
I20240502 01:54:39.048350 26509 raft_consensus.cc:2454] T dc82c30359e84bddb4bcfe1b3b13ced7 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 0.
I20240502 01:54:39.048481 26774 raft_consensus.cc:2454] T dc82c30359e84bddb4bcfe1b3b13ced7 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 0.
I20240502 01:54:39.048770 26578 leader_election.cc:302] T dc82c30359e84bddb4bcfe1b3b13ced7 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:39.049049 27626 raft_consensus.cc:2781] T dc82c30359e84bddb4bcfe1b3b13ced7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:39.049178 27626 raft_consensus.cc:482] T dc82c30359e84bddb4bcfe1b3b13ced7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:39.049268 27626 raft_consensus.cc:3037] T dc82c30359e84bddb4bcfe1b3b13ced7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:39.051231 27626 raft_consensus.cc:504] T dc82c30359e84bddb4bcfe1b3b13ced7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:39.051770 27626 leader_election.cc:288] T dc82c30359e84bddb4bcfe1b3b13ced7 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 1 election: Requested vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:39.052163 26509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc82c30359e84bddb4bcfe1b3b13ced7" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:39.052244 26774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc82c30359e84bddb4bcfe1b3b13ced7" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:39.052421 26509 raft_consensus.cc:3037] T dc82c30359e84bddb4bcfe1b3b13ced7 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:39.052426 26774 raft_consensus.cc:3037] T dc82c30359e84bddb4bcfe1b3b13ced7 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:39.055160 26509 raft_consensus.cc:2454] T dc82c30359e84bddb4bcfe1b3b13ced7 P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 1.
I20240502 01:54:39.055161 26774 raft_consensus.cc:2454] T dc82c30359e84bddb4bcfe1b3b13ced7 P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 1.
I20240502 01:54:39.055590 26578 leader_election.cc:302] T dc82c30359e84bddb4bcfe1b3b13ced7 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:39.055905 27626 raft_consensus.cc:2781] T dc82c30359e84bddb4bcfe1b3b13ced7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:39.056216 27626 raft_consensus.cc:686] T dc82c30359e84bddb4bcfe1b3b13ced7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 LEADER]: Becoming Leader. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:39.056488 27626 consensus_queue.cc:237] T dc82c30359e84bddb4bcfe1b3b13ced7 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:39.059020 26227 catalog_manager.cc:5484] T dc82c30359e84bddb4bcfe1b3b13ced7 P 83c50029f7a94cfca6d1ee68e0728aa2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67). New cstate: current_term: 1 leader_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:39.208182 27586 consensus_queue.cc:1035] T 0ddd395eec1146bc8a20292d81be4868 P 401942254c4c43a1ba02c4ae68d17320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:39.216868 27586 consensus_queue.cc:1035] T 0ddd395eec1146bc8a20292d81be4868 P 401942254c4c43a1ba02c4ae68d17320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:54:39.441480 27666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:39.441787 27666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:39.449296 27666 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:54:39.450716 27638 consensus_queue.cc:1035] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:39.456342 27638 consensus_queue.cc:1035] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
I20240502 01:54:39.630201 27664 consensus_queue.cc:1035] T dc82c30359e84bddb4bcfe1b3b13ced7 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:39.652595 26511 raft_consensus.cc:1261] T dc82c30359e84bddb4bcfe1b3b13ced7 P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Refusing update from remote peer 83c50029f7a94cfca6d1ee68e0728aa2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:39.653388 27664 consensus_queue.cc:1035] T dc82c30359e84bddb4bcfe1b3b13ced7 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:39.690999 27715 mvcc.cc:205] Tried to move back new op lower bound from 7023062547048222720 to 7023062544617627648. Current Snapshot: MvccSnapshot[applied={T|T < 7023062547048222720}]
I20240502 01:54:39.697180 27680 mvcc.cc:205] Tried to move back new op lower bound from 7023062547048222720 to 7023062544617627648. Current Snapshot: MvccSnapshot[applied={T|T < 7023062547048222720}]
INSERT report
rows total: 2000
time total: 248.059 ms
time per row: 0.12403 ms
I20240502 01:54:39.841629 26229 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36868:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:39.850872 26358 tablet_service.cc:1467] Processing CreateTablet for tablet 23dcaac2d7d14adf820d826f05c1abc6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=53863c326d78467f81b7448e15816188]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:39.851071 26489 tablet_service.cc:1467] Processing CreateTablet for tablet 23dcaac2d7d14adf820d826f05c1abc6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=53863c326d78467f81b7448e15816188]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:39.851383 26358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23dcaac2d7d14adf820d826f05c1abc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:39.851503 26489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23dcaac2d7d14adf820d826f05c1abc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:39.852459 26621 tablet_service.cc:1467] Processing CreateTablet for tablet 23dcaac2d7d14adf820d826f05c1abc6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=53863c326d78467f81b7448e15816188]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:39.852466 26622 tablet_service.cc:1467] Processing CreateTablet for tablet 89107a545fcc48aead1e199da1f78311 (DEFAULT_TABLE table=rebalance_test_table_4 [id=53863c326d78467f81b7448e15816188]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:39.852883 26621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23dcaac2d7d14adf820d826f05c1abc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:39.853114 26622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89107a545fcc48aead1e199da1f78311. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:39.854312 26752 tablet_service.cc:1467] Processing CreateTablet for tablet 1ce4dd22bb314eb7855126e3fe1785f6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=53863c326d78467f81b7448e15816188]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:39.854454 26753 tablet_service.cc:1467] Processing CreateTablet for tablet 89107a545fcc48aead1e199da1f78311 (DEFAULT_TABLE table=rebalance_test_table_4 [id=53863c326d78467f81b7448e15816188]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:39.854749 26752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ce4dd22bb314eb7855126e3fe1785f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:39.855047 26753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89107a545fcc48aead1e199da1f78311. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:39.855713 26357 tablet_service.cc:1467] Processing CreateTablet for tablet 1ce4dd22bb314eb7855126e3fe1785f6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=53863c326d78467f81b7448e15816188]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:39.856078 26357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ce4dd22bb314eb7855126e3fe1785f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:39.856532 26490 tablet_service.cc:1467] Processing CreateTablet for tablet 1ce4dd22bb314eb7855126e3fe1785f6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=53863c326d78467f81b7448e15816188]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:39.856880 26490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ce4dd22bb314eb7855126e3fe1785f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:39.861348 26491 tablet_service.cc:1467] Processing CreateTablet for tablet 89107a545fcc48aead1e199da1f78311 (DEFAULT_TABLE table=rebalance_test_table_4 [id=53863c326d78467f81b7448e15816188]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:39.861687 26491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89107a545fcc48aead1e199da1f78311. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:39.871850 27733 tablet_bootstrap.cc:492] T 1ce4dd22bb314eb7855126e3fe1785f6 P 401942254c4c43a1ba02c4ae68d17320: Bootstrap starting.
I20240502 01:54:39.872280 27732 tablet_bootstrap.cc:492] T 23dcaac2d7d14adf820d826f05c1abc6 P 5e9d825f803c40e089b64e319beccf6e: Bootstrap starting.
I20240502 01:54:39.873546 27730 tablet_bootstrap.cc:492] T 89107a545fcc48aead1e199da1f78311 P 83c50029f7a94cfca6d1ee68e0728aa2: Bootstrap starting.
I20240502 01:54:39.874173 27731 tablet_bootstrap.cc:492] T 89107a545fcc48aead1e199da1f78311 P 4bc1541b47df446297a369de05163012: Bootstrap starting.
I20240502 01:54:39.874960 27733 tablet_bootstrap.cc:654] T 1ce4dd22bb314eb7855126e3fe1785f6 P 401942254c4c43a1ba02c4ae68d17320: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:39.875157 27730 tablet_bootstrap.cc:654] T 89107a545fcc48aead1e199da1f78311 P 83c50029f7a94cfca6d1ee68e0728aa2: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:39.875289 27732 tablet_bootstrap.cc:654] T 23dcaac2d7d14adf820d826f05c1abc6 P 5e9d825f803c40e089b64e319beccf6e: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:39.875824 27731 tablet_bootstrap.cc:654] T 89107a545fcc48aead1e199da1f78311 P 4bc1541b47df446297a369de05163012: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:39.880107 27730 tablet_bootstrap.cc:492] T 89107a545fcc48aead1e199da1f78311 P 83c50029f7a94cfca6d1ee68e0728aa2: No bootstrap required, opened a new log
I20240502 01:54:39.880249 27730 ts_tablet_manager.cc:1367] T 89107a545fcc48aead1e199da1f78311 P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240502 01:54:39.881059 27730 raft_consensus.cc:348] T 89107a545fcc48aead1e199da1f78311 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:39.881279 27730 raft_consensus.cc:374] T 89107a545fcc48aead1e199da1f78311 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:39.881374 27730 raft_consensus.cc:729] T 89107a545fcc48aead1e199da1f78311 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Initialized, Role: FOLLOWER
I20240502 01:54:39.881647 27730 consensus_queue.cc:260] T 89107a545fcc48aead1e199da1f78311 P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:39.882185 27730 ts_tablet_manager.cc:1398] T 89107a545fcc48aead1e199da1f78311 P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:54:39.882993 27730 tablet_bootstrap.cc:492] T 23dcaac2d7d14adf820d826f05c1abc6 P 83c50029f7a94cfca6d1ee68e0728aa2: Bootstrap starting.
I20240502 01:54:39.883603 27731 tablet_bootstrap.cc:492] T 89107a545fcc48aead1e199da1f78311 P 4bc1541b47df446297a369de05163012: No bootstrap required, opened a new log
I20240502 01:54:39.884056 27733 tablet_bootstrap.cc:492] T 1ce4dd22bb314eb7855126e3fe1785f6 P 401942254c4c43a1ba02c4ae68d17320: No bootstrap required, opened a new log
I20240502 01:54:39.884239 27732 tablet_bootstrap.cc:492] T 23dcaac2d7d14adf820d826f05c1abc6 P 5e9d825f803c40e089b64e319beccf6e: No bootstrap required, opened a new log
I20240502 01:54:39.884254 27733 ts_tablet_manager.cc:1367] T 1ce4dd22bb314eb7855126e3fe1785f6 P 401942254c4c43a1ba02c4ae68d17320: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20240502 01:54:39.884446 27732 ts_tablet_manager.cc:1367] T 23dcaac2d7d14adf820d826f05c1abc6 P 5e9d825f803c40e089b64e319beccf6e: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20240502 01:54:39.884630 27731 ts_tablet_manager.cc:1367] T 89107a545fcc48aead1e199da1f78311 P 4bc1541b47df446297a369de05163012: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20240502 01:54:39.884833 27730 tablet_bootstrap.cc:654] T 23dcaac2d7d14adf820d826f05c1abc6 P 83c50029f7a94cfca6d1ee68e0728aa2: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:39.885058 27733 raft_consensus.cc:348] T 1ce4dd22bb314eb7855126e3fe1785f6 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:39.885318 27733 raft_consensus.cc:374] T 1ce4dd22bb314eb7855126e3fe1785f6 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:39.885460 27733 raft_consensus.cc:729] T 1ce4dd22bb314eb7855126e3fe1785f6 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Initialized, Role: FOLLOWER
I20240502 01:54:39.885783 27732 raft_consensus.cc:348] T 23dcaac2d7d14adf820d826f05c1abc6 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:39.885768 27733 consensus_queue.cc:260] T 1ce4dd22bb314eb7855126e3fe1785f6 P 401942254c4c43a1ba02c4ae68d17320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:39.886451 27733 ts_tablet_manager.cc:1398] T 1ce4dd22bb314eb7855126e3fe1785f6 P 401942254c4c43a1ba02c4ae68d17320: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:54:39.886971 27733 tablet_bootstrap.cc:492] T 89107a545fcc48aead1e199da1f78311 P 401942254c4c43a1ba02c4ae68d17320: Bootstrap starting.
I20240502 01:54:39.887970 27731 raft_consensus.cc:348] T 89107a545fcc48aead1e199da1f78311 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:39.888175 27731 raft_consensus.cc:374] T 89107a545fcc48aead1e199da1f78311 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:39.888274 27731 raft_consensus.cc:729] T 89107a545fcc48aead1e199da1f78311 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bc1541b47df446297a369de05163012, State: Initialized, Role: FOLLOWER
I20240502 01:54:39.888478 27732 raft_consensus.cc:374] T 23dcaac2d7d14adf820d826f05c1abc6 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:39.888664 27733 tablet_bootstrap.cc:654] T 89107a545fcc48aead1e199da1f78311 P 401942254c4c43a1ba02c4ae68d17320: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:39.888650 27732 raft_consensus.cc:729] T 23dcaac2d7d14adf820d826f05c1abc6 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Initialized, Role: FOLLOWER
I20240502 01:54:39.889349 27730 tablet_bootstrap.cc:492] T 23dcaac2d7d14adf820d826f05c1abc6 P 83c50029f7a94cfca6d1ee68e0728aa2: No bootstrap required, opened a new log
I20240502 01:54:39.889503 27730 ts_tablet_manager.cc:1367] T 23dcaac2d7d14adf820d826f05c1abc6 P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20240502 01:54:39.889428 27731 consensus_queue.cc:260] T 89107a545fcc48aead1e199da1f78311 P 4bc1541b47df446297a369de05163012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:39.888913 27732 consensus_queue.cc:260] T 23dcaac2d7d14adf820d826f05c1abc6 P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:39.890301 27732 ts_tablet_manager.cc:1398] T 23dcaac2d7d14adf820d826f05c1abc6 P 5e9d825f803c40e089b64e319beccf6e: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:54:39.890218 27730 raft_consensus.cc:348] T 23dcaac2d7d14adf820d826f05c1abc6 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:39.890436 27730 raft_consensus.cc:374] T 23dcaac2d7d14adf820d826f05c1abc6 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:39.890511 27730 raft_consensus.cc:729] T 23dcaac2d7d14adf820d826f05c1abc6 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Initialized, Role: FOLLOWER
I20240502 01:54:39.890635 27732 tablet_bootstrap.cc:492] T 1ce4dd22bb314eb7855126e3fe1785f6 P 5e9d825f803c40e089b64e319beccf6e: Bootstrap starting.
I20240502 01:54:39.890763 27730 consensus_queue.cc:260] T 23dcaac2d7d14adf820d826f05c1abc6 P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:39.891597 27731 ts_tablet_manager.cc:1398] T 89107a545fcc48aead1e199da1f78311 P 4bc1541b47df446297a369de05163012: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 01:54:39.891978 27730 ts_tablet_manager.cc:1398] T 23dcaac2d7d14adf820d826f05c1abc6 P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:54:39.892066 27731 tablet_bootstrap.cc:492] T 1ce4dd22bb314eb7855126e3fe1785f6 P 4bc1541b47df446297a369de05163012: Bootstrap starting.
I20240502 01:54:39.894971 27731 tablet_bootstrap.cc:654] T 1ce4dd22bb314eb7855126e3fe1785f6 P 4bc1541b47df446297a369de05163012: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:39.896184 27732 tablet_bootstrap.cc:654] T 1ce4dd22bb314eb7855126e3fe1785f6 P 5e9d825f803c40e089b64e319beccf6e: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:39.896823 27733 tablet_bootstrap.cc:492] T 89107a545fcc48aead1e199da1f78311 P 401942254c4c43a1ba02c4ae68d17320: No bootstrap required, opened a new log
I20240502 01:54:39.897053 27733 ts_tablet_manager.cc:1367] T 89107a545fcc48aead1e199da1f78311 P 401942254c4c43a1ba02c4ae68d17320: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20240502 01:54:39.897804 27733 raft_consensus.cc:348] T 89107a545fcc48aead1e199da1f78311 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:39.898079 27733 raft_consensus.cc:374] T 89107a545fcc48aead1e199da1f78311 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:39.898187 27733 raft_consensus.cc:729] T 89107a545fcc48aead1e199da1f78311 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Initialized, Role: FOLLOWER
I20240502 01:54:39.898536 27733 consensus_queue.cc:260] T 89107a545fcc48aead1e199da1f78311 P 401942254c4c43a1ba02c4ae68d17320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:39.899036 27732 tablet_bootstrap.cc:492] T 1ce4dd22bb314eb7855126e3fe1785f6 P 5e9d825f803c40e089b64e319beccf6e: No bootstrap required, opened a new log
I20240502 01:54:39.899118 27733 ts_tablet_manager.cc:1398] T 89107a545fcc48aead1e199da1f78311 P 401942254c4c43a1ba02c4ae68d17320: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:54:39.899199 27732 ts_tablet_manager.cc:1367] T 1ce4dd22bb314eb7855126e3fe1785f6 P 5e9d825f803c40e089b64e319beccf6e: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240502 01:54:39.900020 27731 tablet_bootstrap.cc:492] T 1ce4dd22bb314eb7855126e3fe1785f6 P 4bc1541b47df446297a369de05163012: No bootstrap required, opened a new log
I20240502 01:54:39.899932 27732 raft_consensus.cc:348] T 1ce4dd22bb314eb7855126e3fe1785f6 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:39.900139 27732 raft_consensus.cc:374] T 1ce4dd22bb314eb7855126e3fe1785f6 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:39.900167 27731 ts_tablet_manager.cc:1367] T 1ce4dd22bb314eb7855126e3fe1785f6 P 4bc1541b47df446297a369de05163012: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.001s
I20240502 01:54:39.900229 27732 raft_consensus.cc:729] T 1ce4dd22bb314eb7855126e3fe1785f6 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Initialized, Role: FOLLOWER
I20240502 01:54:39.900486 27732 consensus_queue.cc:260] T 1ce4dd22bb314eb7855126e3fe1785f6 P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:39.900940 27731 raft_consensus.cc:348] T 1ce4dd22bb314eb7855126e3fe1785f6 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:39.901147 27732 ts_tablet_manager.cc:1398] T 1ce4dd22bb314eb7855126e3fe1785f6 P 5e9d825f803c40e089b64e319beccf6e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:54:39.901180 27731 raft_consensus.cc:374] T 1ce4dd22bb314eb7855126e3fe1785f6 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:39.901252 27731 raft_consensus.cc:729] T 1ce4dd22bb314eb7855126e3fe1785f6 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bc1541b47df446297a369de05163012, State: Initialized, Role: FOLLOWER
I20240502 01:54:39.901521 27731 consensus_queue.cc:260] T 1ce4dd22bb314eb7855126e3fe1785f6 P 4bc1541b47df446297a369de05163012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:39.903256 27731 ts_tablet_manager.cc:1398] T 1ce4dd22bb314eb7855126e3fe1785f6 P 4bc1541b47df446297a369de05163012: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:39.903517 27731 tablet_bootstrap.cc:492] T 23dcaac2d7d14adf820d826f05c1abc6 P 4bc1541b47df446297a369de05163012: Bootstrap starting.
I20240502 01:54:39.905344 27731 tablet_bootstrap.cc:654] T 23dcaac2d7d14adf820d826f05c1abc6 P 4bc1541b47df446297a369de05163012: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:39.906919 27731 tablet_bootstrap.cc:492] T 23dcaac2d7d14adf820d826f05c1abc6 P 4bc1541b47df446297a369de05163012: No bootstrap required, opened a new log
I20240502 01:54:39.907023 27731 ts_tablet_manager.cc:1367] T 23dcaac2d7d14adf820d826f05c1abc6 P 4bc1541b47df446297a369de05163012: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:54:39.907550 27731 raft_consensus.cc:348] T 23dcaac2d7d14adf820d826f05c1abc6 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:39.907685 27731 raft_consensus.cc:374] T 23dcaac2d7d14adf820d826f05c1abc6 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:39.907749 27731 raft_consensus.cc:729] T 23dcaac2d7d14adf820d826f05c1abc6 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bc1541b47df446297a369de05163012, State: Initialized, Role: FOLLOWER
I20240502 01:54:39.907953 27731 consensus_queue.cc:260] T 23dcaac2d7d14adf820d826f05c1abc6 P 4bc1541b47df446297a369de05163012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:39.908424 27731 ts_tablet_manager.cc:1398] T 23dcaac2d7d14adf820d826f05c1abc6 P 4bc1541b47df446297a369de05163012: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:54:39.917256 27586 raft_consensus.cc:482] T 1ce4dd22bb314eb7855126e3fe1785f6 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:39.917397 27586 raft_consensus.cc:504] T 1ce4dd22bb314eb7855126e3fe1785f6 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:39.917928 27586 leader_election.cc:288] T 1ce4dd22bb314eb7855126e3fe1785f6 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085)
I20240502 01:54:39.918299 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ce4dd22bb314eb7855126e3fe1785f6" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc1541b47df446297a369de05163012" is_pre_election: true
I20240502 01:54:39.918442 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ce4dd22bb314eb7855126e3fe1785f6" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9d825f803c40e089b64e319beccf6e" is_pre_election: true
I20240502 01:54:39.918569 26511 raft_consensus.cc:2454] T 1ce4dd22bb314eb7855126e3fe1785f6 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 401942254c4c43a1ba02c4ae68d17320 in term 0.
I20240502 01:54:39.918660 26379 raft_consensus.cc:2454] T 1ce4dd22bb314eb7855126e3fe1785f6 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 401942254c4c43a1ba02c4ae68d17320 in term 0.
I20240502 01:54:39.918983 26709 leader_election.cc:302] T 1ce4dd22bb314eb7855126e3fe1785f6 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 4bc1541b47df446297a369de05163012; no voters:
I20240502 01:54:39.919262 27586 raft_consensus.cc:2781] T 1ce4dd22bb314eb7855126e3fe1785f6 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:39.919345 27586 raft_consensus.cc:482] T 1ce4dd22bb314eb7855126e3fe1785f6 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:39.919445 27586 raft_consensus.cc:3037] T 1ce4dd22bb314eb7855126e3fe1785f6 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:39.921528 27586 raft_consensus.cc:504] T 1ce4dd22bb314eb7855126e3fe1785f6 P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:39.922046 27586 leader_election.cc:288] T 1ce4dd22bb314eb7855126e3fe1785f6 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 1 election: Requested vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085)
I20240502 01:54:39.922423 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ce4dd22bb314eb7855126e3fe1785f6" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:39.922449 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ce4dd22bb314eb7855126e3fe1785f6" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:39.922628 26379 raft_consensus.cc:3037] T 1ce4dd22bb314eb7855126e3fe1785f6 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:39.922628 26511 raft_consensus.cc:3037] T 1ce4dd22bb314eb7855126e3fe1785f6 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:39.925367 26379 raft_consensus.cc:2454] T 1ce4dd22bb314eb7855126e3fe1785f6 P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 401942254c4c43a1ba02c4ae68d17320 in term 1.
I20240502 01:54:39.925367 26511 raft_consensus.cc:2454] T 1ce4dd22bb314eb7855126e3fe1785f6 P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 401942254c4c43a1ba02c4ae68d17320 in term 1.
I20240502 01:54:39.925782 26709 leader_election.cc:302] T 1ce4dd22bb314eb7855126e3fe1785f6 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 4bc1541b47df446297a369de05163012; no voters:
I20240502 01:54:39.926074 27586 raft_consensus.cc:2781] T 1ce4dd22bb314eb7855126e3fe1785f6 P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:39.926235 27586 raft_consensus.cc:686] T 1ce4dd22bb314eb7855126e3fe1785f6 P 401942254c4c43a1ba02c4ae68d17320 [term 1 LEADER]: Becoming Leader. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Running, Role: LEADER
I20240502 01:54:39.926504 27586 consensus_queue.cc:237] T 1ce4dd22bb314eb7855126e3fe1785f6 P 401942254c4c43a1ba02c4ae68d17320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:39.928948 26234 catalog_manager.cc:5484] T 1ce4dd22bb314eb7855126e3fe1785f6 P 401942254c4c43a1ba02c4ae68d17320 reported cstate change: term changed from 0 to 1, leader changed from <none> to 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68). New cstate: current_term: 1 leader_uuid: "401942254c4c43a1ba02c4ae68d17320" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:39.932955 27664 raft_consensus.cc:482] T 23dcaac2d7d14adf820d826f05c1abc6 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:39.933126 27664 raft_consensus.cc:504] T 23dcaac2d7d14adf820d826f05c1abc6 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:39.933697 27664 leader_election.cc:288] T 23dcaac2d7d14adf820d826f05c1abc6 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:39.934008 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23dcaac2d7d14adf820d826f05c1abc6" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9d825f803c40e089b64e319beccf6e" is_pre_election: true
I20240502 01:54:39.934104 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23dcaac2d7d14adf820d826f05c1abc6" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc1541b47df446297a369de05163012" is_pre_election: true
I20240502 01:54:39.934173 26379 raft_consensus.cc:2454] T 23dcaac2d7d14adf820d826f05c1abc6 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 0.
I20240502 01:54:39.934303 26511 raft_consensus.cc:2454] T 23dcaac2d7d14adf820d826f05c1abc6 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 0.
I20240502 01:54:39.934572 26577 leader_election.cc:302] T 23dcaac2d7d14adf820d826f05c1abc6 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e9d825f803c40e089b64e319beccf6e, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:39.934831 27664 raft_consensus.cc:2781] T 23dcaac2d7d14adf820d826f05c1abc6 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:39.934945 27664 raft_consensus.cc:482] T 23dcaac2d7d14adf820d826f05c1abc6 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:39.935021 27664 raft_consensus.cc:3037] T 23dcaac2d7d14adf820d826f05c1abc6 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:39.936784 27664 raft_consensus.cc:504] T 23dcaac2d7d14adf820d826f05c1abc6 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:39.937403 27664 leader_election.cc:288] T 23dcaac2d7d14adf820d826f05c1abc6 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 1 election: Requested vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:39.937827 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23dcaac2d7d14adf820d826f05c1abc6" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:39.937827 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23dcaac2d7d14adf820d826f05c1abc6" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:39.937963 26511 raft_consensus.cc:3037] T 23dcaac2d7d14adf820d826f05c1abc6 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:39.937975 26379 raft_consensus.cc:3037] T 23dcaac2d7d14adf820d826f05c1abc6 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:39.939863 26511 raft_consensus.cc:2454] T 23dcaac2d7d14adf820d826f05c1abc6 P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 1.
I20240502 01:54:39.940093 26379 raft_consensus.cc:2454] T 23dcaac2d7d14adf820d826f05c1abc6 P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 1.
I20240502 01:54:39.940229 26578 leader_election.cc:302] T 23dcaac2d7d14adf820d826f05c1abc6 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:39.940655 27664 raft_consensus.cc:2781] T 23dcaac2d7d14adf820d826f05c1abc6 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:39.940794 27664 raft_consensus.cc:686] T 23dcaac2d7d14adf820d826f05c1abc6 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 LEADER]: Becoming Leader. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:39.941028 27664 consensus_queue.cc:237] T 23dcaac2d7d14adf820d826f05c1abc6 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:39.943480 26234 catalog_manager.cc:5484] T 23dcaac2d7d14adf820d826f05c1abc6 P 83c50029f7a94cfca6d1ee68e0728aa2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67). New cstate: current_term: 1 leader_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:39.987125 27664 raft_consensus.cc:482] T 89107a545fcc48aead1e199da1f78311 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:39.987337 27664 raft_consensus.cc:504] T 89107a545fcc48aead1e199da1f78311 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:39.987905 27664 leader_election.cc:288] T 89107a545fcc48aead1e199da1f78311 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:39.988982 26773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89107a545fcc48aead1e199da1f78311" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "401942254c4c43a1ba02c4ae68d17320" is_pre_election: true
I20240502 01:54:39.989252 26773 raft_consensus.cc:2454] T 89107a545fcc48aead1e199da1f78311 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 0.
I20240502 01:54:39.989620 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89107a545fcc48aead1e199da1f78311" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc1541b47df446297a369de05163012" is_pre_election: true
I20240502 01:54:39.989861 26511 raft_consensus.cc:2454] T 89107a545fcc48aead1e199da1f78311 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 0.
I20240502 01:54:39.990180 26578 leader_election.cc:302] T 89107a545fcc48aead1e199da1f78311 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:39.990684 27664 raft_consensus.cc:2781] T 89107a545fcc48aead1e199da1f78311 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:39.990789 27664 raft_consensus.cc:482] T 89107a545fcc48aead1e199da1f78311 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:39.990907 27664 raft_consensus.cc:3037] T 89107a545fcc48aead1e199da1f78311 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:39.993037 27664 raft_consensus.cc:504] T 89107a545fcc48aead1e199da1f78311 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:39.993664 27664 leader_election.cc:288] T 89107a545fcc48aead1e199da1f78311 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 1 election: Requested vote from peers 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:39.994086 26773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89107a545fcc48aead1e199da1f78311" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:39.994279 26773 raft_consensus.cc:3037] T 89107a545fcc48aead1e199da1f78311 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:39.994275 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89107a545fcc48aead1e199da1f78311" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:39.994477 26511 raft_consensus.cc:3037] T 89107a545fcc48aead1e199da1f78311 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:39.996995 26773 raft_consensus.cc:2454] T 89107a545fcc48aead1e199da1f78311 P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 1.
I20240502 01:54:39.997146 26511 raft_consensus.cc:2454] T 89107a545fcc48aead1e199da1f78311 P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 1.
I20240502 01:54:39.997560 26578 leader_election.cc:302] T 89107a545fcc48aead1e199da1f78311 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:39.997838 27664 raft_consensus.cc:2781] T 89107a545fcc48aead1e199da1f78311 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:39.998003 27664 raft_consensus.cc:686] T 89107a545fcc48aead1e199da1f78311 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 LEADER]: Becoming Leader. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:39.998279 27664 consensus_queue.cc:237] T 89107a545fcc48aead1e199da1f78311 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:40.001427 26234 catalog_manager.cc:5484] T 89107a545fcc48aead1e199da1f78311 P 83c50029f7a94cfca6d1ee68e0728aa2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67). New cstate: current_term: 1 leader_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:40.319872 27670 consensus_queue.cc:1035] T 1ce4dd22bb314eb7855126e3fe1785f6 P 401942254c4c43a1ba02c4ae68d17320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:40.326589 27670 consensus_queue.cc:1035] T 1ce4dd22bb314eb7855126e3fe1785f6 P 401942254c4c43a1ba02c4ae68d17320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:54:40.327590 27747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:40.328248 27747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:40.336652 27747 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:54:40.350569 27664 consensus_queue.cc:1035] T 23dcaac2d7d14adf820d826f05c1abc6 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:40.357229 27664 consensus_queue.cc:1035] T 23dcaac2d7d14adf820d826f05c1abc6 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_4'
I20240502 01:54:40.481171 26510 raft_consensus.cc:1261] T 89107a545fcc48aead1e199da1f78311 P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Refusing update from remote peer 83c50029f7a94cfca6d1ee68e0728aa2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:40.481830 27626 consensus_queue.cc:1035] T 89107a545fcc48aead1e199da1f78311 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:40.483017 26773 raft_consensus.cc:1261] T 89107a545fcc48aead1e199da1f78311 P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Refusing update from remote peer 83c50029f7a94cfca6d1ee68e0728aa2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:40.484026 27765 consensus_queue.cc:1035] T 89107a545fcc48aead1e199da1f78311 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 203.557 ms
time per row: 0.101779 ms
I20240502 01:54:40.704085 26234 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36868:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:40.713547 26622 tablet_service.cc:1467] Processing CreateTablet for tablet c26358e3969143fcbf2f466241b360a7 (DEFAULT_TABLE table=rebalance_test_table_5 [id=caab9961f7294a318835ef81822d1b34]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:40.713608 26752 tablet_service.cc:1467] Processing CreateTablet for tablet c26358e3969143fcbf2f466241b360a7 (DEFAULT_TABLE table=rebalance_test_table_5 [id=caab9961f7294a318835ef81822d1b34]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:40.714020 26622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c26358e3969143fcbf2f466241b360a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:40.714020 26752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c26358e3969143fcbf2f466241b360a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:40.715399 26621 tablet_service.cc:1467] Processing CreateTablet for tablet daef5905fc19473a8728ccd2627270af (DEFAULT_TABLE table=rebalance_test_table_5 [id=caab9961f7294a318835ef81822d1b34]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:40.715796 26621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daef5905fc19473a8728ccd2627270af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:40.716189 26753 tablet_service.cc:1467] Processing CreateTablet for tablet daef5905fc19473a8728ccd2627270af (DEFAULT_TABLE table=rebalance_test_table_5 [id=caab9961f7294a318835ef81822d1b34]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:40.716562 26753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daef5905fc19473a8728ccd2627270af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:40.717417 26623 tablet_service.cc:1467] Processing CreateTablet for tablet 2fd06590221d414b9e896e02b1be6f3e (DEFAULT_TABLE table=rebalance_test_table_5 [id=caab9961f7294a318835ef81822d1b34]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:40.717818 26623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fd06590221d414b9e896e02b1be6f3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:40.718987 26491 tablet_service.cc:1467] Processing CreateTablet for tablet 2fd06590221d414b9e896e02b1be6f3e (DEFAULT_TABLE table=rebalance_test_table_5 [id=caab9961f7294a318835ef81822d1b34]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:40.719413 26491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fd06590221d414b9e896e02b1be6f3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:40.721450 26358 tablet_service.cc:1467] Processing CreateTablet for tablet c26358e3969143fcbf2f466241b360a7 (DEFAULT_TABLE table=rebalance_test_table_5 [id=caab9961f7294a318835ef81822d1b34]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:40.721880 26358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c26358e3969143fcbf2f466241b360a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:40.725296 26357 tablet_service.cc:1467] Processing CreateTablet for tablet daef5905fc19473a8728ccd2627270af (DEFAULT_TABLE table=rebalance_test_table_5 [id=caab9961f7294a318835ef81822d1b34]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:40.725596 26357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daef5905fc19473a8728ccd2627270af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:40.729364 26359 tablet_service.cc:1467] Processing CreateTablet for tablet 2fd06590221d414b9e896e02b1be6f3e (DEFAULT_TABLE table=rebalance_test_table_5 [id=caab9961f7294a318835ef81822d1b34]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:40.729701 26359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fd06590221d414b9e896e02b1be6f3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:40.733618 27807 tablet_bootstrap.cc:492] T 2fd06590221d414b9e896e02b1be6f3e P 4bc1541b47df446297a369de05163012: Bootstrap starting.
I20240502 01:54:40.738430 27807 tablet_bootstrap.cc:654] T 2fd06590221d414b9e896e02b1be6f3e P 4bc1541b47df446297a369de05163012: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:40.740566 27808 tablet_bootstrap.cc:492] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2: Bootstrap starting.
I20240502 01:54:40.742175 27810 tablet_bootstrap.cc:492] T 2fd06590221d414b9e896e02b1be6f3e P 5e9d825f803c40e089b64e319beccf6e: Bootstrap starting.
I20240502 01:54:40.742316 27808 tablet_bootstrap.cc:654] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:40.743065 27807 tablet_bootstrap.cc:492] T 2fd06590221d414b9e896e02b1be6f3e P 4bc1541b47df446297a369de05163012: No bootstrap required, opened a new log
I20240502 01:54:40.743269 27807 ts_tablet_manager.cc:1367] T 2fd06590221d414b9e896e02b1be6f3e P 4bc1541b47df446297a369de05163012: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240502 01:54:40.744495 27807 raft_consensus.cc:348] T 2fd06590221d414b9e896e02b1be6f3e P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:40.744822 27807 raft_consensus.cc:374] T 2fd06590221d414b9e896e02b1be6f3e P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:40.744971 27807 raft_consensus.cc:729] T 2fd06590221d414b9e896e02b1be6f3e P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bc1541b47df446297a369de05163012, State: Initialized, Role: FOLLOWER
I20240502 01:54:40.745328 27807 consensus_queue.cc:260] T 2fd06590221d414b9e896e02b1be6f3e P 4bc1541b47df446297a369de05163012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:40.746322 27807 ts_tablet_manager.cc:1398] T 2fd06590221d414b9e896e02b1be6f3e P 4bc1541b47df446297a369de05163012: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:40.746728 27806 tablet_bootstrap.cc:492] T daef5905fc19473a8728ccd2627270af P 401942254c4c43a1ba02c4ae68d17320: Bootstrap starting.
I20240502 01:54:40.747756 27810 tablet_bootstrap.cc:654] T 2fd06590221d414b9e896e02b1be6f3e P 5e9d825f803c40e089b64e319beccf6e: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:40.748406 27806 tablet_bootstrap.cc:654] T daef5905fc19473a8728ccd2627270af P 401942254c4c43a1ba02c4ae68d17320: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:40.751377 27808 tablet_bootstrap.cc:492] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2: No bootstrap required, opened a new log
I20240502 01:54:40.751513 27808 ts_tablet_manager.cc:1367] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20240502 01:54:40.752189 27806 tablet_bootstrap.cc:492] T daef5905fc19473a8728ccd2627270af P 401942254c4c43a1ba02c4ae68d17320: No bootstrap required, opened a new log
I20240502 01:54:40.752339 27806 ts_tablet_manager.cc:1367] T daef5905fc19473a8728ccd2627270af P 401942254c4c43a1ba02c4ae68d17320: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 01:54:40.752238 27808 raft_consensus.cc:348] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:40.752442 27808 raft_consensus.cc:374] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:40.752542 27808 raft_consensus.cc:729] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Initialized, Role: FOLLOWER
I20240502 01:54:40.752588 27810 tablet_bootstrap.cc:492] T 2fd06590221d414b9e896e02b1be6f3e P 5e9d825f803c40e089b64e319beccf6e: No bootstrap required, opened a new log
I20240502 01:54:40.752713 27810 ts_tablet_manager.cc:1367] T 2fd06590221d414b9e896e02b1be6f3e P 5e9d825f803c40e089b64e319beccf6e: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240502 01:54:40.752760 27808 consensus_queue.cc:260] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:40.753046 27806 raft_consensus.cc:348] T daef5905fc19473a8728ccd2627270af P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:40.753242 27806 raft_consensus.cc:374] T daef5905fc19473a8728ccd2627270af P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:40.753340 27806 raft_consensus.cc:729] T daef5905fc19473a8728ccd2627270af P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Initialized, Role: FOLLOWER
I20240502 01:54:40.753455 27810 raft_consensus.cc:348] T 2fd06590221d414b9e896e02b1be6f3e P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:40.753546 27806 consensus_queue.cc:260] T daef5905fc19473a8728ccd2627270af P 401942254c4c43a1ba02c4ae68d17320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:40.753664 27810 raft_consensus.cc:374] T 2fd06590221d414b9e896e02b1be6f3e P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:40.753758 27810 raft_consensus.cc:729] T 2fd06590221d414b9e896e02b1be6f3e P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Initialized, Role: FOLLOWER
I20240502 01:54:40.754065 27806 ts_tablet_manager.cc:1398] T daef5905fc19473a8728ccd2627270af P 401942254c4c43a1ba02c4ae68d17320: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:54:40.754370 27808 ts_tablet_manager.cc:1398] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:54:40.754406 27806 tablet_bootstrap.cc:492] T c26358e3969143fcbf2f466241b360a7 P 401942254c4c43a1ba02c4ae68d17320: Bootstrap starting.
I20240502 01:54:40.754745 27810 consensus_queue.cc:260] T 2fd06590221d414b9e896e02b1be6f3e P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:40.755406 27810 ts_tablet_manager.cc:1398] T 2fd06590221d414b9e896e02b1be6f3e P 5e9d825f803c40e089b64e319beccf6e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:40.755674 27810 tablet_bootstrap.cc:492] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e: Bootstrap starting.
I20240502 01:54:40.755947 27806 tablet_bootstrap.cc:654] T c26358e3969143fcbf2f466241b360a7 P 401942254c4c43a1ba02c4ae68d17320: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:40.757750 27806 tablet_bootstrap.cc:492] T c26358e3969143fcbf2f466241b360a7 P 401942254c4c43a1ba02c4ae68d17320: No bootstrap required, opened a new log
I20240502 01:54:40.757962 27806 ts_tablet_manager.cc:1367] T c26358e3969143fcbf2f466241b360a7 P 401942254c4c43a1ba02c4ae68d17320: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240502 01:54:40.757984 27808 tablet_bootstrap.cc:492] T daef5905fc19473a8728ccd2627270af P 83c50029f7a94cfca6d1ee68e0728aa2: Bootstrap starting.
I20240502 01:54:40.758302 27810 tablet_bootstrap.cc:654] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:40.758845 27806 raft_consensus.cc:348] T c26358e3969143fcbf2f466241b360a7 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:40.759136 27806 raft_consensus.cc:374] T c26358e3969143fcbf2f466241b360a7 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:40.759256 27806 raft_consensus.cc:729] T c26358e3969143fcbf2f466241b360a7 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Initialized, Role: FOLLOWER
I20240502 01:54:40.759483 27808 tablet_bootstrap.cc:654] T daef5905fc19473a8728ccd2627270af P 83c50029f7a94cfca6d1ee68e0728aa2: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:40.759624 27806 consensus_queue.cc:260] T c26358e3969143fcbf2f466241b360a7 P 401942254c4c43a1ba02c4ae68d17320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:40.760349 27806 ts_tablet_manager.cc:1398] T c26358e3969143fcbf2f466241b360a7 P 401942254c4c43a1ba02c4ae68d17320: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:54:40.764293 27808 tablet_bootstrap.cc:492] T daef5905fc19473a8728ccd2627270af P 83c50029f7a94cfca6d1ee68e0728aa2: No bootstrap required, opened a new log
I20240502 01:54:40.764477 27808 ts_tablet_manager.cc:1367] T daef5905fc19473a8728ccd2627270af P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.005s
I20240502 01:54:40.764513 27810 tablet_bootstrap.cc:492] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e: No bootstrap required, opened a new log
I20240502 01:54:40.764645 27810 ts_tablet_manager.cc:1367] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240502 01:54:40.765074 27808 raft_consensus.cc:348] T daef5905fc19473a8728ccd2627270af P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:40.765231 27808 raft_consensus.cc:374] T daef5905fc19473a8728ccd2627270af P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:40.765300 27808 raft_consensus.cc:729] T daef5905fc19473a8728ccd2627270af P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Initialized, Role: FOLLOWER
I20240502 01:54:40.765224 27810 raft_consensus.cc:348] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:40.765414 27810 raft_consensus.cc:374] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:40.765491 27810 raft_consensus.cc:729] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Initialized, Role: FOLLOWER
I20240502 01:54:40.765512 27808 consensus_queue.cc:260] T daef5905fc19473a8728ccd2627270af P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:40.765702 27810 consensus_queue.cc:260] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:40.766016 27808 ts_tablet_manager.cc:1398] T daef5905fc19473a8728ccd2627270af P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:54:40.766201 27810 ts_tablet_manager.cc:1398] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:54:40.766369 27808 tablet_bootstrap.cc:492] T 2fd06590221d414b9e896e02b1be6f3e P 83c50029f7a94cfca6d1ee68e0728aa2: Bootstrap starting.
I20240502 01:54:40.766497 27810 tablet_bootstrap.cc:492] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e: Bootstrap starting.
I20240502 01:54:40.767701 27808 tablet_bootstrap.cc:654] T 2fd06590221d414b9e896e02b1be6f3e P 83c50029f7a94cfca6d1ee68e0728aa2: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:40.767951 27810 tablet_bootstrap.cc:654] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:40.770308 27808 tablet_bootstrap.cc:492] T 2fd06590221d414b9e896e02b1be6f3e P 83c50029f7a94cfca6d1ee68e0728aa2: No bootstrap required, opened a new log
I20240502 01:54:40.770470 27808 ts_tablet_manager.cc:1367] T 2fd06590221d414b9e896e02b1be6f3e P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240502 01:54:40.771235 27808 raft_consensus.cc:348] T 2fd06590221d414b9e896e02b1be6f3e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:40.771422 27808 raft_consensus.cc:374] T 2fd06590221d414b9e896e02b1be6f3e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:40.771517 27808 raft_consensus.cc:729] T 2fd06590221d414b9e896e02b1be6f3e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Initialized, Role: FOLLOWER
I20240502 01:54:40.771551 27810 tablet_bootstrap.cc:492] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e: No bootstrap required, opened a new log
I20240502 01:54:40.771670 27810 ts_tablet_manager.cc:1367] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 01:54:40.771730 27808 consensus_queue.cc:260] T 2fd06590221d414b9e896e02b1be6f3e P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:40.772222 27808 ts_tablet_manager.cc:1398] T 2fd06590221d414b9e896e02b1be6f3e P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:54:40.772241 27810 raft_consensus.cc:348] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:40.772408 27810 raft_consensus.cc:374] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:40.772500 27810 raft_consensus.cc:729] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Initialized, Role: FOLLOWER
I20240502 01:54:40.772689 27810 consensus_queue.cc:260] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:40.773155 27810 ts_tablet_manager.cc:1398] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 01:54:40.906486 27800 raft_consensus.cc:482] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:40.906661 27800 raft_consensus.cc:504] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:40.907325 27800 leader_election.cc:288] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:40.907383 27717 raft_consensus.cc:482] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:40.907552 27717 raft_consensus.cc:504] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:40.907999 26773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daef5905fc19473a8728ccd2627270af" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "401942254c4c43a1ba02c4ae68d17320" is_pre_election: true
I20240502 01:54:40.908249 27717 leader_election.cc:288] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:40.908150 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daef5905fc19473a8728ccd2627270af" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" is_pre_election: true
I20240502 01:54:40.908238 26773 raft_consensus.cc:2454] T daef5905fc19473a8728ccd2627270af P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 0.
I20240502 01:54:40.908404 26642 raft_consensus.cc:2454] T daef5905fc19473a8728ccd2627270af P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 0.
I20240502 01:54:40.908761 26314 leader_election.cc:302] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:40.908918 26773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c26358e3969143fcbf2f466241b360a7" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "401942254c4c43a1ba02c4ae68d17320" is_pre_election: true
I20240502 01:54:40.909029 27717 raft_consensus.cc:2781] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:40.909054 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c26358e3969143fcbf2f466241b360a7" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" is_pre_election: true
I20240502 01:54:40.909138 27717 raft_consensus.cc:482] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:40.909148 26773 raft_consensus.cc:2454] T c26358e3969143fcbf2f466241b360a7 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 0.
I20240502 01:54:40.909229 27717 raft_consensus.cc:3037] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:40.909233 26642 raft_consensus.cc:2454] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 0.
I20240502 01:54:40.909564 26314 leader_election.cc:302] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:40.909827 27800 raft_consensus.cc:2781] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:40.909952 27800 raft_consensus.cc:482] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:40.910051 27800 raft_consensus.cc:3037] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:40.911458 27717 raft_consensus.cc:504] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:40.911950 27800 raft_consensus.cc:504] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:40.912000 27717 leader_election.cc:288] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 1 election: Requested vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:40.912356 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daef5905fc19473a8728ccd2627270af" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:40.912550 26642 raft_consensus.cc:3037] T daef5905fc19473a8728ccd2627270af P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:40.912508 26773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daef5905fc19473a8728ccd2627270af" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:40.912673 26773 raft_consensus.cc:3037] T daef5905fc19473a8728ccd2627270af P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:40.912699 27800 leader_election.cc:288] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 1 election: Requested vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:40.912930 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c26358e3969143fcbf2f466241b360a7" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:40.913167 26643 raft_consensus.cc:3037] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:40.913558 26774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c26358e3969143fcbf2f466241b360a7" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:40.913744 26774 raft_consensus.cc:3037] T c26358e3969143fcbf2f466241b360a7 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:40.915463 26773 raft_consensus.cc:2454] T daef5905fc19473a8728ccd2627270af P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 1.
I20240502 01:54:40.915463 26642 raft_consensus.cc:2454] T daef5905fc19473a8728ccd2627270af P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 1.
I20240502 01:54:40.915836 26314 leader_election.cc:302] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:40.915940 26643 raft_consensus.cc:2454] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 1.
I20240502 01:54:40.916225 27800 raft_consensus.cc:2781] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:40.916414 27800 raft_consensus.cc:686] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [term 1 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:40.916404 26774 raft_consensus.cc:2454] T c26358e3969143fcbf2f466241b360a7 P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 1.
I20240502 01:54:40.916816 26314 leader_election.cc:302] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:40.916783 27800 consensus_queue.cc:237] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:40.917171 27717 raft_consensus.cc:2781] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:40.917645 27717 raft_consensus.cc:686] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [term 1 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:40.917919 27717 consensus_queue.cc:237] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:40.920373 26234 catalog_manager.cc:5484] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 1 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:40.924754 26234 catalog_manager.cc:5484] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 1 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:40.956671 27678 raft_consensus.cc:482] T 2fd06590221d414b9e896e02b1be6f3e P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:40.956854 27678 raft_consensus.cc:504] T 2fd06590221d414b9e896e02b1be6f3e P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:40.958264 27678 leader_election.cc:288] T 2fd06590221d414b9e896e02b1be6f3e P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:40.966162 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fd06590221d414b9e896e02b1be6f3e" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9d825f803c40e089b64e319beccf6e" is_pre_election: true
I20240502 01:54:40.966310 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fd06590221d414b9e896e02b1be6f3e" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" is_pre_election: true
I20240502 01:54:40.966414 26379 raft_consensus.cc:2454] T 2fd06590221d414b9e896e02b1be6f3e P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 0.
I20240502 01:54:40.966531 26643 raft_consensus.cc:2454] T 2fd06590221d414b9e896e02b1be6f3e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 0.
I20240502 01:54:40.967058 26446 leader_election.cc:302] T 2fd06590221d414b9e896e02b1be6f3e P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:40.967494 27678 raft_consensus.cc:2781] T 2fd06590221d414b9e896e02b1be6f3e P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:40.967629 27678 raft_consensus.cc:482] T 2fd06590221d414b9e896e02b1be6f3e P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:40.967747 27678 raft_consensus.cc:3037] T 2fd06590221d414b9e896e02b1be6f3e P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:40.969686 27678 raft_consensus.cc:504] T 2fd06590221d414b9e896e02b1be6f3e P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:40.970211 27678 leader_election.cc:288] T 2fd06590221d414b9e896e02b1be6f3e P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 1 election: Requested vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:40.970597 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fd06590221d414b9e896e02b1be6f3e" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:40.970642 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fd06590221d414b9e896e02b1be6f3e" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:40.970762 26379 raft_consensus.cc:3037] T 2fd06590221d414b9e896e02b1be6f3e P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:40.970813 26643 raft_consensus.cc:3037] T 2fd06590221d414b9e896e02b1be6f3e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:40.972625 26643 raft_consensus.cc:2454] T 2fd06590221d414b9e896e02b1be6f3e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 1.
I20240502 01:54:40.972944 26446 leader_election.cc:302] T 2fd06590221d414b9e896e02b1be6f3e P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:40.973248 27678 raft_consensus.cc:2781] T 2fd06590221d414b9e896e02b1be6f3e P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:40.973820 27678 raft_consensus.cc:686] T 2fd06590221d414b9e896e02b1be6f3e P 4bc1541b47df446297a369de05163012 [term 1 LEADER]: Becoming Leader. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:40.974362 27678 consensus_queue.cc:237] T 2fd06590221d414b9e896e02b1be6f3e P 4bc1541b47df446297a369de05163012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:40.975240 26379 raft_consensus.cc:2454] T 2fd06590221d414b9e896e02b1be6f3e P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 1.
I20240502 01:54:40.977926 26234 catalog_manager.cc:5484] T 2fd06590221d414b9e896e02b1be6f3e P 4bc1541b47df446297a369de05163012 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bc1541b47df446297a369de05163012 (127.11.42.66). New cstate: current_term: 1 leader_uuid: "4bc1541b47df446297a369de05163012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:41.334138 27717 consensus_queue.cc:1035] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:41.341183 27800 consensus_queue.cc:1035] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:54:41.353775 27827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:41.354043 27827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:41.361528 27827 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:54:41.368950 27678 consensus_queue.cc:1035] T 2fd06590221d414b9e896e02b1be6f3e P 4bc1541b47df446297a369de05163012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_5'
I20240502 01:54:41.391098 27678 consensus_queue.cc:1035] T 2fd06590221d414b9e896e02b1be6f3e P 4bc1541b47df446297a369de05163012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:41.410445 27800 consensus_queue.cc:1035] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:41.415778 27717 consensus_queue.cc:1035] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 236.882 ms
time per row: 0.118441 ms
I20240502 01:54:41.741869 26231 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36868:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:41.750869 26491 tablet_service.cc:1467] Processing CreateTablet for tablet c9e4788d50d04d129e8d03125073c6f2 (DEFAULT_TABLE table=rebalance_test_table_6 [id=04c6b62f338646ab9d4de66df458d6ad]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:41.750984 26359 tablet_service.cc:1467] Processing CreateTablet for tablet c9e4788d50d04d129e8d03125073c6f2 (DEFAULT_TABLE table=rebalance_test_table_6 [id=04c6b62f338646ab9d4de66df458d6ad]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:41.751372 26359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9e4788d50d04d129e8d03125073c6f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:41.751379 26491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9e4788d50d04d129e8d03125073c6f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:41.751536 26489 tablet_service.cc:1467] Processing CreateTablet for tablet 22d4681165d1488db2c45ec83cb49cb4 (DEFAULT_TABLE table=rebalance_test_table_6 [id=04c6b62f338646ab9d4de66df458d6ad]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:41.751972 26489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22d4681165d1488db2c45ec83cb49cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:41.752506 26753 tablet_service.cc:1467] Processing CreateTablet for tablet c9e4788d50d04d129e8d03125073c6f2 (DEFAULT_TABLE table=rebalance_test_table_6 [id=04c6b62f338646ab9d4de66df458d6ad]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:41.752908 26753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9e4788d50d04d129e8d03125073c6f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:41.753964 26358 tablet_service.cc:1467] Processing CreateTablet for tablet b5ab243d9f1c4b568d38ed03027adb44 (DEFAULT_TABLE table=rebalance_test_table_6 [id=04c6b62f338646ab9d4de66df458d6ad]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:41.754295 26358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5ab243d9f1c4b568d38ed03027adb44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:41.756718 26490 tablet_service.cc:1467] Processing CreateTablet for tablet b5ab243d9f1c4b568d38ed03027adb44 (DEFAULT_TABLE table=rebalance_test_table_6 [id=04c6b62f338646ab9d4de66df458d6ad]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:41.756733 26754 tablet_service.cc:1467] Processing CreateTablet for tablet b5ab243d9f1c4b568d38ed03027adb44 (DEFAULT_TABLE table=rebalance_test_table_6 [id=04c6b62f338646ab9d4de66df458d6ad]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:41.757109 26754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5ab243d9f1c4b568d38ed03027adb44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:41.757212 26490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5ab243d9f1c4b568d38ed03027adb44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:41.757416 26752 tablet_service.cc:1467] Processing CreateTablet for tablet 22d4681165d1488db2c45ec83cb49cb4 (DEFAULT_TABLE table=rebalance_test_table_6 [id=04c6b62f338646ab9d4de66df458d6ad]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:41.757750 26752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22d4681165d1488db2c45ec83cb49cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:41.761430 26623 tablet_service.cc:1467] Processing CreateTablet for tablet 22d4681165d1488db2c45ec83cb49cb4 (DEFAULT_TABLE table=rebalance_test_table_6 [id=04c6b62f338646ab9d4de66df458d6ad]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:41.761848 26623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22d4681165d1488db2c45ec83cb49cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:41.767565 27894 tablet_bootstrap.cc:492] T c9e4788d50d04d129e8d03125073c6f2 P 5e9d825f803c40e089b64e319beccf6e: Bootstrap starting.
I20240502 01:54:41.767928 27892 tablet_bootstrap.cc:492] T c9e4788d50d04d129e8d03125073c6f2 P 4bc1541b47df446297a369de05163012: Bootstrap starting.
I20240502 01:54:41.769541 27892 tablet_bootstrap.cc:654] T c9e4788d50d04d129e8d03125073c6f2 P 4bc1541b47df446297a369de05163012: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:41.769790 27894 tablet_bootstrap.cc:654] T c9e4788d50d04d129e8d03125073c6f2 P 5e9d825f803c40e089b64e319beccf6e: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:41.770164 27893 tablet_bootstrap.cc:492] T b5ab243d9f1c4b568d38ed03027adb44 P 401942254c4c43a1ba02c4ae68d17320: Bootstrap starting.
I20240502 01:54:41.771756 27893 tablet_bootstrap.cc:654] T b5ab243d9f1c4b568d38ed03027adb44 P 401942254c4c43a1ba02c4ae68d17320: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:41.774423 27894 tablet_bootstrap.cc:492] T c9e4788d50d04d129e8d03125073c6f2 P 5e9d825f803c40e089b64e319beccf6e: No bootstrap required, opened a new log
I20240502 01:54:41.774569 27894 ts_tablet_manager.cc:1367] T c9e4788d50d04d129e8d03125073c6f2 P 5e9d825f803c40e089b64e319beccf6e: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240502 01:54:41.775631 27894 raft_consensus.cc:348] T c9e4788d50d04d129e8d03125073c6f2 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:41.775867 27894 raft_consensus.cc:374] T c9e4788d50d04d129e8d03125073c6f2 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:41.775961 27894 raft_consensus.cc:729] T c9e4788d50d04d129e8d03125073c6f2 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Initialized, Role: FOLLOWER
I20240502 01:54:41.776191 27894 consensus_queue.cc:260] T c9e4788d50d04d129e8d03125073c6f2 P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:41.776798 27894 ts_tablet_manager.cc:1398] T c9e4788d50d04d129e8d03125073c6f2 P 5e9d825f803c40e089b64e319beccf6e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:54:41.777191 27894 tablet_bootstrap.cc:492] T b5ab243d9f1c4b568d38ed03027adb44 P 5e9d825f803c40e089b64e319beccf6e: Bootstrap starting.
I20240502 01:54:41.778751 27894 tablet_bootstrap.cc:654] T b5ab243d9f1c4b568d38ed03027adb44 P 5e9d825f803c40e089b64e319beccf6e: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:41.781733 27893 tablet_bootstrap.cc:492] T b5ab243d9f1c4b568d38ed03027adb44 P 401942254c4c43a1ba02c4ae68d17320: No bootstrap required, opened a new log
I20240502 01:54:41.781898 27898 tablet_bootstrap.cc:492] T 22d4681165d1488db2c45ec83cb49cb4 P 83c50029f7a94cfca6d1ee68e0728aa2: Bootstrap starting.
I20240502 01:54:41.782012 27893 ts_tablet_manager.cc:1367] T b5ab243d9f1c4b568d38ed03027adb44 P 401942254c4c43a1ba02c4ae68d17320: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20240502 01:54:41.782896 27893 raft_consensus.cc:348] T b5ab243d9f1c4b568d38ed03027adb44 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:41.783187 27893 raft_consensus.cc:374] T b5ab243d9f1c4b568d38ed03027adb44 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:41.783335 27893 raft_consensus.cc:729] T b5ab243d9f1c4b568d38ed03027adb44 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Initialized, Role: FOLLOWER
I20240502 01:54:41.783576 27898 tablet_bootstrap.cc:654] T 22d4681165d1488db2c45ec83cb49cb4 P 83c50029f7a94cfca6d1ee68e0728aa2: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:41.783679 27893 consensus_queue.cc:260] T b5ab243d9f1c4b568d38ed03027adb44 P 401942254c4c43a1ba02c4ae68d17320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:41.786031 27893 ts_tablet_manager.cc:1398] T b5ab243d9f1c4b568d38ed03027adb44 P 401942254c4c43a1ba02c4ae68d17320: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:54:41.786356 27893 tablet_bootstrap.cc:492] T c9e4788d50d04d129e8d03125073c6f2 P 401942254c4c43a1ba02c4ae68d17320: Bootstrap starting.
I20240502 01:54:41.786623 27894 tablet_bootstrap.cc:492] T b5ab243d9f1c4b568d38ed03027adb44 P 5e9d825f803c40e089b64e319beccf6e: No bootstrap required, opened a new log
I20240502 01:54:41.786790 27894 ts_tablet_manager.cc:1367] T b5ab243d9f1c4b568d38ed03027adb44 P 5e9d825f803c40e089b64e319beccf6e: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20240502 01:54:41.786803 27898 tablet_bootstrap.cc:492] T 22d4681165d1488db2c45ec83cb49cb4 P 83c50029f7a94cfca6d1ee68e0728aa2: No bootstrap required, opened a new log
I20240502 01:54:41.786938 27898 ts_tablet_manager.cc:1367] T 22d4681165d1488db2c45ec83cb49cb4 P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 01:54:41.787601 27894 raft_consensus.cc:348] T b5ab243d9f1c4b568d38ed03027adb44 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:41.787817 27894 raft_consensus.cc:374] T b5ab243d9f1c4b568d38ed03027adb44 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:41.787766 27898 raft_consensus.cc:348] T 22d4681165d1488db2c45ec83cb49cb4 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:41.787914 27894 raft_consensus.cc:729] T b5ab243d9f1c4b568d38ed03027adb44 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Initialized, Role: FOLLOWER
I20240502 01:54:41.787972 27898 raft_consensus.cc:374] T 22d4681165d1488db2c45ec83cb49cb4 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:41.788067 27898 raft_consensus.cc:729] T 22d4681165d1488db2c45ec83cb49cb4 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Initialized, Role: FOLLOWER
I20240502 01:54:41.788331 27898 consensus_queue.cc:260] T 22d4681165d1488db2c45ec83cb49cb4 P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:41.788906 27898 ts_tablet_manager.cc:1398] T 22d4681165d1488db2c45ec83cb49cb4 P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:54:41.788177 27894 consensus_queue.cc:260] T b5ab243d9f1c4b568d38ed03027adb44 P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:41.789502 27894 ts_tablet_manager.cc:1398] T b5ab243d9f1c4b568d38ed03027adb44 P 5e9d825f803c40e089b64e319beccf6e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:41.790026 27893 tablet_bootstrap.cc:654] T c9e4788d50d04d129e8d03125073c6f2 P 401942254c4c43a1ba02c4ae68d17320: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:41.795310 27893 tablet_bootstrap.cc:492] T c9e4788d50d04d129e8d03125073c6f2 P 401942254c4c43a1ba02c4ae68d17320: No bootstrap required, opened a new log
I20240502 01:54:41.795354 27892 tablet_bootstrap.cc:492] T c9e4788d50d04d129e8d03125073c6f2 P 4bc1541b47df446297a369de05163012: No bootstrap required, opened a new log
I20240502 01:54:41.795487 27892 ts_tablet_manager.cc:1367] T c9e4788d50d04d129e8d03125073c6f2 P 4bc1541b47df446297a369de05163012: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.011s
I20240502 01:54:41.795487 27893 ts_tablet_manager.cc:1367] T c9e4788d50d04d129e8d03125073c6f2 P 401942254c4c43a1ba02c4ae68d17320: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.003s
I20240502 01:54:41.796187 27893 raft_consensus.cc:348] T c9e4788d50d04d129e8d03125073c6f2 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:41.796233 27892 raft_consensus.cc:348] T c9e4788d50d04d129e8d03125073c6f2 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:41.796415 27892 raft_consensus.cc:374] T c9e4788d50d04d129e8d03125073c6f2 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:41.796415 27893 raft_consensus.cc:374] T c9e4788d50d04d129e8d03125073c6f2 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:41.796499 27893 raft_consensus.cc:729] T c9e4788d50d04d129e8d03125073c6f2 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Initialized, Role: FOLLOWER
I20240502 01:54:41.796509 27892 raft_consensus.cc:729] T c9e4788d50d04d129e8d03125073c6f2 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bc1541b47df446297a369de05163012, State: Initialized, Role: FOLLOWER
I20240502 01:54:41.796742 27893 consensus_queue.cc:260] T c9e4788d50d04d129e8d03125073c6f2 P 401942254c4c43a1ba02c4ae68d17320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:41.796716 27892 consensus_queue.cc:260] T c9e4788d50d04d129e8d03125073c6f2 P 4bc1541b47df446297a369de05163012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:41.797263 27893 ts_tablet_manager.cc:1398] T c9e4788d50d04d129e8d03125073c6f2 P 401942254c4c43a1ba02c4ae68d17320: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:54:41.797287 27892 ts_tablet_manager.cc:1398] T c9e4788d50d04d129e8d03125073c6f2 P 4bc1541b47df446297a369de05163012: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:41.797551 27892 tablet_bootstrap.cc:492] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012: Bootstrap starting.
I20240502 01:54:41.797572 27893 tablet_bootstrap.cc:492] T 22d4681165d1488db2c45ec83cb49cb4 P 401942254c4c43a1ba02c4ae68d17320: Bootstrap starting.
I20240502 01:54:41.799022 27893 tablet_bootstrap.cc:654] T 22d4681165d1488db2c45ec83cb49cb4 P 401942254c4c43a1ba02c4ae68d17320: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:41.799217 27892 tablet_bootstrap.cc:654] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:41.803118 27892 tablet_bootstrap.cc:492] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012: No bootstrap required, opened a new log
I20240502 01:54:41.803282 27892 ts_tablet_manager.cc:1367] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20240502 01:54:41.803658 27893 tablet_bootstrap.cc:492] T 22d4681165d1488db2c45ec83cb49cb4 P 401942254c4c43a1ba02c4ae68d17320: No bootstrap required, opened a new log
I20240502 01:54:41.803795 27893 ts_tablet_manager.cc:1367] T 22d4681165d1488db2c45ec83cb49cb4 P 401942254c4c43a1ba02c4ae68d17320: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20240502 01:54:41.803994 27892 raft_consensus.cc:348] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:41.804205 27892 raft_consensus.cc:374] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:41.804289 27892 raft_consensus.cc:729] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bc1541b47df446297a369de05163012, State: Initialized, Role: FOLLOWER
I20240502 01:54:41.804520 27893 raft_consensus.cc:348] T 22d4681165d1488db2c45ec83cb49cb4 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:41.804520 27892 consensus_queue.cc:260] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:41.804698 27893 raft_consensus.cc:374] T 22d4681165d1488db2c45ec83cb49cb4 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:41.804793 27893 raft_consensus.cc:729] T 22d4681165d1488db2c45ec83cb49cb4 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Initialized, Role: FOLLOWER
I20240502 01:54:41.805055 27892 ts_tablet_manager.cc:1398] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:54:41.805033 27893 consensus_queue.cc:260] T 22d4681165d1488db2c45ec83cb49cb4 P 401942254c4c43a1ba02c4ae68d17320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:41.805362 27892 tablet_bootstrap.cc:492] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012: Bootstrap starting.
I20240502 01:54:41.806155 27893 ts_tablet_manager.cc:1398] T 22d4681165d1488db2c45ec83cb49cb4 P 401942254c4c43a1ba02c4ae68d17320: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:54:41.806735 27892 tablet_bootstrap.cc:654] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:41.808748 27892 tablet_bootstrap.cc:492] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012: No bootstrap required, opened a new log
I20240502 01:54:41.808874 27892 ts_tablet_manager.cc:1367] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240502 01:54:41.809612 27892 raft_consensus.cc:348] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:41.809800 27892 raft_consensus.cc:374] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:41.809900 27892 raft_consensus.cc:729] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bc1541b47df446297a369de05163012, State: Initialized, Role: FOLLOWER
I20240502 01:54:41.810050 27892 consensus_queue.cc:260] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:41.810470 27892 ts_tablet_manager.cc:1398] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:54:41.836957 27825 raft_consensus.cc:482] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:41.837148 27825 raft_consensus.cc:504] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:41.837838 27825 leader_election.cc:288] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:41.838438 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22d4681165d1488db2c45ec83cb49cb4" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" is_pre_election: true
I20240502 01:54:41.838716 26643 raft_consensus.cc:2454] T 22d4681165d1488db2c45ec83cb49cb4 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 0.
I20240502 01:54:41.839098 26446 leader_election.cc:302] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:41.839406 27825 raft_consensus.cc:2781] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:41.839529 27825 raft_consensus.cc:482] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:41.839629 27825 raft_consensus.cc:3037] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:41.841980 27825 raft_consensus.cc:504] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:41.842711 27825 leader_election.cc:288] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 1 election: Requested vote from peers 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:41.843185 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22d4681165d1488db2c45ec83cb49cb4" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:41.843427 26643 raft_consensus.cc:3037] T 22d4681165d1488db2c45ec83cb49cb4 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:41.845439 26773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22d4681165d1488db2c45ec83cb49cb4" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "401942254c4c43a1ba02c4ae68d17320" is_pre_election: true
I20240502 01:54:41.845536 26774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22d4681165d1488db2c45ec83cb49cb4" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:41.845695 26773 raft_consensus.cc:2454] T 22d4681165d1488db2c45ec83cb49cb4 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 0.
I20240502 01:54:41.845758 26643 raft_consensus.cc:2454] T 22d4681165d1488db2c45ec83cb49cb4 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 1.
I20240502 01:54:41.846207 26446 leader_election.cc:302] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4bc1541b47df446297a369de05163012, 83c50029f7a94cfca6d1ee68e0728aa2; no voters: 401942254c4c43a1ba02c4ae68d17320
I20240502 01:54:41.846292 27717 raft_consensus.cc:482] T c9e4788d50d04d129e8d03125073c6f2 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:41.846541 27825 raft_consensus.cc:2781] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:41.846482 27717 raft_consensus.cc:504] T c9e4788d50d04d129e8d03125073c6f2 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:41.846673 27825 raft_consensus.cc:686] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012 [term 1 LEADER]: Becoming Leader. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:41.846969 27825 consensus_queue.cc:237] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:41.847196 27717 leader_election.cc:288] T c9e4788d50d04d129e8d03125073c6f2 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:41.847931 26510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e4788d50d04d129e8d03125073c6f2" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc1541b47df446297a369de05163012" is_pre_election: true
I20240502 01:54:41.848361 26773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e4788d50d04d129e8d03125073c6f2" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "401942254c4c43a1ba02c4ae68d17320" is_pre_election: true
I20240502 01:54:41.848541 26773 raft_consensus.cc:2454] T c9e4788d50d04d129e8d03125073c6f2 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 0.
I20240502 01:54:41.848613 26510 raft_consensus.cc:2454] T c9e4788d50d04d129e8d03125073c6f2 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 0.
I20240502 01:54:41.848969 26314 leader_election.cc:302] T c9e4788d50d04d129e8d03125073c6f2 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:41.849310 27717 raft_consensus.cc:2781] T c9e4788d50d04d129e8d03125073c6f2 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:41.849418 27717 raft_consensus.cc:482] T c9e4788d50d04d129e8d03125073c6f2 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:41.849525 27717 raft_consensus.cc:3037] T c9e4788d50d04d129e8d03125073c6f2 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:41.850556 26229 catalog_manager.cc:5484] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bc1541b47df446297a369de05163012 (127.11.42.66). New cstate: current_term: 1 leader_uuid: "4bc1541b47df446297a369de05163012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:41.851930 27717 raft_consensus.cc:504] T c9e4788d50d04d129e8d03125073c6f2 P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:41.852463 27717 leader_election.cc:288] T c9e4788d50d04d129e8d03125073c6f2 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 1 election: Requested vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:41.852864 26510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e4788d50d04d129e8d03125073c6f2" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:41.852916 26773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e4788d50d04d129e8d03125073c6f2" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:41.853106 26773 raft_consensus.cc:3037] T c9e4788d50d04d129e8d03125073c6f2 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:41.853107 26510 raft_consensus.cc:3037] T c9e4788d50d04d129e8d03125073c6f2 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:41.855826 26773 raft_consensus.cc:2454] T c9e4788d50d04d129e8d03125073c6f2 P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 1.
I20240502 01:54:41.855842 26510 raft_consensus.cc:2454] T c9e4788d50d04d129e8d03125073c6f2 P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 1.
I20240502 01:54:41.856199 26314 leader_election.cc:302] T c9e4788d50d04d129e8d03125073c6f2 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:41.856501 27717 raft_consensus.cc:2781] T c9e4788d50d04d129e8d03125073c6f2 P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:41.856698 27717 raft_consensus.cc:686] T c9e4788d50d04d129e8d03125073c6f2 P 5e9d825f803c40e089b64e319beccf6e [term 1 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:41.856949 27717 consensus_queue.cc:237] T c9e4788d50d04d129e8d03125073c6f2 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:41.859611 26229 catalog_manager.cc:5484] T c9e4788d50d04d129e8d03125073c6f2 P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 1 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:41.997687 27825 raft_consensus.cc:482] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:41.997911 27825 raft_consensus.cc:504] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:41.998565 27825 leader_election.cc:288] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959), 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085)
I20240502 01:54:41.999171 26773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ab243d9f1c4b568d38ed03027adb44" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "401942254c4c43a1ba02c4ae68d17320" is_pre_election: true
I20240502 01:54:41.999181 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ab243d9f1c4b568d38ed03027adb44" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9d825f803c40e089b64e319beccf6e" is_pre_election: true
I20240502 01:54:41.999442 26379 raft_consensus.cc:2454] T b5ab243d9f1c4b568d38ed03027adb44 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 0.
I20240502 01:54:41.999444 26773 raft_consensus.cc:2454] T b5ab243d9f1c4b568d38ed03027adb44 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 0.
I20240502 01:54:41.999915 26446 leader_election.cc:302] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:42.000222 27825 raft_consensus.cc:2781] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:42.000314 27825 raft_consensus.cc:482] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:42.000414 27825 raft_consensus.cc:3037] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:42.002338 27825 raft_consensus.cc:504] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:42.002967 27825 leader_election.cc:288] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 1 election: Requested vote from peers 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959), 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085)
I20240502 01:54:42.003445 26773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ab243d9f1c4b568d38ed03027adb44" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:42.003511 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ab243d9f1c4b568d38ed03027adb44" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:42.003646 26773 raft_consensus.cc:3037] T b5ab243d9f1c4b568d38ed03027adb44 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:42.003705 26379 raft_consensus.cc:3037] T b5ab243d9f1c4b568d38ed03027adb44 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:42.005509 26773 raft_consensus.cc:2454] T b5ab243d9f1c4b568d38ed03027adb44 P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 1.
I20240502 01:54:42.005649 26379 raft_consensus.cc:2454] T b5ab243d9f1c4b568d38ed03027adb44 P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 1.
I20240502 01:54:42.005898 26447 leader_election.cc:302] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 4bc1541b47df446297a369de05163012; no voters:
I20240502 01:54:42.006167 27825 raft_consensus.cc:2781] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:42.006318 27825 raft_consensus.cc:686] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012 [term 1 LEADER]: Becoming Leader. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:42.006603 27825 consensus_queue.cc:237] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:42.009326 26229 catalog_manager.cc:5484] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bc1541b47df446297a369de05163012 (127.11.42.66). New cstate: current_term: 1 leader_uuid: "4bc1541b47df446297a369de05163012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:42.227399 26773 raft_consensus.cc:3037] T 22d4681165d1488db2c45ec83cb49cb4 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:42.231026 27825 consensus_queue.cc:1035] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:42.237002 27825 consensus_queue.cc:1035] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:42.304076 27717 consensus_queue.cc:1035] T c9e4788d50d04d129e8d03125073c6f2 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:42.311046 27887 consensus_queue.cc:1035] T c9e4788d50d04d129e8d03125073c6f2 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:42.384533 27678 consensus_queue.cc:1035] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:42.389796 27924 consensus_queue.cc:1035] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:54:42.397166 27911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:42.397457 27911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:42.404848 27911 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
W20240502 01:54:42.676962 26314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.42.68:37959, user_credentials={real_user=slave}} blocked reactor thread for 60278.4us
INSERT report
rows total: 2000
time total: 274.318 ms
time per row: 0.137159 ms
I20240502 01:54:42.819752 26228 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36868:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:42.828728 26490 tablet_service.cc:1467] Processing CreateTablet for tablet beff61f07ecf4ffab508da651a686c8e (DEFAULT_TABLE table=rebalance_test_table_7 [id=60d6595ac75b42c288cc49300a9a250f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:42.828886 26359 tablet_service.cc:1467] Processing CreateTablet for tablet beff61f07ecf4ffab508da651a686c8e (DEFAULT_TABLE table=rebalance_test_table_7 [id=60d6595ac75b42c288cc49300a9a250f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:42.829247 26490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beff61f07ecf4ffab508da651a686c8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:42.829327 26359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beff61f07ecf4ffab508da651a686c8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:42.829859 26489 tablet_service.cc:1467] Processing CreateTablet for tablet 14a747ac14354fdab2a2993594d3398f (DEFAULT_TABLE table=rebalance_test_table_7 [id=60d6595ac75b42c288cc49300a9a250f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:42.830332 26489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14a747ac14354fdab2a2993594d3398f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:42.830559 26358 tablet_service.cc:1467] Processing CreateTablet for tablet ef474f44fe5b4a06b2c8dda12b1bc496 (DEFAULT_TABLE table=rebalance_test_table_7 [id=60d6595ac75b42c288cc49300a9a250f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:42.830979 26358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef474f44fe5b4a06b2c8dda12b1bc496. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:42.834671 26623 tablet_service.cc:1467] Processing CreateTablet for tablet beff61f07ecf4ffab508da651a686c8e (DEFAULT_TABLE table=rebalance_test_table_7 [id=60d6595ac75b42c288cc49300a9a250f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:42.835132 26623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beff61f07ecf4ffab508da651a686c8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:42.835813 26752 tablet_service.cc:1467] Processing CreateTablet for tablet 14a747ac14354fdab2a2993594d3398f (DEFAULT_TABLE table=rebalance_test_table_7 [id=60d6595ac75b42c288cc49300a9a250f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:42.836254 26752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14a747ac14354fdab2a2993594d3398f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:42.837442 26622 tablet_service.cc:1467] Processing CreateTablet for tablet 14a747ac14354fdab2a2993594d3398f (DEFAULT_TABLE table=rebalance_test_table_7 [id=60d6595ac75b42c288cc49300a9a250f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:42.837765 26622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14a747ac14354fdab2a2993594d3398f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:42.839001 27974 tablet_bootstrap.cc:492] T beff61f07ecf4ffab508da651a686c8e P 5e9d825f803c40e089b64e319beccf6e: Bootstrap starting.
I20240502 01:54:42.840626 27974 tablet_bootstrap.cc:654] T beff61f07ecf4ffab508da651a686c8e P 5e9d825f803c40e089b64e319beccf6e: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:42.841368 26621 tablet_service.cc:1467] Processing CreateTablet for tablet ef474f44fe5b4a06b2c8dda12b1bc496 (DEFAULT_TABLE table=rebalance_test_table_7 [id=60d6595ac75b42c288cc49300a9a250f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:42.841725 26621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef474f44fe5b4a06b2c8dda12b1bc496. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:42.844518 26754 tablet_service.cc:1467] Processing CreateTablet for tablet ef474f44fe5b4a06b2c8dda12b1bc496 (DEFAULT_TABLE table=rebalance_test_table_7 [id=60d6595ac75b42c288cc49300a9a250f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:42.844911 26754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef474f44fe5b4a06b2c8dda12b1bc496. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:42.849614 27974 tablet_bootstrap.cc:492] T beff61f07ecf4ffab508da651a686c8e P 5e9d825f803c40e089b64e319beccf6e: No bootstrap required, opened a new log
I20240502 01:54:42.849754 27974 ts_tablet_manager.cc:1367] T beff61f07ecf4ffab508da651a686c8e P 5e9d825f803c40e089b64e319beccf6e: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20240502 01:54:42.850467 27974 raft_consensus.cc:348] T beff61f07ecf4ffab508da651a686c8e P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:42.850654 27974 raft_consensus.cc:374] T beff61f07ecf4ffab508da651a686c8e P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:42.850731 27974 raft_consensus.cc:729] T beff61f07ecf4ffab508da651a686c8e P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Initialized, Role: FOLLOWER
I20240502 01:54:42.850941 27974 consensus_queue.cc:260] T beff61f07ecf4ffab508da651a686c8e P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:42.851471 27974 ts_tablet_manager.cc:1398] T beff61f07ecf4ffab508da651a686c8e P 5e9d825f803c40e089b64e319beccf6e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:54:42.851771 27974 tablet_bootstrap.cc:492] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e: Bootstrap starting.
I20240502 01:54:42.853250 27974 tablet_bootstrap.cc:654] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:42.858189 27975 tablet_bootstrap.cc:492] T beff61f07ecf4ffab508da651a686c8e P 4bc1541b47df446297a369de05163012: Bootstrap starting.
I20240502 01:54:42.861450 27975 tablet_bootstrap.cc:654] T beff61f07ecf4ffab508da651a686c8e P 4bc1541b47df446297a369de05163012: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:42.868072 27975 tablet_bootstrap.cc:492] T beff61f07ecf4ffab508da651a686c8e P 4bc1541b47df446297a369de05163012: No bootstrap required, opened a new log
I20240502 01:54:42.868289 27975 ts_tablet_manager.cc:1367] T beff61f07ecf4ffab508da651a686c8e P 4bc1541b47df446297a369de05163012: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20240502 01:54:42.869354 27975 raft_consensus.cc:348] T beff61f07ecf4ffab508da651a686c8e P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:42.869648 27975 raft_consensus.cc:374] T beff61f07ecf4ffab508da651a686c8e P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:42.869776 27975 raft_consensus.cc:729] T beff61f07ecf4ffab508da651a686c8e P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bc1541b47df446297a369de05163012, State: Initialized, Role: FOLLOWER
I20240502 01:54:42.870082 27975 consensus_queue.cc:260] T beff61f07ecf4ffab508da651a686c8e P 4bc1541b47df446297a369de05163012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:42.870751 27975 ts_tablet_manager.cc:1398] T beff61f07ecf4ffab508da651a686c8e P 4bc1541b47df446297a369de05163012: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:42.871098 27975 tablet_bootstrap.cc:492] T 14a747ac14354fdab2a2993594d3398f P 4bc1541b47df446297a369de05163012: Bootstrap starting.
I20240502 01:54:42.875708 27979 tablet_bootstrap.cc:492] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2: Bootstrap starting.
I20240502 01:54:42.876207 27975 tablet_bootstrap.cc:654] T 14a747ac14354fdab2a2993594d3398f P 4bc1541b47df446297a369de05163012: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:42.877503 27979 tablet_bootstrap.cc:654] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:42.879093 27977 tablet_bootstrap.cc:492] T 14a747ac14354fdab2a2993594d3398f P 401942254c4c43a1ba02c4ae68d17320: Bootstrap starting.
I20240502 01:54:42.879630 27975 tablet_bootstrap.cc:492] T 14a747ac14354fdab2a2993594d3398f P 4bc1541b47df446297a369de05163012: No bootstrap required, opened a new log
I20240502 01:54:42.879853 27975 ts_tablet_manager.cc:1367] T 14a747ac14354fdab2a2993594d3398f P 4bc1541b47df446297a369de05163012: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.002s
I20240502 01:54:42.880759 27977 tablet_bootstrap.cc:654] T 14a747ac14354fdab2a2993594d3398f P 401942254c4c43a1ba02c4ae68d17320: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:42.880645 27975 raft_consensus.cc:348] T 14a747ac14354fdab2a2993594d3398f P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:42.880949 27975 raft_consensus.cc:374] T 14a747ac14354fdab2a2993594d3398f P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:42.881685 27975 raft_consensus.cc:729] T 14a747ac14354fdab2a2993594d3398f P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bc1541b47df446297a369de05163012, State: Initialized, Role: FOLLOWER
I20240502 01:54:42.882076 27975 consensus_queue.cc:260] T 14a747ac14354fdab2a2993594d3398f P 4bc1541b47df446297a369de05163012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:42.882258 27974 tablet_bootstrap.cc:492] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e: No bootstrap required, opened a new log
I20240502 01:54:42.882274 27979 tablet_bootstrap.cc:492] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2: No bootstrap required, opened a new log
I20240502 01:54:42.882423 27979 ts_tablet_manager.cc:1367] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20240502 01:54:42.882445 27974 ts_tablet_manager.cc:1367] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e: Time spent bootstrapping tablet: real 0.031s user 0.014s sys 0.014s
I20240502 01:54:42.882897 27975 ts_tablet_manager.cc:1398] T 14a747ac14354fdab2a2993594d3398f P 4bc1541b47df446297a369de05163012: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:54:42.883208 27974 raft_consensus.cc:348] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:42.883293 27979 raft_consensus.cc:348] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:42.883468 27979 raft_consensus.cc:374] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:42.883460 27974 raft_consensus.cc:374] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:42.883531 27974 raft_consensus.cc:729] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Initialized, Role: FOLLOWER
I20240502 01:54:42.883546 27979 raft_consensus.cc:729] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Initialized, Role: FOLLOWER
I20240502 01:54:42.883770 27974 consensus_queue.cc:260] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:42.883800 27979 consensus_queue.cc:260] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:42.884335 27974 ts_tablet_manager.cc:1398] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:54:42.884996 27979 ts_tablet_manager.cc:1398] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:54:42.885653 27979 tablet_bootstrap.cc:492] T 14a747ac14354fdab2a2993594d3398f P 83c50029f7a94cfca6d1ee68e0728aa2: Bootstrap starting.
I20240502 01:54:42.889111 27977 tablet_bootstrap.cc:492] T 14a747ac14354fdab2a2993594d3398f P 401942254c4c43a1ba02c4ae68d17320: No bootstrap required, opened a new log
I20240502 01:54:42.889250 27977 ts_tablet_manager.cc:1367] T 14a747ac14354fdab2a2993594d3398f P 401942254c4c43a1ba02c4ae68d17320: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20240502 01:54:42.889322 27979 tablet_bootstrap.cc:654] T 14a747ac14354fdab2a2993594d3398f P 83c50029f7a94cfca6d1ee68e0728aa2: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:42.889883 27977 raft_consensus.cc:348] T 14a747ac14354fdab2a2993594d3398f P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:42.890251 27977 raft_consensus.cc:374] T 14a747ac14354fdab2a2993594d3398f P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:42.890364 27977 raft_consensus.cc:729] T 14a747ac14354fdab2a2993594d3398f P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Initialized, Role: FOLLOWER
I20240502 01:54:42.890630 27977 consensus_queue.cc:260] T 14a747ac14354fdab2a2993594d3398f P 401942254c4c43a1ba02c4ae68d17320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:42.891057 27979 tablet_bootstrap.cc:492] T 14a747ac14354fdab2a2993594d3398f P 83c50029f7a94cfca6d1ee68e0728aa2: No bootstrap required, opened a new log
I20240502 01:54:42.891110 27977 ts_tablet_manager.cc:1398] T 14a747ac14354fdab2a2993594d3398f P 401942254c4c43a1ba02c4ae68d17320: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:54:42.891222 27979 ts_tablet_manager.cc:1367] T 14a747ac14354fdab2a2993594d3398f P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20240502 01:54:42.891506 27977 tablet_bootstrap.cc:492] T ef474f44fe5b4a06b2c8dda12b1bc496 P 401942254c4c43a1ba02c4ae68d17320: Bootstrap starting.
I20240502 01:54:42.891904 27979 raft_consensus.cc:348] T 14a747ac14354fdab2a2993594d3398f P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:42.892125 27979 raft_consensus.cc:374] T 14a747ac14354fdab2a2993594d3398f P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:42.892211 27979 raft_consensus.cc:729] T 14a747ac14354fdab2a2993594d3398f P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Initialized, Role: FOLLOWER
I20240502 01:54:42.892491 27979 consensus_queue.cc:260] T 14a747ac14354fdab2a2993594d3398f P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:42.892824 27977 tablet_bootstrap.cc:654] T ef474f44fe5b4a06b2c8dda12b1bc496 P 401942254c4c43a1ba02c4ae68d17320: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:42.893157 27979 ts_tablet_manager.cc:1398] T 14a747ac14354fdab2a2993594d3398f P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:54:42.893626 27979 tablet_bootstrap.cc:492] T ef474f44fe5b4a06b2c8dda12b1bc496 P 83c50029f7a94cfca6d1ee68e0728aa2: Bootstrap starting.
I20240502 01:54:42.895072 27979 tablet_bootstrap.cc:654] T ef474f44fe5b4a06b2c8dda12b1bc496 P 83c50029f7a94cfca6d1ee68e0728aa2: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:42.900705 27977 tablet_bootstrap.cc:492] T ef474f44fe5b4a06b2c8dda12b1bc496 P 401942254c4c43a1ba02c4ae68d17320: No bootstrap required, opened a new log
I20240502 01:54:42.900846 27977 ts_tablet_manager.cc:1367] T ef474f44fe5b4a06b2c8dda12b1bc496 P 401942254c4c43a1ba02c4ae68d17320: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20240502 01:54:42.901552 27977 raft_consensus.cc:348] T ef474f44fe5b4a06b2c8dda12b1bc496 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:42.901744 27977 raft_consensus.cc:374] T ef474f44fe5b4a06b2c8dda12b1bc496 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:42.901813 27977 raft_consensus.cc:729] T ef474f44fe5b4a06b2c8dda12b1bc496 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Initialized, Role: FOLLOWER
I20240502 01:54:42.902033 27977 consensus_queue.cc:260] T ef474f44fe5b4a06b2c8dda12b1bc496 P 401942254c4c43a1ba02c4ae68d17320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:42.902505 27977 ts_tablet_manager.cc:1398] T ef474f44fe5b4a06b2c8dda12b1bc496 P 401942254c4c43a1ba02c4ae68d17320: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:54:42.902896 27979 tablet_bootstrap.cc:492] T ef474f44fe5b4a06b2c8dda12b1bc496 P 83c50029f7a94cfca6d1ee68e0728aa2: No bootstrap required, opened a new log
I20240502 01:54:42.903033 27979 ts_tablet_manager.cc:1367] T ef474f44fe5b4a06b2c8dda12b1bc496 P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20240502 01:54:42.903616 27979 raft_consensus.cc:348] T ef474f44fe5b4a06b2c8dda12b1bc496 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:42.903762 27979 raft_consensus.cc:374] T ef474f44fe5b4a06b2c8dda12b1bc496 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:42.903831 27979 raft_consensus.cc:729] T ef474f44fe5b4a06b2c8dda12b1bc496 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Initialized, Role: FOLLOWER
I20240502 01:54:42.904035 27979 consensus_queue.cc:260] T ef474f44fe5b4a06b2c8dda12b1bc496 P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:42.904700 27979 ts_tablet_manager.cc:1398] T ef474f44fe5b4a06b2c8dda12b1bc496 P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:54:42.936699 27670 raft_consensus.cc:482] T 14a747ac14354fdab2a2993594d3398f P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:42.936846 27670 raft_consensus.cc:504] T 14a747ac14354fdab2a2993594d3398f P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:42.937527 27670 leader_election.cc:288] T 14a747ac14354fdab2a2993594d3398f P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:42.937885 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a747ac14354fdab2a2993594d3398f" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" is_pre_election: true
I20240502 01:54:42.938107 26643 raft_consensus.cc:2454] T 14a747ac14354fdab2a2993594d3398f P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 401942254c4c43a1ba02c4ae68d17320 in term 0.
I20240502 01:54:42.938063 26510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a747ac14354fdab2a2993594d3398f" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc1541b47df446297a369de05163012" is_pre_election: true
I20240502 01:54:42.938309 26510 raft_consensus.cc:2454] T 14a747ac14354fdab2a2993594d3398f P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 401942254c4c43a1ba02c4ae68d17320 in term 0.
I20240502 01:54:42.938484 26708 leader_election.cc:302] T 14a747ac14354fdab2a2993594d3398f P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:42.938795 27670 raft_consensus.cc:2781] T 14a747ac14354fdab2a2993594d3398f P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:42.938913 27670 raft_consensus.cc:482] T 14a747ac14354fdab2a2993594d3398f P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:42.938999 27670 raft_consensus.cc:3037] T 14a747ac14354fdab2a2993594d3398f P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:42.940897 27670 raft_consensus.cc:504] T 14a747ac14354fdab2a2993594d3398f P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:42.941401 27670 leader_election.cc:288] T 14a747ac14354fdab2a2993594d3398f P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 1 election: Requested vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:42.941800 26510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a747ac14354fdab2a2993594d3398f" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:42.941864 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a747ac14354fdab2a2993594d3398f" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:42.941957 26510 raft_consensus.cc:3037] T 14a747ac14354fdab2a2993594d3398f P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:42.942010 26643 raft_consensus.cc:3037] T 14a747ac14354fdab2a2993594d3398f P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:42.943828 26643 raft_consensus.cc:2454] T 14a747ac14354fdab2a2993594d3398f P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 401942254c4c43a1ba02c4ae68d17320 in term 1.
I20240502 01:54:42.943840 26510 raft_consensus.cc:2454] T 14a747ac14354fdab2a2993594d3398f P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 401942254c4c43a1ba02c4ae68d17320 in term 1.
I20240502 01:54:42.944185 26708 leader_election.cc:302] T 14a747ac14354fdab2a2993594d3398f P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:42.944454 27670 raft_consensus.cc:2781] T 14a747ac14354fdab2a2993594d3398f P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:42.944578 27670 raft_consensus.cc:686] T 14a747ac14354fdab2a2993594d3398f P 401942254c4c43a1ba02c4ae68d17320 [term 1 LEADER]: Becoming Leader. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Running, Role: LEADER
I20240502 01:54:42.944829 27670 consensus_queue.cc:237] T 14a747ac14354fdab2a2993594d3398f P 401942254c4c43a1ba02c4ae68d17320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:42.947615 26235 catalog_manager.cc:5484] T 14a747ac14354fdab2a2993594d3398f P 401942254c4c43a1ba02c4ae68d17320 reported cstate change: term changed from 0 to 1, leader changed from <none> to 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68). New cstate: current_term: 1 leader_uuid: "401942254c4c43a1ba02c4ae68d17320" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:43.020522 27969 raft_consensus.cc:482] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:43.020726 27969 raft_consensus.cc:504] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:43.021557 27969 leader_election.cc:288] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085)
I20240502 01:54:43.021909 26510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beff61f07ecf4ffab508da651a686c8e" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc1541b47df446297a369de05163012" is_pre_election: true
I20240502 01:54:43.022219 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beff61f07ecf4ffab508da651a686c8e" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9d825f803c40e089b64e319beccf6e" is_pre_election: true
I20240502 01:54:43.022287 26510 raft_consensus.cc:2454] T beff61f07ecf4ffab508da651a686c8e P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 0.
I20240502 01:54:43.022483 26379 raft_consensus.cc:2454] T beff61f07ecf4ffab508da651a686c8e P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 0.
I20240502 01:54:43.022686 26578 leader_election.cc:302] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:43.023008 27850 raft_consensus.cc:2781] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:43.023109 27850 raft_consensus.cc:482] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:43.023223 27850 raft_consensus.cc:3037] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:43.025315 27850 raft_consensus.cc:504] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:43.025890 27850 leader_election.cc:288] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 1 election: Requested vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085)
I20240502 01:54:43.026165 26510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beff61f07ecf4ffab508da651a686c8e" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:43.026331 26510 raft_consensus.cc:3037] T beff61f07ecf4ffab508da651a686c8e P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:43.026291 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beff61f07ecf4ffab508da651a686c8e" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:43.026476 26379 raft_consensus.cc:3037] T beff61f07ecf4ffab508da651a686c8e P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:43.028509 26510 raft_consensus.cc:2454] T beff61f07ecf4ffab508da651a686c8e P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 1.
I20240502 01:54:43.028743 26379 raft_consensus.cc:2454] T beff61f07ecf4ffab508da651a686c8e P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 1.
I20240502 01:54:43.028846 26578 leader_election.cc:302] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:43.029114 27850 raft_consensus.cc:2781] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:43.029258 27850 raft_consensus.cc:686] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 LEADER]: Becoming Leader. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:43.029500 27850 consensus_queue.cc:237] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:43.032136 27968 raft_consensus.cc:482] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:43.032264 27968 raft_consensus.cc:504] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:43.032366 26235 catalog_manager.cc:5484] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67). New cstate: current_term: 1 leader_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:43.032897 27968 leader_election.cc:288] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:43.033325 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef474f44fe5b4a06b2c8dda12b1bc496" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" is_pre_election: true
I20240502 01:54:43.033463 26772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef474f44fe5b4a06b2c8dda12b1bc496" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "401942254c4c43a1ba02c4ae68d17320" is_pre_election: true
I20240502 01:54:43.033546 26643 raft_consensus.cc:2454] T ef474f44fe5b4a06b2c8dda12b1bc496 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 0.
I20240502 01:54:43.033703 26772 raft_consensus.cc:2454] T ef474f44fe5b4a06b2c8dda12b1bc496 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 0.
I20240502 01:54:43.034070 26313 leader_election.cc:302] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e9d825f803c40e089b64e319beccf6e, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:43.034402 27968 raft_consensus.cc:2781] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:43.034518 27968 raft_consensus.cc:482] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:43.034631 27968 raft_consensus.cc:3037] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:43.036774 27968 raft_consensus.cc:504] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:43.037324 27968 leader_election.cc:288] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 1 election: Requested vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:43.037659 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef474f44fe5b4a06b2c8dda12b1bc496" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:43.037729 26772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef474f44fe5b4a06b2c8dda12b1bc496" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:43.037853 26643 raft_consensus.cc:3037] T ef474f44fe5b4a06b2c8dda12b1bc496 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:43.037892 26772 raft_consensus.cc:3037] T ef474f44fe5b4a06b2c8dda12b1bc496 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:43.039678 26643 raft_consensus.cc:2454] T ef474f44fe5b4a06b2c8dda12b1bc496 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 1.
I20240502 01:54:43.039790 26772 raft_consensus.cc:2454] T ef474f44fe5b4a06b2c8dda12b1bc496 P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 1.
I20240502 01:54:43.040021 26313 leader_election.cc:302] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e9d825f803c40e089b64e319beccf6e, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:43.040331 27968 raft_consensus.cc:2781] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:43.040462 27968 raft_consensus.cc:686] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [term 1 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:43.040699 27968 consensus_queue.cc:237] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:43.043514 26235 catalog_manager.cc:5484] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 1 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } }
W20240502 01:54:43.414425 27991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:43.414713 27991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 01:54:43.421577 27968 consensus_queue.cc:1035] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:54:43.422451 27991 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:54:43.427276 27968 consensus_queue.cc:1035] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_7'
I20240502 01:54:43.486431 27670 consensus_queue.cc:1035] T 14a747ac14354fdab2a2993594d3398f P 401942254c4c43a1ba02c4ae68d17320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20240502 01:54:43.504801 27670 consensus_queue.cc:1035] T 14a747ac14354fdab2a2993594d3398f P 401942254c4c43a1ba02c4ae68d17320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:43.562526 27850 consensus_queue.cc:1035] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:43.564657 27850 consensus_queue.cc:1035] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 293.939 ms
time per row: 0.146969 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 01:54:44.725992 26231 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36868:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:44.735260 26359 tablet_service.cc:1467] Processing CreateTablet for tablet b4ba7c4fe9614ab3abb3645c1da3fcd3 (DEFAULT_TABLE table=rebalance_test_table_8 [id=35ee5d3ae9844019958277e4e2e5f151]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:44.735766 26359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4ba7c4fe9614ab3abb3645c1da3fcd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:44.736042 26623 tablet_service.cc:1467] Processing CreateTablet for tablet b4ba7c4fe9614ab3abb3645c1da3fcd3 (DEFAULT_TABLE table=rebalance_test_table_8 [id=35ee5d3ae9844019958277e4e2e5f151]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:44.736497 26623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4ba7c4fe9614ab3abb3645c1da3fcd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:44.737535 26489 tablet_service.cc:1467] Processing CreateTablet for tablet b4ba7c4fe9614ab3abb3645c1da3fcd3 (DEFAULT_TABLE table=rebalance_test_table_8 [id=35ee5d3ae9844019958277e4e2e5f151]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:44.737655 26752 tablet_service.cc:1467] Processing CreateTablet for tablet 8cf6da25f535446b9515444e093cae7f (DEFAULT_TABLE table=rebalance_test_table_8 [id=35ee5d3ae9844019958277e4e2e5f151]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:44.737974 26489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4ba7c4fe9614ab3abb3645c1da3fcd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:44.738073 26752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cf6da25f535446b9515444e093cae7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:44.741571 26754 tablet_service.cc:1467] Processing CreateTablet for tablet d4b8975c68d64c2d804fcc852384afde (DEFAULT_TABLE table=rebalance_test_table_8 [id=35ee5d3ae9844019958277e4e2e5f151]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:44.741928 26754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4b8975c68d64c2d804fcc852384afde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:44.743589 26622 tablet_service.cc:1467] Processing CreateTablet for tablet d4b8975c68d64c2d804fcc852384afde (DEFAULT_TABLE table=rebalance_test_table_8 [id=35ee5d3ae9844019958277e4e2e5f151]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:44.743999 26622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4b8975c68d64c2d804fcc852384afde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:44.745477 26490 tablet_service.cc:1467] Processing CreateTablet for tablet 8cf6da25f535446b9515444e093cae7f (DEFAULT_TABLE table=rebalance_test_table_8 [id=35ee5d3ae9844019958277e4e2e5f151]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:44.745877 26490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cf6da25f535446b9515444e093cae7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:44.746484 26358 tablet_service.cc:1467] Processing CreateTablet for tablet 8cf6da25f535446b9515444e093cae7f (DEFAULT_TABLE table=rebalance_test_table_8 [id=35ee5d3ae9844019958277e4e2e5f151]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:44.746870 26358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cf6da25f535446b9515444e093cae7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:44.749375 26491 tablet_service.cc:1467] Processing CreateTablet for tablet d4b8975c68d64c2d804fcc852384afde (DEFAULT_TABLE table=rebalance_test_table_8 [id=35ee5d3ae9844019958277e4e2e5f151]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:44.749743 26491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4b8975c68d64c2d804fcc852384afde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:44.751257 28060 tablet_bootstrap.cc:492] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 83c50029f7a94cfca6d1ee68e0728aa2: Bootstrap starting.
I20240502 01:54:44.752884 28060 tablet_bootstrap.cc:654] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 83c50029f7a94cfca6d1ee68e0728aa2: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:44.759454 28061 tablet_bootstrap.cc:492] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 5e9d825f803c40e089b64e319beccf6e: Bootstrap starting.
I20240502 01:54:44.760924 28061 tablet_bootstrap.cc:654] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 5e9d825f803c40e089b64e319beccf6e: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:44.765926 28060 tablet_bootstrap.cc:492] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 83c50029f7a94cfca6d1ee68e0728aa2: No bootstrap required, opened a new log
I20240502 01:54:44.766081 28060 ts_tablet_manager.cc:1367] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.004s
I20240502 01:54:44.766829 28060 raft_consensus.cc:348] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:44.767048 28060 raft_consensus.cc:374] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:44.767135 28060 raft_consensus.cc:729] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Initialized, Role: FOLLOWER
I20240502 01:54:44.767391 28060 consensus_queue.cc:260] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:44.769389 28061 tablet_bootstrap.cc:492] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 5e9d825f803c40e089b64e319beccf6e: No bootstrap required, opened a new log
I20240502 01:54:44.769553 28061 ts_tablet_manager.cc:1367] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 5e9d825f803c40e089b64e319beccf6e: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20240502 01:54:44.770015 28060 ts_tablet_manager.cc:1398] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 01:54:44.770298 28060 tablet_bootstrap.cc:492] T d4b8975c68d64c2d804fcc852384afde P 83c50029f7a94cfca6d1ee68e0728aa2: Bootstrap starting.
I20240502 01:54:44.770331 28061 raft_consensus.cc:348] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:44.770541 28061 raft_consensus.cc:374] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:44.770633 28061 raft_consensus.cc:729] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Initialized, Role: FOLLOWER
I20240502 01:54:44.770829 28061 consensus_queue.cc:260] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:44.771840 28060 tablet_bootstrap.cc:654] T d4b8975c68d64c2d804fcc852384afde P 83c50029f7a94cfca6d1ee68e0728aa2: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:44.771934 28061 ts_tablet_manager.cc:1398] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 5e9d825f803c40e089b64e319beccf6e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:44.772194 28061 tablet_bootstrap.cc:492] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e: Bootstrap starting.
I20240502 01:54:44.774412 28061 tablet_bootstrap.cc:654] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:44.779356 28061 tablet_bootstrap.cc:492] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e: No bootstrap required, opened a new log
I20240502 01:54:44.779582 28061 ts_tablet_manager.cc:1367] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240502 01:54:44.780405 28061 raft_consensus.cc:348] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:44.780694 28061 raft_consensus.cc:374] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:44.780825 28061 raft_consensus.cc:729] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Initialized, Role: FOLLOWER
I20240502 01:54:44.781126 28061 consensus_queue.cc:260] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:44.781991 28061 ts_tablet_manager.cc:1398] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:44.784396 28066 tablet_bootstrap.cc:492] T 8cf6da25f535446b9515444e093cae7f P 401942254c4c43a1ba02c4ae68d17320: Bootstrap starting.
I20240502 01:54:44.786198 28066 tablet_bootstrap.cc:654] T 8cf6da25f535446b9515444e093cae7f P 401942254c4c43a1ba02c4ae68d17320: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:44.788621 28064 tablet_bootstrap.cc:492] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 4bc1541b47df446297a369de05163012: Bootstrap starting.
I20240502 01:54:44.790640 28064 tablet_bootstrap.cc:654] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 4bc1541b47df446297a369de05163012: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:44.792542 28060 tablet_bootstrap.cc:492] T d4b8975c68d64c2d804fcc852384afde P 83c50029f7a94cfca6d1ee68e0728aa2: No bootstrap required, opened a new log
I20240502 01:54:44.792729 28060 ts_tablet_manager.cc:1367] T d4b8975c68d64c2d804fcc852384afde P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent bootstrapping tablet: real 0.022s user 0.014s sys 0.000s
I20240502 01:54:44.792889 28066 tablet_bootstrap.cc:492] T 8cf6da25f535446b9515444e093cae7f P 401942254c4c43a1ba02c4ae68d17320: No bootstrap required, opened a new log
I20240502 01:54:44.793066 28066 ts_tablet_manager.cc:1367] T 8cf6da25f535446b9515444e093cae7f P 401942254c4c43a1ba02c4ae68d17320: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240502 01:54:44.793532 28060 raft_consensus.cc:348] T d4b8975c68d64c2d804fcc852384afde P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:44.793761 28060 raft_consensus.cc:374] T d4b8975c68d64c2d804fcc852384afde P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:44.793848 28060 raft_consensus.cc:729] T d4b8975c68d64c2d804fcc852384afde P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Initialized, Role: FOLLOWER
I20240502 01:54:44.793787 28066 raft_consensus.cc:348] T 8cf6da25f535446b9515444e093cae7f P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:44.794124 28060 consensus_queue.cc:260] T d4b8975c68d64c2d804fcc852384afde P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:44.794588 28064 tablet_bootstrap.cc:492] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 4bc1541b47df446297a369de05163012: No bootstrap required, opened a new log
I20240502 01:54:44.794704 28060 ts_tablet_manager.cc:1398] T d4b8975c68d64c2d804fcc852384afde P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:44.794728 28064 ts_tablet_manager.cc:1367] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 4bc1541b47df446297a369de05163012: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240502 01:54:44.795333 28064 raft_consensus.cc:348] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:44.795485 28064 raft_consensus.cc:374] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:44.795559 28064 raft_consensus.cc:729] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bc1541b47df446297a369de05163012, State: Initialized, Role: FOLLOWER
I20240502 01:54:44.795754 28064 consensus_queue.cc:260] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 4bc1541b47df446297a369de05163012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:44.795993 28066 raft_consensus.cc:374] T 8cf6da25f535446b9515444e093cae7f P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:44.796087 28066 raft_consensus.cc:729] T 8cf6da25f535446b9515444e093cae7f P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Initialized, Role: FOLLOWER
I20240502 01:54:44.796276 28066 consensus_queue.cc:260] T 8cf6da25f535446b9515444e093cae7f P 401942254c4c43a1ba02c4ae68d17320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:44.796900 28066 ts_tablet_manager.cc:1398] T 8cf6da25f535446b9515444e093cae7f P 401942254c4c43a1ba02c4ae68d17320: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:54:44.797866 28064 ts_tablet_manager.cc:1398] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 4bc1541b47df446297a369de05163012: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:54:44.797247 28066 tablet_bootstrap.cc:492] T d4b8975c68d64c2d804fcc852384afde P 401942254c4c43a1ba02c4ae68d17320: Bootstrap starting.
I20240502 01:54:44.799149 28066 tablet_bootstrap.cc:654] T d4b8975c68d64c2d804fcc852384afde P 401942254c4c43a1ba02c4ae68d17320: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:44.799468 28064 tablet_bootstrap.cc:492] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012: Bootstrap starting.
I20240502 01:54:44.800735 28064 tablet_bootstrap.cc:654] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:44.802951 28064 tablet_bootstrap.cc:492] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012: No bootstrap required, opened a new log
I20240502 01:54:44.803117 28064 ts_tablet_manager.cc:1367] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:54:44.803750 28066 tablet_bootstrap.cc:492] T d4b8975c68d64c2d804fcc852384afde P 401942254c4c43a1ba02c4ae68d17320: No bootstrap required, opened a new log
I20240502 01:54:44.803695 28064 raft_consensus.cc:348] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:44.803858 28064 raft_consensus.cc:374] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:44.803900 28066 ts_tablet_manager.cc:1367] T d4b8975c68d64c2d804fcc852384afde P 401942254c4c43a1ba02c4ae68d17320: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 01:54:44.803930 28064 raft_consensus.cc:729] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bc1541b47df446297a369de05163012, State: Initialized, Role: FOLLOWER
I20240502 01:54:44.804134 28064 consensus_queue.cc:260] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:44.804591 28064 ts_tablet_manager.cc:1398] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 01:54:44.804548 28066 raft_consensus.cc:348] T d4b8975c68d64c2d804fcc852384afde P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:44.804723 28066 raft_consensus.cc:374] T d4b8975c68d64c2d804fcc852384afde P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:44.804795 28066 raft_consensus.cc:729] T d4b8975c68d64c2d804fcc852384afde P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Initialized, Role: FOLLOWER
I20240502 01:54:44.804879 28064 tablet_bootstrap.cc:492] T 8cf6da25f535446b9515444e093cae7f P 4bc1541b47df446297a369de05163012: Bootstrap starting.
I20240502 01:54:44.805058 28066 consensus_queue.cc:260] T d4b8975c68d64c2d804fcc852384afde P 401942254c4c43a1ba02c4ae68d17320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:44.805581 28066 ts_tablet_manager.cc:1398] T d4b8975c68d64c2d804fcc852384afde P 401942254c4c43a1ba02c4ae68d17320: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:54:44.806301 28064 tablet_bootstrap.cc:654] T 8cf6da25f535446b9515444e093cae7f P 4bc1541b47df446297a369de05163012: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:44.808609 28064 tablet_bootstrap.cc:492] T 8cf6da25f535446b9515444e093cae7f P 4bc1541b47df446297a369de05163012: No bootstrap required, opened a new log
I20240502 01:54:44.808748 28064 ts_tablet_manager.cc:1367] T 8cf6da25f535446b9515444e093cae7f P 4bc1541b47df446297a369de05163012: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:54:44.809399 28064 raft_consensus.cc:348] T 8cf6da25f535446b9515444e093cae7f P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:44.809626 28064 raft_consensus.cc:374] T 8cf6da25f535446b9515444e093cae7f P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:44.809741 28064 raft_consensus.cc:729] T 8cf6da25f535446b9515444e093cae7f P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bc1541b47df446297a369de05163012, State: Initialized, Role: FOLLOWER
I20240502 01:54:44.810158 28064 consensus_queue.cc:260] T 8cf6da25f535446b9515444e093cae7f P 4bc1541b47df446297a369de05163012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:44.811302 28064 ts_tablet_manager.cc:1398] T 8cf6da25f535446b9515444e093cae7f P 4bc1541b47df446297a369de05163012: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:44.995172 28067 raft_consensus.cc:482] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:44.995357 28067 raft_consensus.cc:504] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:44.996018 28067 leader_election.cc:288] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:44.996448 26772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cf6da25f535446b9515444e093cae7f" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "401942254c4c43a1ba02c4ae68d17320" is_pre_election: true
I20240502 01:54:44.996613 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cf6da25f535446b9515444e093cae7f" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc1541b47df446297a369de05163012" is_pre_election: true
I20240502 01:54:44.996737 26772 raft_consensus.cc:2454] T 8cf6da25f535446b9515444e093cae7f P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 0.
I20240502 01:54:44.996857 26511 raft_consensus.cc:2454] T 8cf6da25f535446b9515444e093cae7f P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 0.
I20240502 01:54:44.997323 26314 leader_election.cc:302] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:44.997624 28067 raft_consensus.cc:2781] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:44.997730 28067 raft_consensus.cc:482] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:44.997817 28067 raft_consensus.cc:3037] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:44.999950 28067 raft_consensus.cc:504] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:45.000547 27825 raft_consensus.cc:482] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:45.000555 28067 leader_election.cc:288] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 1 election: Requested vote from peers 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:45.000707 27825 raft_consensus.cc:504] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:45.000980 26772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cf6da25f535446b9515444e093cae7f" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:45.001051 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cf6da25f535446b9515444e093cae7f" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:45.001253 26511 raft_consensus.cc:3037] T 8cf6da25f535446b9515444e093cae7f P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:45.001248 26772 raft_consensus.cc:3037] T 8cf6da25f535446b9515444e093cae7f P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:45.001447 27825 leader_election.cc:288] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:45.002171 26773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4b8975c68d64c2d804fcc852384afde" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "401942254c4c43a1ba02c4ae68d17320" is_pre_election: true
I20240502 01:54:45.002382 26773 raft_consensus.cc:2454] T d4b8975c68d64c2d804fcc852384afde P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 0.
I20240502 01:54:45.002776 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4b8975c68d64c2d804fcc852384afde" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" is_pre_election: true
I20240502 01:54:45.002856 26447 leader_election.cc:302] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 4bc1541b47df446297a369de05163012; no voters:
I20240502 01:54:45.003023 26643 raft_consensus.cc:2454] T d4b8975c68d64c2d804fcc852384afde P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 0.
I20240502 01:54:45.003391 27825 raft_consensus.cc:2781] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:45.003504 27825 raft_consensus.cc:482] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:45.003598 27825 raft_consensus.cc:3037] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:45.004032 26772 raft_consensus.cc:2454] T 8cf6da25f535446b9515444e093cae7f P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 1.
I20240502 01:54:45.004060 26511 raft_consensus.cc:2454] T 8cf6da25f535446b9515444e093cae7f P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 1.
I20240502 01:54:45.004420 26314 leader_election.cc:302] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:45.004684 28067 raft_consensus.cc:2781] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:45.004966 28067 raft_consensus.cc:686] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e [term 1 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:45.005333 28067 consensus_queue.cc:237] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:45.005920 27825 raft_consensus.cc:504] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:45.006590 27825 leader_election.cc:288] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 1 election: Requested vote from peers 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:45.006986 26772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4b8975c68d64c2d804fcc852384afde" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:45.007167 26772 raft_consensus.cc:3037] T d4b8975c68d64c2d804fcc852384afde P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:45.008491 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4b8975c68d64c2d804fcc852384afde" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:45.008704 26643 raft_consensus.cc:3037] T d4b8975c68d64c2d804fcc852384afde P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:45.009109 26235 catalog_manager.cc:5484] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 1 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:45.009471 26772 raft_consensus.cc:2454] T d4b8975c68d64c2d804fcc852384afde P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 1.
I20240502 01:54:45.009915 26447 leader_election.cc:302] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 4bc1541b47df446297a369de05163012; no voters:
I20240502 01:54:45.010203 27825 raft_consensus.cc:2781] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:45.011241 27825 raft_consensus.cc:686] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012 [term 1 LEADER]: Becoming Leader. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:45.011323 26643 raft_consensus.cc:2454] T d4b8975c68d64c2d804fcc852384afde P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 1.
I20240502 01:54:45.011535 27825 consensus_queue.cc:237] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:45.014473 26235 catalog_manager.cc:5484] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bc1541b47df446297a369de05163012 (127.11.42.66). New cstate: current_term: 1 leader_uuid: "4bc1541b47df446297a369de05163012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:45.158758 27825 raft_consensus.cc:482] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:45.158936 27825 raft_consensus.cc:504] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:45.159528 27825 leader_election.cc:288] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:45.159930 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ba7c4fe9614ab3abb3645c1da3fcd3" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9d825f803c40e089b64e319beccf6e" is_pre_election: true
I20240502 01:54:45.159981 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ba7c4fe9614ab3abb3645c1da3fcd3" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" is_pre_election: true
I20240502 01:54:45.160122 26379 raft_consensus.cc:2454] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 0.
I20240502 01:54:45.160207 26643 raft_consensus.cc:2454] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 0.
I20240502 01:54:45.160538 26446 leader_election.cc:302] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:45.160799 27825 raft_consensus.cc:2781] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:45.160892 27825 raft_consensus.cc:482] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:45.160987 27825 raft_consensus.cc:3037] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:45.162966 27825 raft_consensus.cc:504] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:45.163455 27825 leader_election.cc:288] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 1 election: Requested vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:45.163815 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ba7c4fe9614ab3abb3645c1da3fcd3" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:45.163852 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ba7c4fe9614ab3abb3645c1da3fcd3" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:45.164028 26643 raft_consensus.cc:3037] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:45.164067 26379 raft_consensus.cc:3037] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:45.165899 26643 raft_consensus.cc:2454] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 1.
I20240502 01:54:45.166005 26379 raft_consensus.cc:2454] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 1.
I20240502 01:54:45.166270 26446 leader_election.cc:302] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:45.166563 27825 raft_consensus.cc:2781] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:45.166713 27825 raft_consensus.cc:686] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 4bc1541b47df446297a369de05163012 [term 1 LEADER]: Becoming Leader. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:45.166960 27825 consensus_queue.cc:237] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 4bc1541b47df446297a369de05163012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:45.169798 26235 catalog_manager.cc:5484] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 4bc1541b47df446297a369de05163012 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bc1541b47df446297a369de05163012 (127.11.42.66). New cstate: current_term: 1 leader_uuid: "4bc1541b47df446297a369de05163012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:45.446509 28067 consensus_queue.cc:1035] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:45.460574 28067 consensus_queue.cc:1035] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:45.480460 27825 consensus_queue.cc:1035] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:45.488063 28076 consensus_queue.cc:1035] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:54:45.540439 28079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:45.540710 28079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:45.547940 28079 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240502 01:54:45.637483 28076 consensus_queue.cc:1035] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 4bc1541b47df446297a369de05163012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:45.642530 28076 consensus_queue.cc:1035] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 4bc1541b47df446297a369de05163012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 259.445 ms
time per row: 0.129723 ms
I20240502 01:54:45.950770 26234 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36868:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:45.959583 26752 tablet_service.cc:1467] Processing CreateTablet for tablet 4254942d7ddc4c408114ae8d4b932a1a (DEFAULT_TABLE table=rebalance_test_table_9 [id=3ac92dc310154c13ad74a501ba844473]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:45.959772 26622 tablet_service.cc:1467] Processing CreateTablet for tablet 4254942d7ddc4c408114ae8d4b932a1a (DEFAULT_TABLE table=rebalance_test_table_9 [id=3ac92dc310154c13ad74a501ba844473]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:45.959987 26359 tablet_service.cc:1467] Processing CreateTablet for tablet 4254942d7ddc4c408114ae8d4b932a1a (DEFAULT_TABLE table=rebalance_test_table_9 [id=3ac92dc310154c13ad74a501ba844473]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:45.960107 26752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4254942d7ddc4c408114ae8d4b932a1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:45.960204 26622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4254942d7ddc4c408114ae8d4b932a1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:45.960412 26359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4254942d7ddc4c408114ae8d4b932a1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:45.966032 26754 tablet_service.cc:1467] Processing CreateTablet for tablet c56cb13e76384d5f9a0e45be2f65c085 (DEFAULT_TABLE table=rebalance_test_table_9 [id=3ac92dc310154c13ad74a501ba844473]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:45.966074 26489 tablet_service.cc:1467] Processing CreateTablet for tablet cf09a605f32845a3aaad81d79063070b (DEFAULT_TABLE table=rebalance_test_table_9 [id=3ac92dc310154c13ad74a501ba844473]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:45.966437 26754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c56cb13e76384d5f9a0e45be2f65c085. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:45.966451 26489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf09a605f32845a3aaad81d79063070b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:45.967286 26358 tablet_service.cc:1467] Processing CreateTablet for tablet c56cb13e76384d5f9a0e45be2f65c085 (DEFAULT_TABLE table=rebalance_test_table_9 [id=3ac92dc310154c13ad74a501ba844473]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:45.967658 26358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c56cb13e76384d5f9a0e45be2f65c085. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:45.973389 26357 tablet_service.cc:1467] Processing CreateTablet for tablet cf09a605f32845a3aaad81d79063070b (DEFAULT_TABLE table=rebalance_test_table_9 [id=3ac92dc310154c13ad74a501ba844473]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:45.973801 26357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf09a605f32845a3aaad81d79063070b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:45.974236 28146 tablet_bootstrap.cc:492] T 4254942d7ddc4c408114ae8d4b932a1a P 5e9d825f803c40e089b64e319beccf6e: Bootstrap starting.
I20240502 01:54:45.976217 28146 tablet_bootstrap.cc:654] T 4254942d7ddc4c408114ae8d4b932a1a P 5e9d825f803c40e089b64e319beccf6e: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:45.977761 26623 tablet_service.cc:1467] Processing CreateTablet for tablet c56cb13e76384d5f9a0e45be2f65c085 (DEFAULT_TABLE table=rebalance_test_table_9 [id=3ac92dc310154c13ad74a501ba844473]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:45.977855 26621 tablet_service.cc:1467] Processing CreateTablet for tablet cf09a605f32845a3aaad81d79063070b (DEFAULT_TABLE table=rebalance_test_table_9 [id=3ac92dc310154c13ad74a501ba844473]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:45.978153 26623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c56cb13e76384d5f9a0e45be2f65c085. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:45.978382 26621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf09a605f32845a3aaad81d79063070b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:45.982900 28145 tablet_bootstrap.cc:492] T 4254942d7ddc4c408114ae8d4b932a1a P 83c50029f7a94cfca6d1ee68e0728aa2: Bootstrap starting.
I20240502 01:54:45.985478 28145 tablet_bootstrap.cc:654] T 4254942d7ddc4c408114ae8d4b932a1a P 83c50029f7a94cfca6d1ee68e0728aa2: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:45.991479 28147 tablet_bootstrap.cc:492] T cf09a605f32845a3aaad81d79063070b P 4bc1541b47df446297a369de05163012: Bootstrap starting.
I20240502 01:54:45.993702 28145 tablet_bootstrap.cc:492] T 4254942d7ddc4c408114ae8d4b932a1a P 83c50029f7a94cfca6d1ee68e0728aa2: No bootstrap required, opened a new log
I20240502 01:54:45.993755 28146 tablet_bootstrap.cc:492] T 4254942d7ddc4c408114ae8d4b932a1a P 5e9d825f803c40e089b64e319beccf6e: No bootstrap required, opened a new log
I20240502 01:54:45.993897 28145 ts_tablet_manager.cc:1367] T 4254942d7ddc4c408114ae8d4b932a1a P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240502 01:54:45.993911 28146 ts_tablet_manager.cc:1367] T 4254942d7ddc4c408114ae8d4b932a1a P 5e9d825f803c40e089b64e319beccf6e: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.009s
I20240502 01:54:45.994226 28147 tablet_bootstrap.cc:654] T cf09a605f32845a3aaad81d79063070b P 4bc1541b47df446297a369de05163012: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:45.994653 28145 raft_consensus.cc:348] T 4254942d7ddc4c408114ae8d4b932a1a P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:45.994812 28145 raft_consensus.cc:374] T 4254942d7ddc4c408114ae8d4b932a1a P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:45.994755 28146 raft_consensus.cc:348] T 4254942d7ddc4c408114ae8d4b932a1a P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:45.994921 28145 raft_consensus.cc:729] T 4254942d7ddc4c408114ae8d4b932a1a P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Initialized, Role: FOLLOWER
I20240502 01:54:45.994956 28146 raft_consensus.cc:374] T 4254942d7ddc4c408114ae8d4b932a1a P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:45.995108 28146 raft_consensus.cc:729] T 4254942d7ddc4c408114ae8d4b932a1a P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Initialized, Role: FOLLOWER
I20240502 01:54:45.995158 28145 consensus_queue.cc:260] T 4254942d7ddc4c408114ae8d4b932a1a P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:45.995510 28146 consensus_queue.cc:260] T 4254942d7ddc4c408114ae8d4b932a1a P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:45.995730 28145 ts_tablet_manager.cc:1398] T 4254942d7ddc4c408114ae8d4b932a1a P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:54:45.996156 28145 tablet_bootstrap.cc:492] T cf09a605f32845a3aaad81d79063070b P 83c50029f7a94cfca6d1ee68e0728aa2: Bootstrap starting.
I20240502 01:54:45.996172 28146 ts_tablet_manager.cc:1398] T 4254942d7ddc4c408114ae8d4b932a1a P 5e9d825f803c40e089b64e319beccf6e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:45.997659 28145 tablet_bootstrap.cc:654] T cf09a605f32845a3aaad81d79063070b P 83c50029f7a94cfca6d1ee68e0728aa2: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:45.997984 28147 tablet_bootstrap.cc:492] T cf09a605f32845a3aaad81d79063070b P 4bc1541b47df446297a369de05163012: No bootstrap required, opened a new log
I20240502 01:54:45.998606 28147 ts_tablet_manager.cc:1367] T cf09a605f32845a3aaad81d79063070b P 4bc1541b47df446297a369de05163012: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 01:54:45.999531 28145 tablet_bootstrap.cc:492] T cf09a605f32845a3aaad81d79063070b P 83c50029f7a94cfca6d1ee68e0728aa2: No bootstrap required, opened a new log
I20240502 01:54:45.999734 28145 ts_tablet_manager.cc:1367] T cf09a605f32845a3aaad81d79063070b P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 01:54:46.000326 28146 tablet_bootstrap.cc:492] T c56cb13e76384d5f9a0e45be2f65c085 P 5e9d825f803c40e089b64e319beccf6e: Bootstrap starting.
I20240502 01:54:46.000989 28144 tablet_bootstrap.cc:492] T 4254942d7ddc4c408114ae8d4b932a1a P 401942254c4c43a1ba02c4ae68d17320: Bootstrap starting.
I20240502 01:54:46.000969 28145 raft_consensus.cc:348] T cf09a605f32845a3aaad81d79063070b P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:46.001250 28145 raft_consensus.cc:374] T cf09a605f32845a3aaad81d79063070b P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:46.001595 28147 raft_consensus.cc:348] T cf09a605f32845a3aaad81d79063070b P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:46.001379 28145 raft_consensus.cc:729] T cf09a605f32845a3aaad81d79063070b P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Initialized, Role: FOLLOWER
I20240502 01:54:46.002034 28147 raft_consensus.cc:374] T cf09a605f32845a3aaad81d79063070b P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:46.002002 28145 consensus_queue.cc:260] T cf09a605f32845a3aaad81d79063070b P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:46.002131 28147 raft_consensus.cc:729] T cf09a605f32845a3aaad81d79063070b P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bc1541b47df446297a369de05163012, State: Initialized, Role: FOLLOWER
I20240502 01:54:46.002365 28147 consensus_queue.cc:260] T cf09a605f32845a3aaad81d79063070b P 4bc1541b47df446297a369de05163012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:46.002808 28145 ts_tablet_manager.cc:1398] T cf09a605f32845a3aaad81d79063070b P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:54:46.002861 28147 ts_tablet_manager.cc:1398] T cf09a605f32845a3aaad81d79063070b P 4bc1541b47df446297a369de05163012: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:54:46.003734 28145 tablet_bootstrap.cc:492] T c56cb13e76384d5f9a0e45be2f65c085 P 83c50029f7a94cfca6d1ee68e0728aa2: Bootstrap starting.
I20240502 01:54:46.004356 28146 tablet_bootstrap.cc:654] T c56cb13e76384d5f9a0e45be2f65c085 P 5e9d825f803c40e089b64e319beccf6e: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:46.005084 28144 tablet_bootstrap.cc:654] T 4254942d7ddc4c408114ae8d4b932a1a P 401942254c4c43a1ba02c4ae68d17320: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:46.006464 28145 tablet_bootstrap.cc:654] T c56cb13e76384d5f9a0e45be2f65c085 P 83c50029f7a94cfca6d1ee68e0728aa2: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:46.012465 28145 tablet_bootstrap.cc:492] T c56cb13e76384d5f9a0e45be2f65c085 P 83c50029f7a94cfca6d1ee68e0728aa2: No bootstrap required, opened a new log
I20240502 01:54:46.012662 28145 ts_tablet_manager.cc:1367] T c56cb13e76384d5f9a0e45be2f65c085 P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20240502 01:54:46.013499 28145 raft_consensus.cc:348] T c56cb13e76384d5f9a0e45be2f65c085 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:46.013768 28145 raft_consensus.cc:374] T c56cb13e76384d5f9a0e45be2f65c085 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:46.013904 28145 raft_consensus.cc:729] T c56cb13e76384d5f9a0e45be2f65c085 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Initialized, Role: FOLLOWER
I20240502 01:54:46.014041 28144 tablet_bootstrap.cc:492] T 4254942d7ddc4c408114ae8d4b932a1a P 401942254c4c43a1ba02c4ae68d17320: No bootstrap required, opened a new log
I20240502 01:54:46.014227 28144 ts_tablet_manager.cc:1367] T 4254942d7ddc4c408114ae8d4b932a1a P 401942254c4c43a1ba02c4ae68d17320: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.002s
I20240502 01:54:46.014180 28145 consensus_queue.cc:260] T c56cb13e76384d5f9a0e45be2f65c085 P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:46.014894 28144 raft_consensus.cc:348] T 4254942d7ddc4c408114ae8d4b932a1a P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:46.015049 28144 raft_consensus.cc:374] T 4254942d7ddc4c408114ae8d4b932a1a P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:46.015164 28144 raft_consensus.cc:729] T 4254942d7ddc4c408114ae8d4b932a1a P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Initialized, Role: FOLLOWER
I20240502 01:54:46.015488 28145 ts_tablet_manager.cc:1398] T c56cb13e76384d5f9a0e45be2f65c085 P 83c50029f7a94cfca6d1ee68e0728aa2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:46.015430 28144 consensus_queue.cc:260] T 4254942d7ddc4c408114ae8d4b932a1a P 401942254c4c43a1ba02c4ae68d17320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:46.016258 28144 ts_tablet_manager.cc:1398] T 4254942d7ddc4c408114ae8d4b932a1a P 401942254c4c43a1ba02c4ae68d17320: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:54:46.016537 28144 tablet_bootstrap.cc:492] T c56cb13e76384d5f9a0e45be2f65c085 P 401942254c4c43a1ba02c4ae68d17320: Bootstrap starting.
I20240502 01:54:46.017103 28146 tablet_bootstrap.cc:492] T c56cb13e76384d5f9a0e45be2f65c085 P 5e9d825f803c40e089b64e319beccf6e: No bootstrap required, opened a new log
I20240502 01:54:46.017319 28146 ts_tablet_manager.cc:1367] T c56cb13e76384d5f9a0e45be2f65c085 P 5e9d825f803c40e089b64e319beccf6e: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.007s
I20240502 01:54:46.018134 28144 tablet_bootstrap.cc:654] T c56cb13e76384d5f9a0e45be2f65c085 P 401942254c4c43a1ba02c4ae68d17320: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:46.018119 28146 raft_consensus.cc:348] T c56cb13e76384d5f9a0e45be2f65c085 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:46.018412 28146 raft_consensus.cc:374] T c56cb13e76384d5f9a0e45be2f65c085 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:46.018538 28146 raft_consensus.cc:729] T c56cb13e76384d5f9a0e45be2f65c085 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Initialized, Role: FOLLOWER
I20240502 01:54:46.019735 28146 consensus_queue.cc:260] T c56cb13e76384d5f9a0e45be2f65c085 P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:46.020216 28144 tablet_bootstrap.cc:492] T c56cb13e76384d5f9a0e45be2f65c085 P 401942254c4c43a1ba02c4ae68d17320: No bootstrap required, opened a new log
I20240502 01:54:46.020328 28146 ts_tablet_manager.cc:1398] T c56cb13e76384d5f9a0e45be2f65c085 P 5e9d825f803c40e089b64e319beccf6e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:54:46.020383 28144 ts_tablet_manager.cc:1367] T c56cb13e76384d5f9a0e45be2f65c085 P 401942254c4c43a1ba02c4ae68d17320: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:54:46.020633 28146 tablet_bootstrap.cc:492] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e: Bootstrap starting.
I20240502 01:54:46.021190 28144 raft_consensus.cc:348] T c56cb13e76384d5f9a0e45be2f65c085 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:46.021414 28144 raft_consensus.cc:374] T c56cb13e76384d5f9a0e45be2f65c085 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:46.021503 28144 raft_consensus.cc:729] T c56cb13e76384d5f9a0e45be2f65c085 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Initialized, Role: FOLLOWER
I20240502 01:54:46.021744 28144 consensus_queue.cc:260] T c56cb13e76384d5f9a0e45be2f65c085 P 401942254c4c43a1ba02c4ae68d17320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:46.022259 28146 tablet_bootstrap.cc:654] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:46.022250 28144 ts_tablet_manager.cc:1398] T c56cb13e76384d5f9a0e45be2f65c085 P 401942254c4c43a1ba02c4ae68d17320: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:54:46.024909 28146 tablet_bootstrap.cc:492] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e: No bootstrap required, opened a new log
I20240502 01:54:46.025060 28146 ts_tablet_manager.cc:1367] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 01:54:46.025652 28146 raft_consensus.cc:348] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:46.025799 28146 raft_consensus.cc:374] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:46.025879 28146 raft_consensus.cc:729] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Initialized, Role: FOLLOWER
I20240502 01:54:46.026060 28146 consensus_queue.cc:260] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:46.026538 28146 ts_tablet_manager.cc:1398] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:54:46.033004 28028 raft_consensus.cc:482] T 4254942d7ddc4c408114ae8d4b932a1a P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:46.033159 28028 raft_consensus.cc:504] T 4254942d7ddc4c408114ae8d4b932a1a P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:46.033720 28028 leader_election.cc:288] T 4254942d7ddc4c408114ae8d4b932a1a P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959), 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085)
I20240502 01:54:46.034106 26772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4254942d7ddc4c408114ae8d4b932a1a" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "401942254c4c43a1ba02c4ae68d17320" is_pre_election: true
I20240502 01:54:46.034171 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4254942d7ddc4c408114ae8d4b932a1a" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9d825f803c40e089b64e319beccf6e" is_pre_election: true
I20240502 01:54:46.034350 26378 raft_consensus.cc:2454] T 4254942d7ddc4c408114ae8d4b932a1a P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 0.
I20240502 01:54:46.034363 26772 raft_consensus.cc:2454] T 4254942d7ddc4c408114ae8d4b932a1a P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 0.
I20240502 01:54:46.034735 26577 leader_election.cc:302] T 4254942d7ddc4c408114ae8d4b932a1a P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e9d825f803c40e089b64e319beccf6e, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:46.034973 28028 raft_consensus.cc:2781] T 4254942d7ddc4c408114ae8d4b932a1a P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:46.035112 28028 raft_consensus.cc:482] T 4254942d7ddc4c408114ae8d4b932a1a P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:46.035203 28028 raft_consensus.cc:3037] T 4254942d7ddc4c408114ae8d4b932a1a P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:46.037092 28028 raft_consensus.cc:504] T 4254942d7ddc4c408114ae8d4b932a1a P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:46.037657 28028 leader_election.cc:288] T 4254942d7ddc4c408114ae8d4b932a1a P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 1 election: Requested vote from peers 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959), 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085)
I20240502 01:54:46.038060 26772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4254942d7ddc4c408114ae8d4b932a1a" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:46.038103 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4254942d7ddc4c408114ae8d4b932a1a" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:46.038235 26772 raft_consensus.cc:3037] T 4254942d7ddc4c408114ae8d4b932a1a P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:46.038259 26378 raft_consensus.cc:3037] T 4254942d7ddc4c408114ae8d4b932a1a P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:46.040073 26772 raft_consensus.cc:2454] T 4254942d7ddc4c408114ae8d4b932a1a P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 1.
I20240502 01:54:46.040091 26378 raft_consensus.cc:2454] T 4254942d7ddc4c408114ae8d4b932a1a P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 1.
I20240502 01:54:46.040421 26578 leader_election.cc:302] T 4254942d7ddc4c408114ae8d4b932a1a P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:46.040678 28028 raft_consensus.cc:2781] T 4254942d7ddc4c408114ae8d4b932a1a P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:46.040794 28028 raft_consensus.cc:686] T 4254942d7ddc4c408114ae8d4b932a1a P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 LEADER]: Becoming Leader. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:46.041085 28028 consensus_queue.cc:237] T 4254942d7ddc4c408114ae8d4b932a1a P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:46.043876 26228 catalog_manager.cc:5484] T 4254942d7ddc4c408114ae8d4b932a1a P 83c50029f7a94cfca6d1ee68e0728aa2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67). New cstate: current_term: 1 leader_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:46.234890 28067 raft_consensus.cc:482] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:46.235065 28067 raft_consensus.cc:504] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:46.235664 28067 leader_election.cc:288] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:46.236043 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf09a605f32845a3aaad81d79063070b" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" is_pre_election: true
I20240502 01:54:46.236197 26510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf09a605f32845a3aaad81d79063070b" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc1541b47df446297a369de05163012" is_pre_election: true
I20240502 01:54:46.236276 26643 raft_consensus.cc:2454] T cf09a605f32845a3aaad81d79063070b P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 0.
I20240502 01:54:46.236440 26510 raft_consensus.cc:2454] T cf09a605f32845a3aaad81d79063070b P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 0.
I20240502 01:54:46.236673 26313 leader_election.cc:302] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e9d825f803c40e089b64e319beccf6e, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:46.236968 28067 raft_consensus.cc:2781] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:46.237092 28067 raft_consensus.cc:482] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:46.237201 28067 raft_consensus.cc:3037] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:46.239387 28067 raft_consensus.cc:504] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:46.240069 28067 leader_election.cc:288] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 1 election: Requested vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:46.240561 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf09a605f32845a3aaad81d79063070b" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:46.240767 26643 raft_consensus.cc:3037] T cf09a605f32845a3aaad81d79063070b P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:46.240723 26510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf09a605f32845a3aaad81d79063070b" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:46.240901 26510 raft_consensus.cc:3037] T cf09a605f32845a3aaad81d79063070b P 4bc1541b47df446297a369de05163012 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:46.243319 26510 raft_consensus.cc:2454] T cf09a605f32845a3aaad81d79063070b P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 1.
I20240502 01:54:46.243465 26643 raft_consensus.cc:2454] T cf09a605f32845a3aaad81d79063070b P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 1.
I20240502 01:54:46.243681 26314 leader_election.cc:302] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:46.243937 28067 raft_consensus.cc:2781] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:46.244092 28067 raft_consensus.cc:686] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [term 1 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:46.244356 28067 consensus_queue.cc:237] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:46.246948 28028 raft_consensus.cc:482] T c56cb13e76384d5f9a0e45be2f65c085 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:46.247109 28028 raft_consensus.cc:504] T c56cb13e76384d5f9a0e45be2f65c085 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:46.247294 26228 catalog_manager.cc:5484] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 1 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:46.247752 28028 leader_election.cc:288] T c56cb13e76384d5f9a0e45be2f65c085 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:46.248258 26772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c56cb13e76384d5f9a0e45be2f65c085" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "401942254c4c43a1ba02c4ae68d17320" is_pre_election: true
I20240502 01:54:46.248571 26772 raft_consensus.cc:2454] T c56cb13e76384d5f9a0e45be2f65c085 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 0.
I20240502 01:54:46.248550 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c56cb13e76384d5f9a0e45be2f65c085" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9d825f803c40e089b64e319beccf6e" is_pre_election: true
I20240502 01:54:46.248864 26378 raft_consensus.cc:2454] T c56cb13e76384d5f9a0e45be2f65c085 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 0.
I20240502 01:54:46.249505 26577 leader_election.cc:302] T c56cb13e76384d5f9a0e45be2f65c085 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e9d825f803c40e089b64e319beccf6e, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:46.249794 28028 raft_consensus.cc:2781] T c56cb13e76384d5f9a0e45be2f65c085 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:46.249898 28028 raft_consensus.cc:482] T c56cb13e76384d5f9a0e45be2f65c085 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:46.250005 28028 raft_consensus.cc:3037] T c56cb13e76384d5f9a0e45be2f65c085 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:46.251698 28028 raft_consensus.cc:504] T c56cb13e76384d5f9a0e45be2f65c085 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:46.252225 28028 leader_election.cc:288] T c56cb13e76384d5f9a0e45be2f65c085 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 1 election: Requested vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:46.252797 26772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c56cb13e76384d5f9a0e45be2f65c085" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:46.252930 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c56cb13e76384d5f9a0e45be2f65c085" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:46.253038 26772 raft_consensus.cc:3037] T c56cb13e76384d5f9a0e45be2f65c085 P 401942254c4c43a1ba02c4ae68d17320 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:46.253369 26378 raft_consensus.cc:3037] T c56cb13e76384d5f9a0e45be2f65c085 P 5e9d825f803c40e089b64e319beccf6e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:46.255365 26378 raft_consensus.cc:2454] T c56cb13e76384d5f9a0e45be2f65c085 P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 1.
I20240502 01:54:46.255715 26577 leader_election.cc:302] T c56cb13e76384d5f9a0e45be2f65c085 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e9d825f803c40e089b64e319beccf6e, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:46.255833 26772 raft_consensus.cc:2454] T c56cb13e76384d5f9a0e45be2f65c085 P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 1.
I20240502 01:54:46.255985 28028 raft_consensus.cc:2781] T c56cb13e76384d5f9a0e45be2f65c085 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:46.256161 28028 raft_consensus.cc:686] T c56cb13e76384d5f9a0e45be2f65c085 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 LEADER]: Becoming Leader. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:46.256417 28028 consensus_queue.cc:237] T c56cb13e76384d5f9a0e45be2f65c085 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:46.259721 26228 catalog_manager.cc:5484] T c56cb13e76384d5f9a0e45be2f65c085 P 83c50029f7a94cfca6d1ee68e0728aa2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67). New cstate: current_term: 1 leader_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:46.448271 28028 consensus_queue.cc:1035] T 4254942d7ddc4c408114ae8d4b932a1a P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:46.453339 28028 consensus_queue.cc:1035] T 4254942d7ddc4c408114ae8d4b932a1a P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:54:46.617517 28161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:46.617833 28161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:46.625093 28161 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:54:46.639973 28115 consensus_queue.cc:1035] T c56cb13e76384d5f9a0e45be2f65c085 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:46.649432 28028 consensus_queue.cc:1035] T c56cb13e76384d5f9a0e45be2f65c085 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_9'
I20240502 01:54:46.672729 28067 consensus_queue.cc:1035] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:46.678658 28083 consensus_queue.cc:1035] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 223.216 ms
time per row: 0.111608 ms
I20240502 01:54:46.997187 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.69:33779
--local_ip_for_outbound_sockets=127.11.42.69
--tserver_master_addrs=127.11.42.126:36275
--webserver_port=38851
--webserver_interface=127.11.42.69
--builtin_ntp_servers=127.11.42.84:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 01:54:47.325241 28223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:47.325500 28223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:47.325621 28223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:47.333432 28223 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:47.333560 28223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:47.333703 28223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.69
I20240502 01:54:47.342324 28223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.69:33779
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.11.42.69
--webserver_port=38851
--tserver_master_addrs=127.11.42.126:36275
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.69
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:47.343495 28223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:47.345139 28223 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:47.357316 28229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:47.357757 28231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:47.358086 28228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:47.358606 28223 server_base.cc:1034] running on GCE node
I20240502 01:54:47.359158 28223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:47.360066 28223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:47.361269 28223 hybrid_clock.cc:648] HybridClock initialized: now 1714614887361211 us; error 57 us; skew 500 ppm
I20240502 01:54:47.361572 28223 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:47.363893 28223 webserver.cc:460] Webserver started at http://127.11.42.69:38851/ using document root <none> and password file <none>
I20240502 01:54:47.364562 28223 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:47.364678 28223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:47.369608 28223 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:54:47.372186 28237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:47.373431 28223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:54:47.373565 28223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "807eabb8a90f448390ae330c2c07330c"
format_stamp: "Formatted at 2024-05-02 01:54:28 on dist-test-slave-xvk5"
I20240502 01:54:47.374330 28223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:47.391139 28223 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:54:47.391899 28223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:47.392158 28223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:47.393183 28223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:47.394685 28223 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:47.394774 28223 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:47.394894 28223 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:47.394945 28223 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:47.424117 28223 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.69:33779
I20240502 01:54:47.424197 28350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.69:33779 every 8 connection(s)
I20240502 01:54:47.425570 28223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:54:47.427491 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 28223
I20240502 01:54:47.428077 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.70:39005
--local_ip_for_outbound_sockets=127.11.42.70
--tserver_master_addrs=127.11.42.126:36275
--webserver_port=35285
--webserver_interface=127.11.42.70
--builtin_ntp_servers=127.11.42.84:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:54:47.439453 28351 heartbeater.cc:347] Connected to a master server at 127.11.42.126:36275
I20240502 01:54:47.439711 28351 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:47.440479 28351 heartbeater.cc:511] Master 127.11.42.126:36275 requested a full tablet report, sending...
I20240502 01:54:47.442204 26234 ts_manager.cc:194] Re-registered known tserver with Master: 807eabb8a90f448390ae330c2c07330c (127.11.42.69:33779)
I20240502 01:54:47.443082 26234 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.69:47521
W20240502 01:54:47.770170 28355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:47.770422 28355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:47.770530 28355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:47.778630 28355 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:47.778771 28355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:47.778931 28355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.70
I20240502 01:54:47.787931 28355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.70:39005
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.11.42.70
--webserver_port=35285
--tserver_master_addrs=127.11.42.126:36275
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.70
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:47.789180 28355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:47.790880 28355 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:47.802520 28360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:47.802959 28361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:47.803340 28363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:47.803877 28355 server_base.cc:1034] running on GCE node
I20240502 01:54:47.804406 28355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:47.805241 28355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:47.806434 28355 hybrid_clock.cc:648] HybridClock initialized: now 1714614887806390 us; error 39 us; skew 500 ppm
I20240502 01:54:47.806718 28355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:47.809085 28355 webserver.cc:460] Webserver started at http://127.11.42.70:35285/ using document root <none> and password file <none>
I20240502 01:54:47.809768 28355 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:47.809918 28355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:47.814888 28355 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 01:54:47.817561 28369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:47.818725 28355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:54:47.818917 28355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "f62b051e38d3421087d97b594674043f"
format_stamp: "Formatted at 2024-05-02 01:54:29 on dist-test-slave-xvk5"
I20240502 01:54:47.819720 28355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:47.878715 28355 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:54:47.879540 28355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:47.879784 28355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:47.880693 28355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:47.882265 28355 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:47.882354 28355 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:47.882452 28355 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:47.882493 28355 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:47.915267 28355 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.70:39005
I20240502 01:54:47.915321 28481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.70:39005 every 8 connection(s)
I20240502 01:54:47.916775 28355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 01:54:47.921270 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 28355
I20240502 01:54:47.921828 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.71:41219
--local_ip_for_outbound_sockets=127.11.42.71
--tserver_master_addrs=127.11.42.126:36275
--webserver_port=40485
--webserver_interface=127.11.42.71
--builtin_ntp_servers=127.11.42.84:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:54:47.929225 28482 heartbeater.cc:347] Connected to a master server at 127.11.42.126:36275
I20240502 01:54:47.929522 28482 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:47.930416 28482 heartbeater.cc:511] Master 127.11.42.126:36275 requested a full tablet report, sending...
I20240502 01:54:47.931847 26234 ts_manager.cc:194] Re-registered known tserver with Master: f62b051e38d3421087d97b594674043f (127.11.42.70:39005)
I20240502 01:54:47.932648 26234 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.70:58043
W20240502 01:54:48.260926 28485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:48.261207 28485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:48.261338 28485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:48.268949 28485 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:48.269110 28485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:48.269294 28485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.71
I20240502 01:54:48.278721 28485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.71:41219
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.11.42.71
--webserver_port=40485
--tserver_master_addrs=127.11.42.126:36275
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.71
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:48.280270 28485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:48.282075 28485 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:48.293401 28491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:48.293831 28492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:48.294338 28494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:48.294859 28485 server_base.cc:1034] running on GCE node
I20240502 01:54:48.295516 28485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:48.296519 28485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:48.300063 28485 hybrid_clock.cc:648] HybridClock initialized: now 1714614888299958 us; error 95 us; skew 500 ppm
I20240502 01:54:48.300406 28485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:48.302632 28485 webserver.cc:460] Webserver started at http://127.11.42.71:40485/ using document root <none> and password file <none>
I20240502 01:54:48.303295 28485 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:48.303411 28485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:48.308326 28485 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:48.310930 28500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:48.312147 28485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:48.312278 28485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "f2aea971554443a7afbf01e7e79a025d"
format_stamp: "Formatted at 2024-05-02 01:54:29 on dist-test-slave-xvk5"
I20240502 01:54:48.313042 28485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:48.330739 28485 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:54:48.331442 28485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:48.331689 28485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:48.332515 28485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:48.334117 28485 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:48.334189 28485 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:48.334311 28485 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:48.334383 28485 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:48.365603 28485 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.71:41219
I20240502 01:54:48.365695 28612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.71:41219 every 8 connection(s)
I20240502 01:54:48.367105 28485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 01:54:48.373718 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 28485
I20240502 01:54:48.374287 11433 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
/tmp/dist-test-taskVgyxVB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.42.72:33965
--local_ip_for_outbound_sockets=127.11.42.72
--tserver_master_addrs=127.11.42.126:36275
--webserver_port=41651
--webserver_interface=127.11.42.72
--builtin_ntp_servers=127.11.42.84:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:54:48.380319 28613 heartbeater.cc:347] Connected to a master server at 127.11.42.126:36275
I20240502 01:54:48.380620 28613 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:48.381553 28613 heartbeater.cc:511] Master 127.11.42.126:36275 requested a full tablet report, sending...
I20240502 01:54:48.382946 26234 ts_manager.cc:194] Re-registered known tserver with Master: f2aea971554443a7afbf01e7e79a025d (127.11.42.71:41219)
I20240502 01:54:48.383783 26234 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.71:48513
I20240502 01:54:48.445377 28351 heartbeater.cc:503] Master 127.11.42.126:36275 was elected leader, sending a full tablet report...
W20240502 01:54:48.714267 28616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:48.714574 28616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:48.714720 28616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:48.723239 28616 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:48.723372 28616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:48.723521 28616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.42.72
I20240502 01:54:48.732369 28616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.42.84:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.42.72:33965
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.11.42.72
--webserver_port=41651
--tserver_master_addrs=127.11.42.126:36275
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.42.72
--log_dir=/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:48.733633 28616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:48.735257 28616 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:48.746829 28623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:48.747323 28626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:48.747746 28624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:48.748299 28616 server_base.cc:1034] running on GCE node
I20240502 01:54:48.748854 28616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:48.749778 28616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:48.750968 28616 hybrid_clock.cc:648] HybridClock initialized: now 1714614888750925 us; error 32 us; skew 500 ppm
I20240502 01:54:48.751274 28616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:48.753386 28616 webserver.cc:460] Webserver started at http://127.11.42.72:41651/ using document root <none> and password file <none>
I20240502 01:54:48.754068 28616 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:48.754217 28616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:48.759832 28616 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 01:54:48.762806 28632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:48.764124 28616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240502 01:54:48.764261 28616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "0bd788251a5d4622a1117f2000224a2a"
format_stamp: "Formatted at 2024-05-02 01:54:30 on dist-test-slave-xvk5"
I20240502 01:54:48.765033 28616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:48.795816 28616 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:54:48.796702 28616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:48.797044 28616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:48.798169 28616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:48.799778 28616 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:48.799855 28616 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:48.800015 28616 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:48.800113 28616 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:48.839344 28616 rpc_server.cc:307] RPC server started. Bound to: 127.11.42.72:33965
I20240502 01:54:48.839476 28744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.42.72:33965 every 8 connection(s)
I20240502 01:54:48.840728 28616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVgyxVB/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714614746691041-11433-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 01:54:48.847756 11433 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVgyxVB/build/asan/bin/kudu as pid 28616
I20240502 01:54:48.853610 28745 heartbeater.cc:347] Connected to a master server at 127.11.42.126:36275
I20240502 01:54:48.853870 28745 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:48.854817 28745 heartbeater.cc:511] Master 127.11.42.126:36275 requested a full tablet report, sending...
I20240502 01:54:48.856765 26234 ts_manager.cc:194] Re-registered known tserver with Master: 0bd788251a5d4622a1117f2000224a2a (127.11.42.72:33965)
I20240502 01:54:48.857825 26234 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.42.72:34115
I20240502 01:54:48.894969 26642 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cf09a605f32845a3aaad81d79063070b"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:48.895227 26642 raft_consensus.cc:482] T cf09a605f32845a3aaad81d79063070b P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:48.895339 26642 raft_consensus.cc:3037] T cf09a605f32845a3aaad81d79063070b P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:48.897343 26642 raft_consensus.cc:504] T cf09a605f32845a3aaad81d79063070b P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:48.898097 26642 leader_election.cc:288] T cf09a605f32845a3aaad81d79063070b P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 2 election: Requested vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:48.898682 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf09a605f32845a3aaad81d79063070b" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:48.898886 26379 raft_consensus.cc:3032] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:48.898973 26379 raft_consensus.cc:729] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:48.898957 26510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf09a605f32845a3aaad81d79063070b" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:48.899175 26510 raft_consensus.cc:3037] T cf09a605f32845a3aaad81d79063070b P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:48.899287 26379 consensus_queue.cc:260] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:48.899696 26379 raft_consensus.cc:3037] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:48.901686 26379 raft_consensus.cc:2454] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 2.
I20240502 01:54:48.901901 26510 raft_consensus.cc:2454] T cf09a605f32845a3aaad81d79063070b P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 2.
I20240502 01:54:48.902443 26642 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4254942d7ddc4c408114ae8d4b932a1a"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:48.902624 26642 raft_consensus.cc:465] T 4254942d7ddc4c408114ae8d4b932a1a P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 01:54:48.902721 26578 leader_election.cc:302] T cf09a605f32845a3aaad81d79063070b P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:48.903021 28617 raft_consensus.cc:2781] T cf09a605f32845a3aaad81d79063070b P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:48.903167 28617 raft_consensus.cc:686] T cf09a605f32845a3aaad81d79063070b P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 LEADER]: Becoming Leader. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:48.903532 26643 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d4b8975c68d64c2d804fcc852384afde"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:48.903501 28617 consensus_queue.cc:237] T cf09a605f32845a3aaad81d79063070b P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:48.903734 26643 raft_consensus.cc:482] T d4b8975c68d64c2d804fcc852384afde P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:48.903848 26643 raft_consensus.cc:3037] T d4b8975c68d64c2d804fcc852384afde P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:48.906674 26643 raft_consensus.cc:504] T d4b8975c68d64c2d804fcc852384afde P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:48.907362 26643 leader_election.cc:288] T d4b8975c68d64c2d804fcc852384afde P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 2 election: Requested vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:48.908337 26641 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ef474f44fe5b4a06b2c8dda12b1bc496"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:48.908520 26641 raft_consensus.cc:482] T ef474f44fe5b4a06b2c8dda12b1bc496 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:48.908646 26641 raft_consensus.cc:3037] T ef474f44fe5b4a06b2c8dda12b1bc496 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:48.911362 26641 raft_consensus.cc:504] T ef474f44fe5b4a06b2c8dda12b1bc496 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:48.912456 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef474f44fe5b4a06b2c8dda12b1bc496" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:48.912679 26379 raft_consensus.cc:3032] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:48.912756 26379 raft_consensus.cc:729] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:48.913033 26379 consensus_queue.cc:260] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:48.913411 26379 raft_consensus.cc:3037] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:48.914155 26510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4b8975c68d64c2d804fcc852384afde" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:48.914485 26510 raft_consensus.cc:3032] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:48.914579 26510 raft_consensus.cc:729] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:48.914994 26510 consensus_queue.cc:260] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:48.915007 26234 catalog_manager.cc:5484] T cf09a605f32845a3aaad81d79063070b P 83c50029f7a94cfca6d1ee68e0728aa2 reported cstate change: term changed from 1 to 2, leader changed from 5e9d825f803c40e089b64e319beccf6e (127.11.42.65) to 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67). New cstate: current_term: 2 leader_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:48.915527 26510 raft_consensus.cc:3037] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:48.915869 26379 raft_consensus.cc:2454] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 2.
I20240502 01:54:48.916807 26772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4b8975c68d64c2d804fcc852384afde" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:48.917068 26772 raft_consensus.cc:3037] T d4b8975c68d64c2d804fcc852384afde P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:48.918169 26510 raft_consensus.cc:2454] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 2.
I20240502 01:54:48.919395 26772 raft_consensus.cc:2454] T d4b8975c68d64c2d804fcc852384afde P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 2.
I20240502 01:54:48.919859 26578 leader_election.cc:302] T d4b8975c68d64c2d804fcc852384afde P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:48.920194 28755 raft_consensus.cc:2781] T d4b8975c68d64c2d804fcc852384afde P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:48.920451 28755 raft_consensus.cc:686] T d4b8975c68d64c2d804fcc852384afde P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 LEADER]: Becoming Leader. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:48.920751 28755 consensus_queue.cc:237] T d4b8975c68d64c2d804fcc852384afde P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:48.921545 26577 leader_election.cc:302] T ef474f44fe5b4a06b2c8dda12b1bc496 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e9d825f803c40e089b64e319beccf6e, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:48.921862 28028 raft_consensus.cc:2781] T ef474f44fe5b4a06b2c8dda12b1bc496 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:48.922012 28028 raft_consensus.cc:686] T ef474f44fe5b4a06b2c8dda12b1bc496 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 LEADER]: Becoming Leader. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:48.922241 28028 consensus_queue.cc:237] T ef474f44fe5b4a06b2c8dda12b1bc496 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:48.922456 26641 leader_election.cc:288] T ef474f44fe5b4a06b2c8dda12b1bc496 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 2 election: Requested vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:48.925174 26642 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "14a747ac14354fdab2a2993594d3398f"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:48.925643 26642 raft_consensus.cc:482] T 14a747ac14354fdab2a2993594d3398f P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:48.925969 26642 raft_consensus.cc:3037] T 14a747ac14354fdab2a2993594d3398f P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:48.926970 26772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef474f44fe5b4a06b2c8dda12b1bc496" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:48.927160 26772 raft_consensus.cc:3037] T ef474f44fe5b4a06b2c8dda12b1bc496 P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:48.929821 26772 raft_consensus.cc:2454] T ef474f44fe5b4a06b2c8dda12b1bc496 P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 2.
I20240502 01:54:48.931793 26642 raft_consensus.cc:504] T 14a747ac14354fdab2a2993594d3398f P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:48.935412 28482 heartbeater.cc:503] Master 127.11.42.126:36275 was elected leader, sending a full tablet report...
I20240502 01:54:48.937958 26642 leader_election.cc:288] T 14a747ac14354fdab2a2993594d3398f P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 2 election: Requested vote from peers 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:48.938247 26772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a747ac14354fdab2a2993594d3398f" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:48.938482 26772 raft_consensus.cc:3032] T 14a747ac14354fdab2a2993594d3398f P 401942254c4c43a1ba02c4ae68d17320 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:48.938577 26772 raft_consensus.cc:729] T 14a747ac14354fdab2a2993594d3398f P 401942254c4c43a1ba02c4ae68d17320 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Running, Role: LEADER
I20240502 01:54:48.938884 26772 consensus_queue.cc:260] T 14a747ac14354fdab2a2993594d3398f P 401942254c4c43a1ba02c4ae68d17320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:48.939430 26772 raft_consensus.cc:3037] T 14a747ac14354fdab2a2993594d3398f P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:48.939630 26643 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3d751cdd58eb4cef8039fd4c50855e99"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:48.939792 26643 raft_consensus.cc:465] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 01:54:48.940719 26643 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9ead21d62e564a37a8c3d5a452621d13"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:48.940901 26643 raft_consensus.cc:465] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 01:54:48.941833 26643 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1c47a74a6dc54d5eb0b47818d816b7fd"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:48.942005 26643 raft_consensus.cc:482] T 1c47a74a6dc54d5eb0b47818d816b7fd P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:48.942119 26643 raft_consensus.cc:3037] T 1c47a74a6dc54d5eb0b47818d816b7fd P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:48.942143 26772 raft_consensus.cc:2454] T 14a747ac14354fdab2a2993594d3398f P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 2.
I20240502 01:54:48.942698 26578 leader_election.cc:302] T 14a747ac14354fdab2a2993594d3398f P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:48.942981 28617 raft_consensus.cc:2781] T 14a747ac14354fdab2a2993594d3398f P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:48.942941 26510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a747ac14354fdab2a2993594d3398f" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:48.943126 26510 raft_consensus.cc:3037] T 14a747ac14354fdab2a2993594d3398f P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:48.944727 26643 raft_consensus.cc:504] T 1c47a74a6dc54d5eb0b47818d816b7fd P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:48.945569 26643 leader_election.cc:288] T 1c47a74a6dc54d5eb0b47818d816b7fd P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 2 election: Requested vote from peers 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:48.945966 26773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c47a74a6dc54d5eb0b47818d816b7fd" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:48.946153 26773 raft_consensus.cc:3032] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:48.946291 26773 raft_consensus.cc:729] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Running, Role: LEADER
I20240502 01:54:48.946559 26773 consensus_queue.cc:260] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:48.946696 26643 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "daef5905fc19473a8728ccd2627270af"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:48.946931 26773 raft_consensus.cc:3037] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:48.947058 26643 raft_consensus.cc:482] T daef5905fc19473a8728ccd2627270af P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:48.947221 26643 raft_consensus.cc:3037] T daef5905fc19473a8728ccd2627270af P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:48.949721 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c47a74a6dc54d5eb0b47818d816b7fd" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:48.949939 26511 raft_consensus.cc:3037] T 1c47a74a6dc54d5eb0b47818d816b7fd P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:48.950312 26510 raft_consensus.cc:2454] T 14a747ac14354fdab2a2993594d3398f P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 2.
I20240502 01:54:48.951532 26643 raft_consensus.cc:504] T daef5905fc19473a8728ccd2627270af P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:48.952081 26643 leader_election.cc:288] T daef5905fc19473a8728ccd2627270af P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 2 election: Requested vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:48.952447 26772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daef5905fc19473a8728ccd2627270af" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:48.952617 26772 raft_consensus.cc:3037] T daef5905fc19473a8728ccd2627270af P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:48.952859 26641 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c56cb13e76384d5f9a0e45be2f65c085"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:48.953043 26641 raft_consensus.cc:465] T c56cb13e76384d5f9a0e45be2f65c085 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 01:54:48.952939 26511 raft_consensus.cc:2454] T 1c47a74a6dc54d5eb0b47818d816b7fd P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 2.
I20240502 01:54:48.953593 26578 leader_election.cc:302] T 1c47a74a6dc54d5eb0b47818d816b7fd P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:48.953873 26641 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d9dd99b54b7c4d19a80c229bf99d7976"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:48.954046 26641 raft_consensus.cc:482] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:48.954075 26773 raft_consensus.cc:2454] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 2.
I20240502 01:54:48.954133 26641 raft_consensus.cc:3037] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:48.955895 26772 raft_consensus.cc:2454] T daef5905fc19473a8728ccd2627270af P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 2.
I20240502 01:54:48.956638 26641 raft_consensus.cc:504] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:48.957440 26641 leader_election.cc:288] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 2 election: Requested vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:48.958321 26641 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "23dcaac2d7d14adf820d826f05c1abc6"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:48.958534 26641 raft_consensus.cc:465] T 23dcaac2d7d14adf820d826f05c1abc6 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 01:54:48.959535 26641 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "beff61f07ecf4ffab508da651a686c8e"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:48.959746 26641 raft_consensus.cc:465] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 01:54:48.960769 26641 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b4ba7c4fe9614ab3abb3645c1da3fcd3"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:48.960992 26641 raft_consensus.cc:482] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:48.961184 26641 raft_consensus.cc:3037] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:48.964427 26641 raft_consensus.cc:504] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:48.968948 26772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9dd99b54b7c4d19a80c229bf99d7976" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:48.968981 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9dd99b54b7c4d19a80c229bf99d7976" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:48.969188 26511 raft_consensus.cc:3037] T d9dd99b54b7c4d19a80c229bf99d7976 P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:48.969233 26772 raft_consensus.cc:3032] T d9dd99b54b7c4d19a80c229bf99d7976 P 401942254c4c43a1ba02c4ae68d17320 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:48.969421 26772 raft_consensus.cc:729] T d9dd99b54b7c4d19a80c229bf99d7976 P 401942254c4c43a1ba02c4ae68d17320 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Running, Role: LEADER
I20240502 01:54:48.969745 26772 consensus_queue.cc:260] T d9dd99b54b7c4d19a80c229bf99d7976 P 401942254c4c43a1ba02c4ae68d17320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:48.970155 26772 raft_consensus.cc:3037] T d9dd99b54b7c4d19a80c229bf99d7976 P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:48.970500 26641 leader_election.cc:288] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 2 election: Requested vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:48.971753 26511 raft_consensus.cc:2454] T d9dd99b54b7c4d19a80c229bf99d7976 P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 2.
I20240502 01:54:48.965785 28762 raft_consensus.cc:2781] T 1c47a74a6dc54d5eb0b47818d816b7fd P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:48.972386 28762 raft_consensus.cc:686] T 1c47a74a6dc54d5eb0b47818d816b7fd P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 LEADER]: Becoming Leader. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:48.972695 26772 raft_consensus.cc:2454] T d9dd99b54b7c4d19a80c229bf99d7976 P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 2.
I20240502 01:54:48.972735 28762 consensus_queue.cc:237] T 1c47a74a6dc54d5eb0b47818d816b7fd P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:48.973920 26643 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d2287fdff0db43df90ff5580c0430b8c"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:48.974274 26643 raft_consensus.cc:482] T d2287fdff0db43df90ff5580c0430b8c P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:48.974728 26642 raft_consensus.cc:1226] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Rejecting Update request from peer 4bc1541b47df446297a369de05163012 for earlier term 1. Current term is 2. Ops: []
I20240502 01:54:48.974795 26643 raft_consensus.cc:3037] T d2287fdff0db43df90ff5580c0430b8c P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:48.975329 28162 consensus_queue.cc:1046] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 4bc1541b47df446297a369de05163012 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:48.975936 28617 raft_consensus.cc:686] T 14a747ac14354fdab2a2993594d3398f P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 LEADER]: Becoming Leader. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:48.976363 28617 consensus_queue.cc:237] T 14a747ac14354fdab2a2993594d3398f P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:48.974960 26578 leader_election.cc:302] T daef5905fc19473a8728ccd2627270af P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:48.977794 28762 raft_consensus.cc:2781] T daef5905fc19473a8728ccd2627270af P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:48.977800 26578 leader_election.cc:302] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:48.979974 28762 raft_consensus.cc:686] T daef5905fc19473a8728ccd2627270af P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 LEADER]: Becoming Leader. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:48.980623 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ba7c4fe9614ab3abb3645c1da3fcd3" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:48.980876 28762 consensus_queue.cc:237] T daef5905fc19473a8728ccd2627270af P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:48.980257 28760 raft_consensus.cc:2781] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:48.983219 28760 raft_consensus.cc:686] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 LEADER]: Becoming Leader. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:48.983551 28760 consensus_queue.cc:237] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:48.989001 26643 raft_consensus.cc:504] T d2287fdff0db43df90ff5580c0430b8c P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:48.990085 26643 leader_election.cc:288] T d2287fdff0db43df90ff5580c0430b8c P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 2 election: Requested vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:48.990473 26510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2287fdff0db43df90ff5580c0430b8c" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:48.990685 26510 raft_consensus.cc:3037] T d2287fdff0db43df90ff5580c0430b8c P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:48.992115 26641 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "53b7377c0d144e07a4501c2d626f6494"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:48.992292 26641 raft_consensus.cc:465] T 53b7377c0d144e07a4501c2d626f6494 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 01:54:48.993291 26640 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "99a4b7d022e1461a8233952ab47ef5d3"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:48.993492 26510 raft_consensus.cc:2454] T d2287fdff0db43df90ff5580c0430b8c P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 2.
I20240502 01:54:48.993552 26640 raft_consensus.cc:482] T 99a4b7d022e1461a8233952ab47ef5d3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:48.993713 26640 raft_consensus.cc:3037] T 99a4b7d022e1461a8233952ab47ef5d3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:48.995885 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ba7c4fe9614ab3abb3645c1da3fcd3" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:48.996098 26378 raft_consensus.cc:3037] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:48.996737 26640 raft_consensus.cc:504] T 99a4b7d022e1461a8233952ab47ef5d3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:48.997320 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daef5905fc19473a8728ccd2627270af" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:48.997542 26379 raft_consensus.cc:3032] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:48.997623 26379 raft_consensus.cc:729] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:48.997900 26379 consensus_queue.cc:260] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:48.998252 26379 raft_consensus.cc:3037] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:48.999163 26378 raft_consensus.cc:2454] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 2.
I20240502 01:54:49.000048 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2287fdff0db43df90ff5580c0430b8c" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:49.000247 26378 raft_consensus.cc:3032] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:49.000326 26378 raft_consensus.cc:729] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:49.000706 26378 consensus_queue.cc:260] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.001104 26378 raft_consensus.cc:3037] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.001175 26379 raft_consensus.cc:2454] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 2.
I20240502 01:54:49.001986 26578 leader_election.cc:302] T d2287fdff0db43df90ff5580c0430b8c P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:49.002223 26510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a4b7d022e1461a8233952ab47ef5d3" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:49.002413 26510 raft_consensus.cc:3037] T 99a4b7d022e1461a8233952ab47ef5d3 P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.002415 26640 leader_election.cc:288] T 99a4b7d022e1461a8233952ab47ef5d3 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 2 election: Requested vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085)
I20240502 01:54:49.003324 26640 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0ddd395eec1146bc8a20292d81be4868"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:49.003504 26640 raft_consensus.cc:482] T 0ddd395eec1146bc8a20292d81be4868 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.003590 26640 raft_consensus.cc:3037] T 0ddd395eec1146bc8a20292d81be4868 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.003793 26378 raft_consensus.cc:2454] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 2.
I20240502 01:54:49.004972 26510 raft_consensus.cc:2454] T 99a4b7d022e1461a8233952ab47ef5d3 P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 2.
I20240502 01:54:49.005376 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a4b7d022e1461a8233952ab47ef5d3" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:49.005579 26378 raft_consensus.cc:3032] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:49.005662 26378 raft_consensus.cc:729] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:49.005946 26378 consensus_queue.cc:260] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.006271 26378 raft_consensus.cc:3037] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.006381 26640 raft_consensus.cc:504] T 0ddd395eec1146bc8a20292d81be4868 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.006968 26640 leader_election.cc:288] T 0ddd395eec1146bc8a20292d81be4868 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 2 election: Requested vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:49.008123 26772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ddd395eec1146bc8a20292d81be4868" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:49.008358 26772 raft_consensus.cc:3032] T 0ddd395eec1146bc8a20292d81be4868 P 401942254c4c43a1ba02c4ae68d17320 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:49.008462 26772 raft_consensus.cc:729] T 0ddd395eec1146bc8a20292d81be4868 P 401942254c4c43a1ba02c4ae68d17320 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Running, Role: LEADER
I20240502 01:54:49.008744 26378 raft_consensus.cc:2454] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 2.
I20240502 01:54:49.008746 26772 consensus_queue.cc:260] T 0ddd395eec1146bc8a20292d81be4868 P 401942254c4c43a1ba02c4ae68d17320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.009138 26772 raft_consensus.cc:3037] T 0ddd395eec1146bc8a20292d81be4868 P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.009208 26577 leader_election.cc:302] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e9d825f803c40e089b64e319beccf6e, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:49.009922 26577 leader_election.cc:302] T 99a4b7d022e1461a8233952ab47ef5d3 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e9d825f803c40e089b64e319beccf6e, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:49.010689 28763 raft_consensus.cc:2781] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:49.010378 28762 raft_consensus.cc:2781] T d2287fdff0db43df90ff5580c0430b8c P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:49.011772 26772 raft_consensus.cc:2454] T 0ddd395eec1146bc8a20292d81be4868 P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 2.
I20240502 01:54:49.012454 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ddd395eec1146bc8a20292d81be4868" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:49.012668 26379 raft_consensus.cc:3037] T 0ddd395eec1146bc8a20292d81be4868 P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.012964 28028 raft_consensus.cc:2781] T 99a4b7d022e1461a8233952ab47ef5d3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:49.014881 28028 raft_consensus.cc:686] T 99a4b7d022e1461a8233952ab47ef5d3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 LEADER]: Becoming Leader. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:49.015223 26379 raft_consensus.cc:2454] T 0ddd395eec1146bc8a20292d81be4868 P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 2.
I20240502 01:54:49.015834 26578 leader_election.cc:302] T 0ddd395eec1146bc8a20292d81be4868 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:49.016778 26511 raft_consensus.cc:3032] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 4bc1541b47df446297a369de05163012 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:49.016881 26511 raft_consensus.cc:729] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 4bc1541b47df446297a369de05163012 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:49.016201 26643 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2fd06590221d414b9e896e02b1be6f3e"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:49.017217 26643 raft_consensus.cc:482] T 2fd06590221d414b9e896e02b1be6f3e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.017176 26511 consensus_queue.cc:260] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 4bc1541b47df446297a369de05163012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.017423 26643 raft_consensus.cc:3037] T 2fd06590221d414b9e896e02b1be6f3e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.017441 26511 raft_consensus.cc:3037] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.018064 28028 consensus_queue.cc:237] T 99a4b7d022e1461a8233952ab47ef5d3 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.018555 28761 raft_consensus.cc:2781] T 0ddd395eec1146bc8a20292d81be4868 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:49.019224 28761 raft_consensus.cc:686] T 0ddd395eec1146bc8a20292d81be4868 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 LEADER]: Becoming Leader. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:49.020110 26511 raft_consensus.cc:2454] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 2.
W20240502 01:54:49.021229 28767 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20240502 01:54:49.020026 28761 consensus_queue.cc:237] T 0ddd395eec1146bc8a20292d81be4868 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.023255 26643 raft_consensus.cc:504] T 2fd06590221d414b9e896e02b1be6f3e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.024906 26643 leader_election.cc:288] T 2fd06590221d414b9e896e02b1be6f3e P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 2 election: Requested vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:49.025183 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fd06590221d414b9e896e02b1be6f3e" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:49.025404 26511 raft_consensus.cc:3032] T 2fd06590221d414b9e896e02b1be6f3e P 4bc1541b47df446297a369de05163012 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:49.025487 26511 raft_consensus.cc:729] T 2fd06590221d414b9e896e02b1be6f3e P 4bc1541b47df446297a369de05163012 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:49.025733 26511 consensus_queue.cc:260] T 2fd06590221d414b9e896e02b1be6f3e P 4bc1541b47df446297a369de05163012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.026319 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fd06590221d414b9e896e02b1be6f3e" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:49.026517 26378 raft_consensus.cc:3037] T 2fd06590221d414b9e896e02b1be6f3e P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.027709 26511 raft_consensus.cc:3037] T 2fd06590221d414b9e896e02b1be6f3e P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.029536 26643 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "89107a545fcc48aead1e199da1f78311"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:49.029717 26643 raft_consensus.cc:465] T 89107a545fcc48aead1e199da1f78311 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 01:54:49.030545 26511 raft_consensus.cc:2454] T 2fd06590221d414b9e896e02b1be6f3e P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 2.
I20240502 01:54:49.030566 26640 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dc82c30359e84bddb4bcfe1b3b13ced7"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:49.030727 26640 raft_consensus.cc:465] T dc82c30359e84bddb4bcfe1b3b13ced7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 01:54:49.030916 26578 leader_election.cc:302] T 2fd06590221d414b9e896e02b1be6f3e P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:49.031179 28028 raft_consensus.cc:2781] T 2fd06590221d414b9e896e02b1be6f3e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:49.031482 28028 raft_consensus.cc:686] T 2fd06590221d414b9e896e02b1be6f3e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 LEADER]: Becoming Leader. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:49.031680 26640 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c26358e3969143fcbf2f466241b360a7"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:49.031736 28028 consensus_queue.cc:237] T 2fd06590221d414b9e896e02b1be6f3e P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.031875 26640 raft_consensus.cc:482] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.032009 26640 raft_consensus.cc:3037] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.033532 28763 raft_consensus.cc:686] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 LEADER]: Becoming Leader. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:49.033887 28763 consensus_queue.cc:237] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.035295 28762 raft_consensus.cc:686] T d2287fdff0db43df90ff5580c0430b8c P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 LEADER]: Becoming Leader. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:49.035549 28762 consensus_queue.cc:237] T d2287fdff0db43df90ff5580c0430b8c P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.036056 26640 raft_consensus.cc:504] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.037128 26772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c26358e3969143fcbf2f466241b360a7" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:49.037338 26772 raft_consensus.cc:3037] T c26358e3969143fcbf2f466241b360a7 P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.039501 26378 raft_consensus.cc:2454] T 2fd06590221d414b9e896e02b1be6f3e P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 2.
I20240502 01:54:49.040234 26772 raft_consensus.cc:2454] T c26358e3969143fcbf2f466241b360a7 P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 2.
I20240502 01:54:49.040701 26578 leader_election.cc:302] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:49.041335 26640 leader_election.cc:288] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 2 election: Requested vote from peers 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959), 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085)
I20240502 01:54:49.040983 28772 raft_consensus.cc:2781] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:49.042564 26640 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "22d4681165d1488db2c45ec83cb49cb4"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:49.041963 28772 raft_consensus.cc:686] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 LEADER]: Becoming Leader. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:49.042881 26640 raft_consensus.cc:482] T 22d4681165d1488db2c45ec83cb49cb4 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.042996 26640 raft_consensus.cc:3037] T 22d4681165d1488db2c45ec83cb49cb4 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.043758 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c26358e3969143fcbf2f466241b360a7" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:49.043947 26378 raft_consensus.cc:3032] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:49.044046 26378 raft_consensus.cc:729] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:49.044291 26378 consensus_queue.cc:260] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.044624 26378 raft_consensus.cc:3037] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.045295 28772 consensus_queue.cc:237] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.047127 26378 raft_consensus.cc:2454] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 2.
I20240502 01:54:49.049173 26640 raft_consensus.cc:504] T 22d4681165d1488db2c45ec83cb49cb4 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.050316 26640 leader_election.cc:288] T 22d4681165d1488db2c45ec83cb49cb4 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 2 election: Requested vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:49.050751 26772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22d4681165d1488db2c45ec83cb49cb4" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:49.050985 26772 raft_consensus.cc:3037] T 22d4681165d1488db2c45ec83cb49cb4 P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.053802 26772 raft_consensus.cc:2454] T 22d4681165d1488db2c45ec83cb49cb4 P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 2.
I20240502 01:54:49.054107 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22d4681165d1488db2c45ec83cb49cb4" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:49.054224 26578 leader_election.cc:302] T 22d4681165d1488db2c45ec83cb49cb4 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:49.054334 26511 raft_consensus.cc:3032] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:49.054414 26511 raft_consensus.cc:729] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:49.054545 28755 raft_consensus.cc:2781] T 22d4681165d1488db2c45ec83cb49cb4 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:49.054687 28755 raft_consensus.cc:686] T 22d4681165d1488db2c45ec83cb49cb4 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 LEADER]: Becoming Leader. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:49.054689 26511 consensus_queue.cc:260] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.054924 28755 consensus_queue.cc:237] T 22d4681165d1488db2c45ec83cb49cb4 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.055065 26511 raft_consensus.cc:3037] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.062114 26234 catalog_manager.cc:5484] T 1c47a74a6dc54d5eb0b47818d816b7fd P 83c50029f7a94cfca6d1ee68e0728aa2 reported cstate change: term changed from 1 to 2, leader changed from 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68) to 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67). New cstate: current_term: 2 leader_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.062789 26234 catalog_manager.cc:5484] T daef5905fc19473a8728ccd2627270af P 83c50029f7a94cfca6d1ee68e0728aa2 reported cstate change: term changed from 1 to 2, leader changed from 5e9d825f803c40e089b64e319beccf6e (127.11.42.65) to 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67). New cstate: current_term: 2 leader_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.063206 26511 raft_consensus.cc:2454] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 2.
I20240502 01:54:49.063290 26234 catalog_manager.cc:5484] T ef474f44fe5b4a06b2c8dda12b1bc496 P 83c50029f7a94cfca6d1ee68e0728aa2 reported cstate change: term changed from 1 to 2, leader changed from 5e9d825f803c40e089b64e319beccf6e (127.11.42.65) to 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67). New cstate: current_term: 2 leader_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.063745 26643 raft_consensus.cc:1226] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 LEADER]: Rejecting Update request from peer 5e9d825f803c40e089b64e319beccf6e for earlier term 1. Current term is 2. Ops: []
I20240502 01:54:49.063750 26234 catalog_manager.cc:5484] T d4b8975c68d64c2d804fcc852384afde P 83c50029f7a94cfca6d1ee68e0728aa2 reported cstate change: term changed from 1 to 2, leader changed from 4bc1541b47df446297a369de05163012 (127.11.42.66) to 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67). New cstate: current_term: 2 leader_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:49.064199 26234 catalog_manager.cc:5484] T 14a747ac14354fdab2a2993594d3398f P 83c50029f7a94cfca6d1ee68e0728aa2 reported cstate change: term changed from 1 to 2, leader changed from 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68) to 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67). New cstate: current_term: 2 leader_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.065706 26234 catalog_manager.cc:5484] T d2287fdff0db43df90ff5580c0430b8c P 83c50029f7a94cfca6d1ee68e0728aa2 reported cstate change: term changed from 1 to 2, leader changed from 5e9d825f803c40e089b64e319beccf6e (127.11.42.65) to 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67). New cstate: current_term: 2 leader_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.066154 26234 catalog_manager.cc:5484] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2 reported cstate change: term changed from 1 to 2, leader changed from 5e9d825f803c40e089b64e319beccf6e (127.11.42.65) to 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67). New cstate: current_term: 2 leader_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.066485 26234 catalog_manager.cc:5484] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 83c50029f7a94cfca6d1ee68e0728aa2 reported cstate change: term changed from 1 to 2, leader changed from 4bc1541b47df446297a369de05163012 (127.11.42.66) to 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67). New cstate: current_term: 2 leader_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.066887 26234 catalog_manager.cc:5484] T 2fd06590221d414b9e896e02b1be6f3e P 83c50029f7a94cfca6d1ee68e0728aa2 reported cstate change: term changed from 1 to 2, leader changed from 4bc1541b47df446297a369de05163012 (127.11.42.66) to 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67). New cstate: current_term: 2 leader_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.067389 26234 catalog_manager.cc:5484] T 0ddd395eec1146bc8a20292d81be4868 P 83c50029f7a94cfca6d1ee68e0728aa2 reported cstate change: term changed from 1 to 2, leader changed from 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68) to 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67). New cstate: current_term: 2 leader_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:49.067868 26234 catalog_manager.cc:5484] T 99a4b7d022e1461a8233952ab47ef5d3 P 83c50029f7a94cfca6d1ee68e0728aa2 reported cstate change: term changed from 1 to 2, leader changed from 5e9d825f803c40e089b64e319beccf6e (127.11.42.65) to 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67). New cstate: current_term: 2 leader_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.068300 26234 catalog_manager.cc:5484] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 reported cstate change: term changed from 1 to 2, leader changed from 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68) to 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67). New cstate: current_term: 2 leader_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.076702 26234 catalog_manager.cc:5484] T 22d4681165d1488db2c45ec83cb49cb4 P 83c50029f7a94cfca6d1ee68e0728aa2 reported cstate change: term changed from 1 to 2, leader changed from 4bc1541b47df446297a369de05163012 (127.11.42.66) to 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67). New cstate: current_term: 2 leader_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:49.104351 26378 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cf09a605f32845a3aaad81d79063070b"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:49.104588 26378 raft_consensus.cc:482] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.104703 26378 raft_consensus.cc:3037] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.107306 26378 raft_consensus.cc:504] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.108099 26378 leader_election.cc:288] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 3 election: Requested vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:49.110759 26378 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c56cb13e76384d5f9a0e45be2f65c085"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:49.110960 26378 raft_consensus.cc:482] T c56cb13e76384d5f9a0e45be2f65c085 P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.111068 26378 raft_consensus.cc:3037] T c56cb13e76384d5f9a0e45be2f65c085 P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.113479 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf09a605f32845a3aaad81d79063070b" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.113569 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf09a605f32845a3aaad81d79063070b" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:49.113723 26643 raft_consensus.cc:3032] T cf09a605f32845a3aaad81d79063070b P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:54:49.113699 26378 raft_consensus.cc:504] T c56cb13e76384d5f9a0e45be2f65c085 P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.113802 26643 raft_consensus.cc:729] T cf09a605f32845a3aaad81d79063070b P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:49.113801 26511 raft_consensus.cc:3037] T cf09a605f32845a3aaad81d79063070b P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.114110 26643 consensus_queue.cc:260] T cf09a605f32845a3aaad81d79063070b P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.114562 26643 raft_consensus.cc:3037] T cf09a605f32845a3aaad81d79063070b P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.114809 26640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c56cb13e76384d5f9a0e45be2f65c085" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.114993 26640 raft_consensus.cc:3032] T c56cb13e76384d5f9a0e45be2f65c085 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:49.115087 26640 raft_consensus.cc:729] T c56cb13e76384d5f9a0e45be2f65c085 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:49.115339 26640 consensus_queue.cc:260] T c56cb13e76384d5f9a0e45be2f65c085 P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.115825 26640 raft_consensus.cc:3037] T c56cb13e76384d5f9a0e45be2f65c085 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.116529 26511 raft_consensus.cc:2454] T cf09a605f32845a3aaad81d79063070b P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 3.
I20240502 01:54:49.117066 26314 leader_election.cc:302] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:49.117386 28776 raft_consensus.cc:2781] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:54:49.117524 28776 raft_consensus.cc:686] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [term 3 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:49.117784 28776 consensus_queue.cc:237] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.118430 26643 raft_consensus.cc:2396] T cf09a605f32845a3aaad81d79063070b P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e9d825f803c40e089b64e319beccf6e for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.118685 26640 raft_consensus.cc:2454] T c56cb13e76384d5f9a0e45be2f65c085 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 2.
I20240502 01:54:49.119400 26313 leader_election.cc:302] T c56cb13e76384d5f9a0e45be2f65c085 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e9d825f803c40e089b64e319beccf6e, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:49.119732 28191 raft_consensus.cc:2781] T c56cb13e76384d5f9a0e45be2f65c085 P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:49.120270 26772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c56cb13e76384d5f9a0e45be2f65c085" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:49.120527 26772 raft_consensus.cc:3037] T c56cb13e76384d5f9a0e45be2f65c085 P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.123497 26772 raft_consensus.cc:2454] T c56cb13e76384d5f9a0e45be2f65c085 P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 2.
I20240502 01:54:49.124104 26378 leader_election.cc:288] T c56cb13e76384d5f9a0e45be2f65c085 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 2 election: Requested vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:49.125281 26379 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8cf6da25f535446b9515444e093cae7f"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:49.125473 26379 raft_consensus.cc:465] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 01:54:49.126041 28191 raft_consensus.cc:686] T c56cb13e76384d5f9a0e45be2f65c085 P 5e9d825f803c40e089b64e319beccf6e [term 2 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:49.126466 26379 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ef474f44fe5b4a06b2c8dda12b1bc496"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:49.126629 26379 raft_consensus.cc:482] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.126744 26379 raft_consensus.cc:3037] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.126523 28191 consensus_queue.cc:237] T c56cb13e76384d5f9a0e45be2f65c085 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.129464 26379 raft_consensus.cc:504] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.131201 26640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef474f44fe5b4a06b2c8dda12b1bc496" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.131407 26640 raft_consensus.cc:3032] T ef474f44fe5b4a06b2c8dda12b1bc496 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:54:49.131507 26640 raft_consensus.cc:729] T ef474f44fe5b4a06b2c8dda12b1bc496 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:49.131767 26640 consensus_queue.cc:260] T ef474f44fe5b4a06b2c8dda12b1bc496 P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.132063 26640 raft_consensus.cc:3037] T ef474f44fe5b4a06b2c8dda12b1bc496 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.134992 26234 catalog_manager.cc:5484] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 2 to 3, leader changed from 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67) to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 3 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.135753 26640 raft_consensus.cc:2396] T ef474f44fe5b4a06b2c8dda12b1bc496 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e9d825f803c40e089b64e319beccf6e for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.136477 26379 leader_election.cc:288] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 3 election: Requested vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:49.136977 26772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef474f44fe5b4a06b2c8dda12b1bc496" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:49.137212 26772 raft_consensus.cc:3037] T ef474f44fe5b4a06b2c8dda12b1bc496 P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.137650 26379 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "beff61f07ecf4ffab508da651a686c8e"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:49.137849 26379 raft_consensus.cc:482] T beff61f07ecf4ffab508da651a686c8e P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.137961 26379 raft_consensus.cc:3037] T beff61f07ecf4ffab508da651a686c8e P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.139910 26772 raft_consensus.cc:2454] T ef474f44fe5b4a06b2c8dda12b1bc496 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 3.
I20240502 01:54:49.140515 26314 leader_election.cc:302] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 5e9d825f803c40e089b64e319beccf6e; no voters: 83c50029f7a94cfca6d1ee68e0728aa2
I20240502 01:54:49.140910 28776 raft_consensus.cc:2781] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:54:49.141121 28776 raft_consensus.cc:686] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [term 3 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:49.141373 28776 consensus_queue.cc:237] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.140661 26379 raft_consensus.cc:504] T beff61f07ecf4ffab508da651a686c8e P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.143479 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beff61f07ecf4ffab508da651a686c8e" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:49.143700 26511 raft_consensus.cc:3037] T beff61f07ecf4ffab508da651a686c8e P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.143826 26640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beff61f07ecf4ffab508da651a686c8e" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.144013 26640 raft_consensus.cc:3032] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:49.144115 26640 raft_consensus.cc:729] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:49.144349 26640 consensus_queue.cc:260] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.144709 26640 raft_consensus.cc:3037] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.146379 26511 raft_consensus.cc:2454] T beff61f07ecf4ffab508da651a686c8e P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 2.
I20240502 01:54:49.146804 26314 leader_election.cc:302] T beff61f07ecf4ffab508da651a686c8e P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:49.147105 28776 raft_consensus.cc:2781] T beff61f07ecf4ffab508da651a686c8e P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:49.147183 26640 raft_consensus.cc:2454] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 2.
I20240502 01:54:49.147249 28776 raft_consensus.cc:686] T beff61f07ecf4ffab508da651a686c8e P 5e9d825f803c40e089b64e319beccf6e [term 2 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:49.147568 28776 consensus_queue.cc:237] T beff61f07ecf4ffab508da651a686c8e P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.147696 26379 leader_election.cc:288] T beff61f07ecf4ffab508da651a686c8e P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 2 election: Requested vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:49.148669 26378 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4254942d7ddc4c408114ae8d4b932a1a"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:49.148872 26378 raft_consensus.cc:482] T 4254942d7ddc4c408114ae8d4b932a1a P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.149021 26378 raft_consensus.cc:3037] T 4254942d7ddc4c408114ae8d4b932a1a P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.151597 26378 raft_consensus.cc:504] T 4254942d7ddc4c408114ae8d4b932a1a P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.152580 26772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4254942d7ddc4c408114ae8d4b932a1a" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:49.152789 26772 raft_consensus.cc:3037] T 4254942d7ddc4c408114ae8d4b932a1a P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.155887 26772 raft_consensus.cc:2454] T 4254942d7ddc4c408114ae8d4b932a1a P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 2.
I20240502 01:54:49.156345 26314 leader_election.cc:302] T 4254942d7ddc4c408114ae8d4b932a1a P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:49.156625 28789 raft_consensus.cc:2781] T 4254942d7ddc4c408114ae8d4b932a1a P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:49.156836 26378 leader_election.cc:288] T 4254942d7ddc4c408114ae8d4b932a1a P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 2 election: Requested vote from peers 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:49.157704 26377 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4f9ed0162eb74c4ea422f122da01dce3"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:49.157752 26640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4254942d7ddc4c408114ae8d4b932a1a" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.157891 26377 raft_consensus.cc:482] T 4f9ed0162eb74c4ea422f122da01dce3 P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.157907 26640 raft_consensus.cc:3032] T 4254942d7ddc4c408114ae8d4b932a1a P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:49.158000 26640 raft_consensus.cc:729] T 4254942d7ddc4c408114ae8d4b932a1a P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:49.158020 26377 raft_consensus.cc:3037] T 4f9ed0162eb74c4ea422f122da01dce3 P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.158259 26640 consensus_queue.cc:260] T 4254942d7ddc4c408114ae8d4b932a1a P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.158629 26640 raft_consensus.cc:3037] T 4254942d7ddc4c408114ae8d4b932a1a P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.160742 26377 raft_consensus.cc:504] T 4f9ed0162eb74c4ea422f122da01dce3 P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.161201 26640 raft_consensus.cc:2454] T 4254942d7ddc4c408114ae8d4b932a1a P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 2.
I20240502 01:54:49.161494 26377 leader_election.cc:288] T 4f9ed0162eb74c4ea422f122da01dce3 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 2 election: Requested vote from peers 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:49.162413 26376 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9ead21d62e564a37a8c3d5a452621d13"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:49.162629 26376 raft_consensus.cc:482] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.162766 26376 raft_consensus.cc:3037] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.169754 26376 raft_consensus.cc:504] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.170702 26376 leader_election.cc:288] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 2 election: Requested vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:49.179924 26640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ead21d62e564a37a8c3d5a452621d13" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.180151 26640 raft_consensus.cc:3032] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:49.180254 26640 raft_consensus.cc:729] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:49.180593 26640 consensus_queue.cc:260] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.180999 26640 raft_consensus.cc:3037] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.183959 26640 raft_consensus.cc:2454] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 2.
I20240502 01:54:49.184422 26313 leader_election.cc:302] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e9d825f803c40e089b64e319beccf6e, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:49.184706 28776 raft_consensus.cc:2781] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:49.184859 28776 raft_consensus.cc:686] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e [term 2 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:49.185159 28776 consensus_queue.cc:237] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.198987 26379 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b4ba7c4fe9614ab3abb3645c1da3fcd3"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:49.199178 26379 raft_consensus.cc:482] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.199316 26379 raft_consensus.cc:3037] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.202263 26379 raft_consensus.cc:504] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.203189 26379 leader_election.cc:288] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 3 election: Requested vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:49.203379 26640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ba7c4fe9614ab3abb3645c1da3fcd3" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.203627 26640 raft_consensus.cc:3032] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:54:49.203858 26640 raft_consensus.cc:729] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:49.204257 26379 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "53b7377c0d144e07a4501c2d626f6494"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:49.204258 26640 consensus_queue.cc:260] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.204466 26379 raft_consensus.cc:482] T 53b7377c0d144e07a4501c2d626f6494 P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.204571 26379 raft_consensus.cc:3037] T 53b7377c0d144e07a4501c2d626f6494 P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.204759 26640 raft_consensus.cc:3037] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.206502 28789 raft_consensus.cc:686] T 4254942d7ddc4c408114ae8d4b932a1a P 5e9d825f803c40e089b64e319beccf6e [term 2 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:49.207850 26640 raft_consensus.cc:2396] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e9d825f803c40e089b64e319beccf6e for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.208050 28789 consensus_queue.cc:237] T 4254942d7ddc4c408114ae8d4b932a1a P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.209074 26234 catalog_manager.cc:5484] T c56cb13e76384d5f9a0e45be2f65c085 P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 1 to 2, leader changed from 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67) to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 2 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.210793 26772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9ed0162eb74c4ea422f122da01dce3" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:49.211023 26772 raft_consensus.cc:3032] T 4f9ed0162eb74c4ea422f122da01dce3 P 401942254c4c43a1ba02c4ae68d17320 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:49.211107 26772 raft_consensus.cc:729] T 4f9ed0162eb74c4ea422f122da01dce3 P 401942254c4c43a1ba02c4ae68d17320 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Running, Role: LEADER
I20240502 01:54:49.211417 26772 consensus_queue.cc:260] T 4f9ed0162eb74c4ea422f122da01dce3 P 401942254c4c43a1ba02c4ae68d17320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.211769 26772 raft_consensus.cc:3037] T 4f9ed0162eb74c4ea422f122da01dce3 P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.214349 26772 raft_consensus.cc:2454] T 4f9ed0162eb74c4ea422f122da01dce3 P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 2.
I20240502 01:54:49.215102 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9ed0162eb74c4ea422f122da01dce3" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:49.215364 26511 raft_consensus.cc:3037] T 4f9ed0162eb74c4ea422f122da01dce3 P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.215605 26234 catalog_manager.cc:5484] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 2 to 3, leader changed from 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67) to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 3 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.216118 26234 catalog_manager.cc:5484] T beff61f07ecf4ffab508da651a686c8e P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 1 to 2, leader changed from 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67) to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 2 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.216588 26234 catalog_manager.cc:5484] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 1 to 2, leader changed from 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67) to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 2 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.217192 26234 catalog_manager.cc:5484] T 4254942d7ddc4c408114ae8d4b932a1a P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 1 to 2, leader changed from 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67) to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 2 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:49.218034 26511 raft_consensus.cc:2454] T 4f9ed0162eb74c4ea422f122da01dce3 P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 2.
I20240502 01:54:49.218883 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ead21d62e564a37a8c3d5a452621d13" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:49.219085 26511 raft_consensus.cc:3037] T 9ead21d62e564a37a8c3d5a452621d13 P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.221693 26511 raft_consensus.cc:2454] T 9ead21d62e564a37a8c3d5a452621d13 P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 2.
I20240502 01:54:49.221966 26314 leader_election.cc:302] T 4f9ed0162eb74c4ea422f122da01dce3 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:49.222003 26509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ba7c4fe9614ab3abb3645c1da3fcd3" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:49.222213 26509 raft_consensus.cc:3037] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.222656 28191 raft_consensus.cc:2781] T 4f9ed0162eb74c4ea422f122da01dce3 P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:49.222822 28191 raft_consensus.cc:686] T 4f9ed0162eb74c4ea422f122da01dce3 P 5e9d825f803c40e089b64e319beccf6e [term 2 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:49.223050 28191 consensus_queue.cc:237] T 4f9ed0162eb74c4ea422f122da01dce3 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.224403 26379 raft_consensus.cc:504] T 53b7377c0d144e07a4501c2d626f6494 P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.224591 26509 raft_consensus.cc:2454] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 3.
I20240502 01:54:49.226281 26314 leader_election.cc:302] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4bc1541b47df446297a369de05163012, 5e9d825f803c40e089b64e319beccf6e; no voters: 83c50029f7a94cfca6d1ee68e0728aa2
I20240502 01:54:49.227257 28789 raft_consensus.cc:2781] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:54:49.227527 28789 raft_consensus.cc:686] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 5e9d825f803c40e089b64e319beccf6e [term 3 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:49.227782 28789 consensus_queue.cc:237] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.233893 26234 catalog_manager.cc:5484] T 4f9ed0162eb74c4ea422f122da01dce3 P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 1 to 2, leader changed from 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68) to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 2 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:49.237542 26379 leader_election.cc:288] T 53b7377c0d144e07a4501c2d626f6494 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 2 election: Requested vote from peers 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:49.238154 26640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b7377c0d144e07a4501c2d626f6494" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.238373 26640 raft_consensus.cc:3032] T 53b7377c0d144e07a4501c2d626f6494 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:49.238313 26772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b7377c0d144e07a4501c2d626f6494" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:49.238474 26640 raft_consensus.cc:729] T 53b7377c0d144e07a4501c2d626f6494 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:49.238521 26772 raft_consensus.cc:3037] T 53b7377c0d144e07a4501c2d626f6494 P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.238731 26640 consensus_queue.cc:260] T 53b7377c0d144e07a4501c2d626f6494 P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.239156 26640 raft_consensus.cc:3037] T 53b7377c0d144e07a4501c2d626f6494 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.241148 26772 raft_consensus.cc:2454] T 53b7377c0d144e07a4501c2d626f6494 P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 2.
I20240502 01:54:49.241909 26379 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d2287fdff0db43df90ff5580c0430b8c"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:49.242079 26314 leader_election.cc:302] T 53b7377c0d144e07a4501c2d626f6494 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:49.242098 26379 raft_consensus.cc:482] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.242282 26379 raft_consensus.cc:3037] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.243130 26234 catalog_manager.cc:5484] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 2 to 3, leader changed from 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67) to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 3 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.245126 26379 raft_consensus.cc:504] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.245791 26379 leader_election.cc:288] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 3 election: Requested vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:49.246737 26379 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "245ca06c39774fb5b0f522550a63ac5b"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:49.246850 26509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2287fdff0db43df90ff5580c0430b8c" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:49.246937 26379 raft_consensus.cc:465] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 01:54:49.247071 26509 raft_consensus.cc:3037] T d2287fdff0db43df90ff5580c0430b8c P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.247748 26379 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "daef5905fc19473a8728ccd2627270af"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:49.247936 26379 raft_consensus.cc:482] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.248036 26379 raft_consensus.cc:3037] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.249642 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2287fdff0db43df90ff5580c0430b8c" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.249866 26643 raft_consensus.cc:3032] T d2287fdff0db43df90ff5580c0430b8c P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:54:49.249970 26643 raft_consensus.cc:729] T d2287fdff0db43df90ff5580c0430b8c P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:49.250231 26643 consensus_queue.cc:260] T d2287fdff0db43df90ff5580c0430b8c P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.248958 26509 raft_consensus.cc:2454] T d2287fdff0db43df90ff5580c0430b8c P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 3.
I20240502 01:54:49.250478 26640 raft_consensus.cc:2454] T 53b7377c0d144e07a4501c2d626f6494 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 2.
I20240502 01:54:49.250509 26643 raft_consensus.cc:3037] T d2287fdff0db43df90ff5580c0430b8c P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.253173 28789 raft_consensus.cc:2781] T 53b7377c0d144e07a4501c2d626f6494 P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:49.253247 26643 raft_consensus.cc:2396] T d2287fdff0db43df90ff5580c0430b8c P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e9d825f803c40e089b64e319beccf6e for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.253415 28789 raft_consensus.cc:686] T 53b7377c0d144e07a4501c2d626f6494 P 5e9d825f803c40e089b64e319beccf6e [term 2 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:49.253729 28789 consensus_queue.cc:237] T 53b7377c0d144e07a4501c2d626f6494 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.254278 26314 leader_election.cc:302] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4bc1541b47df446297a369de05163012, 5e9d825f803c40e089b64e319beccf6e; no voters: 83c50029f7a94cfca6d1ee68e0728aa2
I20240502 01:54:49.254725 28191 raft_consensus.cc:2781] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:54:49.254881 28191 raft_consensus.cc:686] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [term 3 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:49.255153 28191 consensus_queue.cc:237] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.259140 26379 raft_consensus.cc:504] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.260186 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daef5905fc19473a8728ccd2627270af" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.260403 26643 raft_consensus.cc:3032] T daef5905fc19473a8728ccd2627270af P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:54:49.260483 26643 raft_consensus.cc:729] T daef5905fc19473a8728ccd2627270af P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:49.260761 26643 consensus_queue.cc:260] T daef5905fc19473a8728ccd2627270af P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.261049 26643 raft_consensus.cc:3037] T daef5905fc19473a8728ccd2627270af P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.263692 26643 raft_consensus.cc:2396] T daef5905fc19473a8728ccd2627270af P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e9d825f803c40e089b64e319beccf6e for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.264786 26772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daef5905fc19473a8728ccd2627270af" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:49.264999 26772 raft_consensus.cc:3037] T daef5905fc19473a8728ccd2627270af P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.267645 26772 raft_consensus.cc:2454] T daef5905fc19473a8728ccd2627270af P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 3.
I20240502 01:54:49.267454 26234 catalog_manager.cc:5484] T 53b7377c0d144e07a4501c2d626f6494 P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 1 to 2, leader changed from 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67) to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 2 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.268103 26314 leader_election.cc:302] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 5e9d825f803c40e089b64e319beccf6e; no voters: 83c50029f7a94cfca6d1ee68e0728aa2
I20240502 01:54:49.268393 26379 leader_election.cc:288] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 3 election: Requested vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:49.270143 26377 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "99a4b7d022e1461a8233952ab47ef5d3"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:49.270364 26377 raft_consensus.cc:482] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.270464 26377 raft_consensus.cc:3037] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.270182 28191 raft_consensus.cc:2781] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:54:49.276021 26377 raft_consensus.cc:504] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.276927 26509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a4b7d022e1461a8233952ab47ef5d3" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:49.277146 26509 raft_consensus.cc:3037] T 99a4b7d022e1461a8233952ab47ef5d3 P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.279785 26509 raft_consensus.cc:2454] T 99a4b7d022e1461a8233952ab47ef5d3 P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 3.
I20240502 01:54:49.280413 26314 leader_election.cc:302] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:49.281754 26234 catalog_manager.cc:5484] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 2 to 3, leader changed from 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67) to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 3 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.282655 28794 raft_consensus.cc:2781] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:54:49.282804 28794 raft_consensus.cc:686] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [term 3 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:49.283080 28794 consensus_queue.cc:237] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.283380 28191 raft_consensus.cc:686] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [term 3 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:49.283668 28191 consensus_queue.cc:237] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.285476 26377 leader_election.cc:288] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 3 election: Requested vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:49.286168 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a4b7d022e1461a8233952ab47ef5d3" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.286378 26643 raft_consensus.cc:3032] T 99a4b7d022e1461a8233952ab47ef5d3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:54:49.286461 26643 raft_consensus.cc:729] T 99a4b7d022e1461a8233952ab47ef5d3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:49.286734 26643 consensus_queue.cc:260] T 99a4b7d022e1461a8233952ab47ef5d3 P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.287050 26643 raft_consensus.cc:3037] T 99a4b7d022e1461a8233952ab47ef5d3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.289781 26643 raft_consensus.cc:2396] T 99a4b7d022e1461a8233952ab47ef5d3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e9d825f803c40e089b64e319beccf6e for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.291556 26377 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0ddd395eec1146bc8a20292d81be4868"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:49.291743 26377 raft_consensus.cc:482] T 0ddd395eec1146bc8a20292d81be4868 P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.291954 26377 raft_consensus.cc:3037] T 0ddd395eec1146bc8a20292d81be4868 P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.294934 26377 raft_consensus.cc:504] T 0ddd395eec1146bc8a20292d81be4868 P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.295886 26773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ddd395eec1146bc8a20292d81be4868" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:49.295990 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ddd395eec1146bc8a20292d81be4868" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.296077 26773 raft_consensus.cc:3037] T 0ddd395eec1146bc8a20292d81be4868 P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.296171 26643 raft_consensus.cc:3032] T 0ddd395eec1146bc8a20292d81be4868 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:54:49.296240 26643 raft_consensus.cc:729] T 0ddd395eec1146bc8a20292d81be4868 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:49.296517 26643 consensus_queue.cc:260] T 0ddd395eec1146bc8a20292d81be4868 P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.296774 26643 raft_consensus.cc:3037] T 0ddd395eec1146bc8a20292d81be4868 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.298705 26773 raft_consensus.cc:2454] T 0ddd395eec1146bc8a20292d81be4868 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 3.
I20240502 01:54:49.299245 26314 leader_election.cc:302] T 0ddd395eec1146bc8a20292d81be4868 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:49.299518 28794 raft_consensus.cc:2781] T 0ddd395eec1146bc8a20292d81be4868 P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:54:49.299423 26234 catalog_manager.cc:5484] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 2 to 3, leader changed from 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67) to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 3 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.299672 28794 raft_consensus.cc:686] T 0ddd395eec1146bc8a20292d81be4868 P 5e9d825f803c40e089b64e319beccf6e [term 3 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:49.299894 28794 consensus_queue.cc:237] T 0ddd395eec1146bc8a20292d81be4868 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.299957 26234 catalog_manager.cc:5484] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 2 to 3, leader changed from 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67) to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 3 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:49.303437 26643 raft_consensus.cc:2396] T 0ddd395eec1146bc8a20292d81be4868 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e9d825f803c40e089b64e319beccf6e for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.303946 26377 leader_election.cc:288] T 0ddd395eec1146bc8a20292d81be4868 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 3 election: Requested vote from peers 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:49.305049 26379 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2fd06590221d414b9e896e02b1be6f3e"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:49.305240 26379 raft_consensus.cc:482] T 2fd06590221d414b9e896e02b1be6f3e P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.305368 26379 raft_consensus.cc:3037] T 2fd06590221d414b9e896e02b1be6f3e P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.307113 26379 raft_consensus.cc:504] T 2fd06590221d414b9e896e02b1be6f3e P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.307626 26379 leader_election.cc:288] T 2fd06590221d414b9e896e02b1be6f3e P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 3 election: Requested vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:49.308322 26509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fd06590221d414b9e896e02b1be6f3e" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:49.308557 26509 raft_consensus.cc:3037] T 2fd06590221d414b9e896e02b1be6f3e P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.308816 26379 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "23dcaac2d7d14adf820d826f05c1abc6"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:49.309005 26379 raft_consensus.cc:482] T 23dcaac2d7d14adf820d826f05c1abc6 P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.309134 26379 raft_consensus.cc:3037] T 23dcaac2d7d14adf820d826f05c1abc6 P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.309072 26234 catalog_manager.cc:5484] T 0ddd395eec1146bc8a20292d81be4868 P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 2 to 3, leader changed from 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67) to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 3 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.311092 26509 raft_consensus.cc:2454] T 2fd06590221d414b9e896e02b1be6f3e P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 3.
I20240502 01:54:49.311620 26314 leader_election.cc:302] T 2fd06590221d414b9e896e02b1be6f3e P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:49.311888 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fd06590221d414b9e896e02b1be6f3e" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.312120 26643 raft_consensus.cc:3032] T 2fd06590221d414b9e896e02b1be6f3e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:54:49.312209 26643 raft_consensus.cc:729] T 2fd06590221d414b9e896e02b1be6f3e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:49.312220 28191 raft_consensus.cc:2781] T 2fd06590221d414b9e896e02b1be6f3e P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:54:49.312371 28191 raft_consensus.cc:686] T 2fd06590221d414b9e896e02b1be6f3e P 5e9d825f803c40e089b64e319beccf6e [term 3 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:49.312503 26643 consensus_queue.cc:260] T 2fd06590221d414b9e896e02b1be6f3e P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.312625 28191 consensus_queue.cc:237] T 2fd06590221d414b9e896e02b1be6f3e P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.312822 26643 raft_consensus.cc:3037] T 2fd06590221d414b9e896e02b1be6f3e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.315171 26379 raft_consensus.cc:504] T 23dcaac2d7d14adf820d826f05c1abc6 P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.315325 26234 catalog_manager.cc:5484] T 2fd06590221d414b9e896e02b1be6f3e P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 2 to 3, leader changed from 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67) to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 3 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.315834 26379 leader_election.cc:288] T 23dcaac2d7d14adf820d826f05c1abc6 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 2 election: Requested vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:49.316198 26509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23dcaac2d7d14adf820d826f05c1abc6" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:49.316323 26640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23dcaac2d7d14adf820d826f05c1abc6" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.316372 26509 raft_consensus.cc:3037] T 23dcaac2d7d14adf820d826f05c1abc6 P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.316486 26640 raft_consensus.cc:3032] T 23dcaac2d7d14adf820d826f05c1abc6 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:49.316572 26640 raft_consensus.cc:729] T 23dcaac2d7d14adf820d826f05c1abc6 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:49.316814 26377 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c9e4788d50d04d129e8d03125073c6f2"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:49.316802 26640 consensus_queue.cc:260] T 23dcaac2d7d14adf820d826f05c1abc6 P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.317004 26377 raft_consensus.cc:465] T c9e4788d50d04d129e8d03125073c6f2 P 5e9d825f803c40e089b64e319beccf6e [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 01:54:49.317183 26640 raft_consensus.cc:3037] T 23dcaac2d7d14adf820d826f05c1abc6 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.317967 26378 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "37e818dc0b834eb185e27fcb3cb972e1"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:49.318167 26378 raft_consensus.cc:465] T 37e818dc0b834eb185e27fcb3cb972e1 P 5e9d825f803c40e089b64e319beccf6e [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 01:54:49.318907 26509 raft_consensus.cc:2454] T 23dcaac2d7d14adf820d826f05c1abc6 P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 2.
I20240502 01:54:49.319041 26378 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c26358e3969143fcbf2f466241b360a7"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:49.319232 26378 raft_consensus.cc:482] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.319330 26378 raft_consensus.cc:3037] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.319552 26314 leader_election.cc:302] T 23dcaac2d7d14adf820d826f05c1abc6 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:49.319703 26640 raft_consensus.cc:2454] T 23dcaac2d7d14adf820d826f05c1abc6 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 2.
I20240502 01:54:49.320459 28191 raft_consensus.cc:2781] T 23dcaac2d7d14adf820d826f05c1abc6 P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:49.320706 28191 raft_consensus.cc:686] T 23dcaac2d7d14adf820d826f05c1abc6 P 5e9d825f803c40e089b64e319beccf6e [term 2 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:49.322233 26378 raft_consensus.cc:504] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.323315 26509 raft_consensus.cc:1261] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Refusing update from remote peer 83c50029f7a94cfca6d1ee68e0728aa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:54:49.323340 26773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c26358e3969143fcbf2f466241b360a7" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:49.323316 26640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c26358e3969143fcbf2f466241b360a7" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.323539 26640 raft_consensus.cc:3032] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:54:49.323552 26773 raft_consensus.cc:3037] T c26358e3969143fcbf2f466241b360a7 P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.323632 26640 raft_consensus.cc:729] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:49.323995 28755 consensus_queue.cc:1035] T d4b8975c68d64c2d804fcc852384afde P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:49.325121 26643 raft_consensus.cc:2396] T 2fd06590221d414b9e896e02b1be6f3e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e9d825f803c40e089b64e319beccf6e for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.323999 26640 consensus_queue.cc:260] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.325816 26640 raft_consensus.cc:3037] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.327153 26378 leader_election.cc:288] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 3 election: Requested vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:49.326992 26773 raft_consensus.cc:2454] T c26358e3969143fcbf2f466241b360a7 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 3.
I20240502 01:54:49.328181 26379 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3d751cdd58eb4cef8039fd4c50855e99"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:49.328363 26379 raft_consensus.cc:482] T 3d751cdd58eb4cef8039fd4c50855e99 P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.328461 26640 raft_consensus.cc:2396] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e9d825f803c40e089b64e319beccf6e for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.328434 26314 leader_election.cc:302] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:49.328495 26379 raft_consensus.cc:3037] T 3d751cdd58eb4cef8039fd4c50855e99 P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.328778 28794 raft_consensus.cc:2781] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:54:49.328930 28794 raft_consensus.cc:686] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [term 3 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:49.329188 28794 consensus_queue.cc:237] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.322237 28191 consensus_queue.cc:237] T 23dcaac2d7d14adf820d826f05c1abc6 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.331743 26379 raft_consensus.cc:504] T 3d751cdd58eb4cef8039fd4c50855e99 P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.332715 26640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d751cdd58eb4cef8039fd4c50855e99" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.332918 26640 raft_consensus.cc:3032] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:49.333045 26640 raft_consensus.cc:729] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:49.333386 26640 consensus_queue.cc:260] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.333819 26640 raft_consensus.cc:3037] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.336580 26640 raft_consensus.cc:2454] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 2.
I20240502 01:54:49.337040 26313 leader_election.cc:302] T 3d751cdd58eb4cef8039fd4c50855e99 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e9d825f803c40e089b64e319beccf6e, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:49.337395 28789 raft_consensus.cc:2781] T 3d751cdd58eb4cef8039fd4c50855e99 P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:49.337555 28789 raft_consensus.cc:686] T 3d751cdd58eb4cef8039fd4c50855e99 P 5e9d825f803c40e089b64e319beccf6e [term 2 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:49.337810 28789 consensus_queue.cc:237] T 3d751cdd58eb4cef8039fd4c50855e99 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.339151 26379 leader_election.cc:288] T 3d751cdd58eb4cef8039fd4c50855e99 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 2 election: Requested vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:49.339774 26773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d751cdd58eb4cef8039fd4c50855e99" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:49.339969 26773 raft_consensus.cc:3037] T 3d751cdd58eb4cef8039fd4c50855e99 P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.342692 26773 raft_consensus.cc:2454] T 3d751cdd58eb4cef8039fd4c50855e99 P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 2.
I20240502 01:54:49.344012 26379 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1ce4dd22bb314eb7855126e3fe1785f6"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:49.344223 26379 raft_consensus.cc:482] T 1ce4dd22bb314eb7855126e3fe1785f6 P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.344336 26379 raft_consensus.cc:3037] T 1ce4dd22bb314eb7855126e3fe1785f6 P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.347029 26379 raft_consensus.cc:504] T 1ce4dd22bb314eb7855126e3fe1785f6 P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.347679 26379 leader_election.cc:288] T 1ce4dd22bb314eb7855126e3fe1785f6 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 2 election: Requested vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:49.348090 26510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ce4dd22bb314eb7855126e3fe1785f6" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:49.348309 26510 raft_consensus.cc:3037] T 1ce4dd22bb314eb7855126e3fe1785f6 P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.349153 26234 catalog_manager.cc:5484] T 23dcaac2d7d14adf820d826f05c1abc6 P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 1 to 2, leader changed from 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67) to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 2 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.350945 26510 raft_consensus.cc:2454] T 1ce4dd22bb314eb7855126e3fe1785f6 P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 2.
I20240502 01:54:49.352200 26314 leader_election.cc:302] T 1ce4dd22bb314eb7855126e3fe1785f6 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:49.353443 26773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ce4dd22bb314eb7855126e3fe1785f6" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:49.353667 26773 raft_consensus.cc:3032] T 1ce4dd22bb314eb7855126e3fe1785f6 P 401942254c4c43a1ba02c4ae68d17320 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:49.353754 26773 raft_consensus.cc:729] T 1ce4dd22bb314eb7855126e3fe1785f6 P 401942254c4c43a1ba02c4ae68d17320 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Running, Role: LEADER
I20240502 01:54:49.354028 26773 consensus_queue.cc:260] T 1ce4dd22bb314eb7855126e3fe1785f6 P 401942254c4c43a1ba02c4ae68d17320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.354454 26773 raft_consensus.cc:3037] T 1ce4dd22bb314eb7855126e3fe1785f6 P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.355629 26379 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b5ab243d9f1c4b568d38ed03027adb44"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:49.355806 26379 raft_consensus.cc:482] T b5ab243d9f1c4b568d38ed03027adb44 P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.356757 26379 raft_consensus.cc:3037] T b5ab243d9f1c4b568d38ed03027adb44 P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.356391 26234 catalog_manager.cc:5484] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 2 to 3, leader changed from 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67) to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 3 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:49.357234 26234 catalog_manager.cc:5484] T 3d751cdd58eb4cef8039fd4c50855e99 P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 1 to 2, leader changed from 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67) to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 2 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.359357 26379 raft_consensus.cc:504] T b5ab243d9f1c4b568d38ed03027adb44 P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.360438 26510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ab243d9f1c4b568d38ed03027adb44" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:49.360637 26510 raft_consensus.cc:3032] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:49.360721 26510 raft_consensus.cc:729] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:49.360986 26510 consensus_queue.cc:260] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.361366 26510 raft_consensus.cc:3037] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.364027 26510 raft_consensus.cc:2454] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 2.
I20240502 01:54:49.364471 26314 leader_election.cc:302] T b5ab243d9f1c4b568d38ed03027adb44 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:49.364789 28789 raft_consensus.cc:2781] T 1ce4dd22bb314eb7855126e3fe1785f6 P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:49.364953 28789 raft_consensus.cc:686] T 1ce4dd22bb314eb7855126e3fe1785f6 P 5e9d825f803c40e089b64e319beccf6e [term 2 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:49.365032 28794 raft_consensus.cc:2781] T b5ab243d9f1c4b568d38ed03027adb44 P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:49.365211 28794 raft_consensus.cc:686] T b5ab243d9f1c4b568d38ed03027adb44 P 5e9d825f803c40e089b64e319beccf6e [term 2 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:49.365265 28789 consensus_queue.cc:237] T 1ce4dd22bb314eb7855126e3fe1785f6 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.365447 28794 consensus_queue.cc:237] T b5ab243d9f1c4b568d38ed03027adb44 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.367059 26772 raft_consensus.cc:1261] T d4b8975c68d64c2d804fcc852384afde P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Refusing update from remote peer 83c50029f7a94cfca6d1ee68e0728aa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:54:49.369047 26773 raft_consensus.cc:2454] T 1ce4dd22bb314eb7855126e3fe1785f6 P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 2.
I20240502 01:54:49.370107 28762 consensus_queue.cc:1035] T d4b8975c68d64c2d804fcc852384afde P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:49.370396 26228 catalog_manager.cc:5484] T 1ce4dd22bb314eb7855126e3fe1785f6 P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 1 to 2, leader changed from 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68) to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 2 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:49.370877 26228 catalog_manager.cc:5484] T b5ab243d9f1c4b568d38ed03027adb44 P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 1 to 2, leader changed from 4bc1541b47df446297a369de05163012 (127.11.42.66) to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 2 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:49.371062 26773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ab243d9f1c4b568d38ed03027adb44" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:49.371275 26773 raft_consensus.cc:3037] T b5ab243d9f1c4b568d38ed03027adb44 P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.374701 26379 leader_election.cc:288] T b5ab243d9f1c4b568d38ed03027adb44 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 2 election: Requested vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:49.376127 26510 raft_consensus.cc:1261] T 1c47a74a6dc54d5eb0b47818d816b7fd P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Refusing update from remote peer 83c50029f7a94cfca6d1ee68e0728aa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:54:49.376746 28772 consensus_queue.cc:1035] T 1c47a74a6dc54d5eb0b47818d816b7fd P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:49.377475 26773 raft_consensus.cc:2454] T b5ab243d9f1c4b568d38ed03027adb44 P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 2.
I20240502 01:54:49.386425 28613 heartbeater.cc:503] Master 127.11.42.126:36275 was elected leader, sending a full tablet report...
I20240502 01:54:49.392258 26772 raft_consensus.cc:1261] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Refusing update from remote peer 83c50029f7a94cfca6d1ee68e0728aa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:54:49.395579 28028 consensus_queue.cc:1035] T 1c47a74a6dc54d5eb0b47818d816b7fd P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:49.401736 26511 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cf09a605f32845a3aaad81d79063070b"
dest_uuid: "4bc1541b47df446297a369de05163012"
from {username='slave'} at 127.0.0.1:41462
I20240502 01:54:49.402037 26511 raft_consensus.cc:482] T cf09a605f32845a3aaad81d79063070b P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.402158 26511 raft_consensus.cc:3037] T cf09a605f32845a3aaad81d79063070b P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.405557 26511 raft_consensus.cc:504] T cf09a605f32845a3aaad81d79063070b P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.406739 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf09a605f32845a3aaad81d79063070b" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:49.406972 26379 raft_consensus.cc:3032] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:54:49.407054 26379 raft_consensus.cc:729] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:49.407049 26511 leader_election.cc:288] T cf09a605f32845a3aaad81d79063070b P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 4 election: Requested vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:49.407327 26379 consensus_queue.cc:260] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.407629 26379 raft_consensus.cc:3037] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.408030 26510 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8cf6da25f535446b9515444e093cae7f"
dest_uuid: "4bc1541b47df446297a369de05163012"
from {username='slave'} at 127.0.0.1:41462
I20240502 01:54:49.408210 26510 raft_consensus.cc:482] T 8cf6da25f535446b9515444e093cae7f P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.408301 26510 raft_consensus.cc:3037] T 8cf6da25f535446b9515444e093cae7f P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.409215 26640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf09a605f32845a3aaad81d79063070b" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.409415 26640 raft_consensus.cc:3037] T cf09a605f32845a3aaad81d79063070b P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.410856 26510 raft_consensus.cc:504] T 8cf6da25f535446b9515444e093cae7f P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.411615 26510 leader_election.cc:288] T 8cf6da25f535446b9515444e093cae7f P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 2 election: Requested vote from peers 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959), 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085)
I20240502 01:54:49.412070 26640 raft_consensus.cc:2396] T cf09a605f32845a3aaad81d79063070b P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bc1541b47df446297a369de05163012 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.412068 26772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cf6da25f535446b9515444e093cae7f" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:49.412536 26772 raft_consensus.cc:3037] T 8cf6da25f535446b9515444e093cae7f P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.412799 26509 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "14a747ac14354fdab2a2993594d3398f"
dest_uuid: "4bc1541b47df446297a369de05163012"
from {username='slave'} at 127.0.0.1:41462
I20240502 01:54:49.412974 26509 raft_consensus.cc:482] T 14a747ac14354fdab2a2993594d3398f P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.413118 26509 raft_consensus.cc:3037] T 14a747ac14354fdab2a2993594d3398f P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.415254 26379 raft_consensus.cc:2396] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bc1541b47df446297a369de05163012 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.415643 26509 raft_consensus.cc:504] T 14a747ac14354fdab2a2993594d3398f P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.416316 26509 leader_election.cc:288] T 14a747ac14354fdab2a2993594d3398f P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 3 election: Requested vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:49.417362 26773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a747ac14354fdab2a2993594d3398f" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:49.417531 26446 leader_election.cc:302] T cf09a605f32845a3aaad81d79063070b P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bc1541b47df446297a369de05163012; no voters: 5e9d825f803c40e089b64e319beccf6e, 83c50029f7a94cfca6d1ee68e0728aa2
I20240502 01:54:49.417595 26773 raft_consensus.cc:3037] T 14a747ac14354fdab2a2993594d3398f P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.418030 28767 raft_consensus.cc:2726] T cf09a605f32845a3aaad81d79063070b P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 01:54:49.418929 26509 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "beff61f07ecf4ffab508da651a686c8e"
dest_uuid: "4bc1541b47df446297a369de05163012"
from {username='slave'} at 127.0.0.1:41462
I20240502 01:54:49.419114 26509 raft_consensus.cc:482] T beff61f07ecf4ffab508da651a686c8e P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.419210 26509 raft_consensus.cc:3037] T beff61f07ecf4ffab508da651a686c8e P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.420049 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cf6da25f535446b9515444e093cae7f" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:49.420250 26773 raft_consensus.cc:2454] T 14a747ac14354fdab2a2993594d3398f P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 3.
I20240502 01:54:49.420281 26379 raft_consensus.cc:3032] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:49.420234 26640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a747ac14354fdab2a2993594d3398f" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.420363 26379 raft_consensus.cc:729] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:49.420528 26640 raft_consensus.cc:3032] T 14a747ac14354fdab2a2993594d3398f P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:54:49.420611 26640 raft_consensus.cc:729] T 14a747ac14354fdab2a2993594d3398f P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:49.420634 26379 consensus_queue.cc:260] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.420874 26640 consensus_queue.cc:260] T 14a747ac14354fdab2a2993594d3398f P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.421048 26379 raft_consensus.cc:3037] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.421192 26640 raft_consensus.cc:3037] T 14a747ac14354fdab2a2993594d3398f P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.421273 26447 leader_election.cc:302] T 14a747ac14354fdab2a2993594d3398f P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 4bc1541b47df446297a369de05163012; no voters:
I20240502 01:54:49.421563 28767 raft_consensus.cc:2781] T 14a747ac14354fdab2a2993594d3398f P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:54:49.421679 28767 raft_consensus.cc:686] T 14a747ac14354fdab2a2993594d3398f P 4bc1541b47df446297a369de05163012 [term 3 LEADER]: Becoming Leader. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:49.421938 28767 consensus_queue.cc:237] T 14a747ac14354fdab2a2993594d3398f P 4bc1541b47df446297a369de05163012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.424384 26510 raft_consensus.cc:1261] T d9dd99b54b7c4d19a80c229bf99d7976 P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Refusing update from remote peer 83c50029f7a94cfca6d1ee68e0728aa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:54:49.425042 28755 consensus_queue.cc:1035] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:49.425559 26640 raft_consensus.cc:2396] T 14a747ac14354fdab2a2993594d3398f P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bc1541b47df446297a369de05163012 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.425916 26509 raft_consensus.cc:504] T beff61f07ecf4ffab508da651a686c8e P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.426527 26509 leader_election.cc:288] T beff61f07ecf4ffab508da651a686c8e P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 3 election: Requested vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:49.426769 26379 raft_consensus.cc:2454] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 2.
I20240502 01:54:49.426985 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beff61f07ecf4ffab508da651a686c8e" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:49.427156 26378 raft_consensus.cc:3032] T beff61f07ecf4ffab508da651a686c8e P 5e9d825f803c40e089b64e319beccf6e [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:54:49.427248 26378 raft_consensus.cc:729] T beff61f07ecf4ffab508da651a686c8e P 5e9d825f803c40e089b64e319beccf6e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:49.427510 26378 consensus_queue.cc:260] T beff61f07ecf4ffab508da651a686c8e P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.427788 26378 raft_consensus.cc:3037] T beff61f07ecf4ffab508da651a686c8e P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.427830 26511 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9ead21d62e564a37a8c3d5a452621d13"
dest_uuid: "4bc1541b47df446297a369de05163012"
from {username='slave'} at 127.0.0.1:41462
I20240502 01:54:49.428082 26511 raft_consensus.cc:482] T 9ead21d62e564a37a8c3d5a452621d13 P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.428244 26511 raft_consensus.cc:3037] T 9ead21d62e564a37a8c3d5a452621d13 P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.430466 26378 raft_consensus.cc:2396] T beff61f07ecf4ffab508da651a686c8e P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bc1541b47df446297a369de05163012 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.431025 26511 raft_consensus.cc:504] T 9ead21d62e564a37a8c3d5a452621d13 P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.431627 26446 leader_election.cc:302] T 8cf6da25f535446b9515444e093cae7f P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:49.431710 26511 leader_election.cc:288] T 9ead21d62e564a37a8c3d5a452621d13 P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 3 election: Requested vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:49.432258 28773 raft_consensus.cc:2781] T 8cf6da25f535446b9515444e093cae7f P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:49.432400 28773 raft_consensus.cc:686] T 8cf6da25f535446b9515444e093cae7f P 4bc1541b47df446297a369de05163012 [term 2 LEADER]: Becoming Leader. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:49.432719 28773 consensus_queue.cc:237] T 8cf6da25f535446b9515444e093cae7f P 4bc1541b47df446297a369de05163012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.434389 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ead21d62e564a37a8c3d5a452621d13" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.434600 26643 raft_consensus.cc:3037] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.437220 26643 raft_consensus.cc:2454] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 3.
I20240502 01:54:49.437368 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ead21d62e564a37a8c3d5a452621d13" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:49.437547 26378 raft_consensus.cc:3032] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:54:49.437629 26378 raft_consensus.cc:729] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:49.437763 26446 leader_election.cc:302] T 9ead21d62e564a37a8c3d5a452621d13 P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:49.437959 26378 consensus_queue.cc:260] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.438102 28773 raft_consensus.cc:2781] T 9ead21d62e564a37a8c3d5a452621d13 P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:54:49.438251 26378 raft_consensus.cc:3037] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.438251 28773 raft_consensus.cc:686] T 9ead21d62e564a37a8c3d5a452621d13 P 4bc1541b47df446297a369de05163012 [term 3 LEADER]: Becoming Leader. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:49.438576 28773 consensus_queue.cc:237] T 9ead21d62e564a37a8c3d5a452621d13 P 4bc1541b47df446297a369de05163012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.432904 26510 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1c47a74a6dc54d5eb0b47818d816b7fd"
dest_uuid: "4bc1541b47df446297a369de05163012"
from {username='slave'} at 127.0.0.1:41462
I20240502 01:54:49.440060 26510 raft_consensus.cc:482] T 1c47a74a6dc54d5eb0b47818d816b7fd P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.440181 26510 raft_consensus.cc:3037] T 1c47a74a6dc54d5eb0b47818d816b7fd P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.442956 26510 raft_consensus.cc:504] T 1c47a74a6dc54d5eb0b47818d816b7fd P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.443099 26772 raft_consensus.cc:2454] T 8cf6da25f535446b9515444e093cae7f P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 2.
I20240502 01:54:49.443544 26510 leader_election.cc:288] T 1c47a74a6dc54d5eb0b47818d816b7fd P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 3 election: Requested vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:49.443992 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c47a74a6dc54d5eb0b47818d816b7fd" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.444255 26643 raft_consensus.cc:3032] T 1c47a74a6dc54d5eb0b47818d816b7fd P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:54:49.444389 26643 raft_consensus.cc:729] T 1c47a74a6dc54d5eb0b47818d816b7fd P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:49.444573 26773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c47a74a6dc54d5eb0b47818d816b7fd" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:49.444802 26378 raft_consensus.cc:2396] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bc1541b47df446297a369de05163012 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.444792 26773 raft_consensus.cc:3037] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.445151 26640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beff61f07ecf4ffab508da651a686c8e" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.445353 26640 raft_consensus.cc:3037] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.444690 26643 consensus_queue.cc:260] T 1c47a74a6dc54d5eb0b47818d816b7fd P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.445850 26643 raft_consensus.cc:3037] T 1c47a74a6dc54d5eb0b47818d816b7fd P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.447410 26773 raft_consensus.cc:2454] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 3.
I20240502 01:54:49.447815 26447 leader_election.cc:302] T 1c47a74a6dc54d5eb0b47818d816b7fd P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 4bc1541b47df446297a369de05163012; no voters:
I20240502 01:54:49.448004 26640 raft_consensus.cc:2454] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 3.
I20240502 01:54:49.448101 28770 raft_consensus.cc:2781] T 1c47a74a6dc54d5eb0b47818d816b7fd P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:54:49.448192 26510 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "245ca06c39774fb5b0f522550a63ac5b"
dest_uuid: "4bc1541b47df446297a369de05163012"
from {username='slave'} at 127.0.0.1:41462
I20240502 01:54:49.448247 28770 raft_consensus.cc:686] T 1c47a74a6dc54d5eb0b47818d816b7fd P 4bc1541b47df446297a369de05163012 [term 3 LEADER]: Becoming Leader. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:49.448357 26510 raft_consensus.cc:482] T 245ca06c39774fb5b0f522550a63ac5b P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.448449 26510 raft_consensus.cc:3037] T 245ca06c39774fb5b0f522550a63ac5b P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.448549 28770 consensus_queue.cc:237] T 1c47a74a6dc54d5eb0b47818d816b7fd P 4bc1541b47df446297a369de05163012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.450156 26446 leader_election.cc:302] T beff61f07ecf4ffab508da651a686c8e P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4bc1541b47df446297a369de05163012, 83c50029f7a94cfca6d1ee68e0728aa2; no voters: 5e9d825f803c40e089b64e319beccf6e
I20240502 01:54:49.450299 26510 raft_consensus.cc:504] T 245ca06c39774fb5b0f522550a63ac5b P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.450474 28765 raft_consensus.cc:2781] T beff61f07ecf4ffab508da651a686c8e P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:54:49.450621 28765 raft_consensus.cc:686] T beff61f07ecf4ffab508da651a686c8e P 4bc1541b47df446297a369de05163012 [term 3 LEADER]: Becoming Leader. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:49.450913 26510 leader_election.cc:288] T 245ca06c39774fb5b0f522550a63ac5b P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 2 election: Requested vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:49.450942 28765 consensus_queue.cc:237] T beff61f07ecf4ffab508da651a686c8e P 4bc1541b47df446297a369de05163012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.452066 26773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "245ca06c39774fb5b0f522550a63ac5b" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:49.452258 26773 raft_consensus.cc:3037] T 245ca06c39774fb5b0f522550a63ac5b P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.454931 26773 raft_consensus.cc:2454] T 245ca06c39774fb5b0f522550a63ac5b P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 2.
I20240502 01:54:49.455345 26447 leader_election.cc:302] T 245ca06c39774fb5b0f522550a63ac5b P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 4bc1541b47df446297a369de05163012; no voters:
I20240502 01:54:49.455684 26510 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d9dd99b54b7c4d19a80c229bf99d7976"
dest_uuid: "4bc1541b47df446297a369de05163012"
from {username='slave'} at 127.0.0.1:41462
I20240502 01:54:49.455724 28770 raft_consensus.cc:2781] T 245ca06c39774fb5b0f522550a63ac5b P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:49.456007 28770 raft_consensus.cc:686] T 245ca06c39774fb5b0f522550a63ac5b P 4bc1541b47df446297a369de05163012 [term 2 LEADER]: Becoming Leader. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:49.456271 28770 consensus_queue.cc:237] T 245ca06c39774fb5b0f522550a63ac5b P 4bc1541b47df446297a369de05163012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.456885 26510 raft_consensus.cc:482] T d9dd99b54b7c4d19a80c229bf99d7976 P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.457003 26510 raft_consensus.cc:3037] T d9dd99b54b7c4d19a80c229bf99d7976 P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.459252 26773 raft_consensus.cc:1261] T d9dd99b54b7c4d19a80c229bf99d7976 P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Refusing update from remote peer 83c50029f7a94cfca6d1ee68e0728aa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:54:49.459362 26510 raft_consensus.cc:504] T d9dd99b54b7c4d19a80c229bf99d7976 P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.459872 28028 consensus_queue.cc:1035] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:49.460233 26510 leader_election.cc:288] T d9dd99b54b7c4d19a80c229bf99d7976 P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 3 election: Requested vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:49.461839 26772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9dd99b54b7c4d19a80c229bf99d7976" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:49.462528 26508 raft_consensus.cc:1226] T d9dd99b54b7c4d19a80c229bf99d7976 P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Rejecting Update request from peer 83c50029f7a94cfca6d1ee68e0728aa2 for earlier term 2. Current term is 3. Ops: []
I20240502 01:54:49.463527 28028 consensus_queue.cc:1046] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:49.463726 26640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9dd99b54b7c4d19a80c229bf99d7976" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.463793 28028 raft_consensus.cc:3032] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:54:49.455293 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "245ca06c39774fb5b0f522550a63ac5b" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:49.463893 28028 raft_consensus.cc:729] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:49.463999 26378 raft_consensus.cc:3032] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:49.464107 26378 raft_consensus.cc:729] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:49.464154 28028 consensus_queue.cc:260] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.464360 26378 consensus_queue.cc:260] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.464515 28028 raft_consensus.cc:3037] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.464740 26378 raft_consensus.cc:3037] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.467506 26378 raft_consensus.cc:2454] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 2.
I20240502 01:54:49.469441 26640 raft_consensus.cc:2454] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 3.
I20240502 01:54:49.473989 26446 leader_election.cc:302] T d9dd99b54b7c4d19a80c229bf99d7976 P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4bc1541b47df446297a369de05163012, 83c50029f7a94cfca6d1ee68e0728aa2; no voters: 401942254c4c43a1ba02c4ae68d17320
I20240502 01:54:49.474303 28770 raft_consensus.cc:2781] T d9dd99b54b7c4d19a80c229bf99d7976 P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:54:49.474485 28770 raft_consensus.cc:686] T d9dd99b54b7c4d19a80c229bf99d7976 P 4bc1541b47df446297a369de05163012 [term 3 LEADER]: Becoming Leader. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:49.474823 28770 consensus_queue.cc:237] T d9dd99b54b7c4d19a80c229bf99d7976 P 4bc1541b47df446297a369de05163012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.476095 26643 raft_consensus.cc:2454] T 1c47a74a6dc54d5eb0b47818d816b7fd P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 3.
I20240502 01:54:49.475986 26228 catalog_manager.cc:5484] T 14a747ac14354fdab2a2993594d3398f P 4bc1541b47df446297a369de05163012 reported cstate change: term changed from 2 to 3, leader changed from 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67) to 4bc1541b47df446297a369de05163012 (127.11.42.66). New cstate: current_term: 3 leader_uuid: "4bc1541b47df446297a369de05163012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:49.481242 26510 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "23dcaac2d7d14adf820d826f05c1abc6"
dest_uuid: "4bc1541b47df446297a369de05163012"
from {username='slave'} at 127.0.0.1:41462
I20240502 01:54:49.481505 26510 raft_consensus.cc:482] T 23dcaac2d7d14adf820d826f05c1abc6 P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.481676 26510 raft_consensus.cc:3037] T 23dcaac2d7d14adf820d826f05c1abc6 P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.488760 26510 raft_consensus.cc:504] T 23dcaac2d7d14adf820d826f05c1abc6 P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.489918 26510 leader_election.cc:288] T 23dcaac2d7d14adf820d826f05c1abc6 P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 3 election: Requested vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:49.491143 26510 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b4ba7c4fe9614ab3abb3645c1da3fcd3"
dest_uuid: "4bc1541b47df446297a369de05163012"
from {username='slave'} at 127.0.0.1:41462
I20240502 01:54:49.491336 26510 raft_consensus.cc:482] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.491291 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23dcaac2d7d14adf820d826f05c1abc6" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.491446 26510 raft_consensus.cc:3037] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.491489 26643 raft_consensus.cc:3037] T 23dcaac2d7d14adf820d826f05c1abc6 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.492880 26228 catalog_manager.cc:5484] T 8cf6da25f535446b9515444e093cae7f P 4bc1541b47df446297a369de05163012 reported cstate change: term changed from 1 to 2, leader changed from 5e9d825f803c40e089b64e319beccf6e (127.11.42.65) to 4bc1541b47df446297a369de05163012 (127.11.42.66). New cstate: current_term: 2 leader_uuid: "4bc1541b47df446297a369de05163012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:49.493533 26228 catalog_manager.cc:5484] T 9ead21d62e564a37a8c3d5a452621d13 P 4bc1541b47df446297a369de05163012 reported cstate change: term changed from 2 to 3, leader changed from 5e9d825f803c40e089b64e319beccf6e (127.11.42.65) to 4bc1541b47df446297a369de05163012 (127.11.42.66). New cstate: current_term: 3 leader_uuid: "4bc1541b47df446297a369de05163012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.493992 26510 raft_consensus.cc:504] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.494109 26643 raft_consensus.cc:2454] T 23dcaac2d7d14adf820d826f05c1abc6 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 3.
I20240502 01:54:49.494010 26228 catalog_manager.cc:5484] T 245ca06c39774fb5b0f522550a63ac5b P 4bc1541b47df446297a369de05163012 reported cstate change: term changed from 1 to 2, leader changed from 5e9d825f803c40e089b64e319beccf6e (127.11.42.65) to 4bc1541b47df446297a369de05163012 (127.11.42.66). New cstate: current_term: 2 leader_uuid: "4bc1541b47df446297a369de05163012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:49.494740 26510 leader_election.cc:288] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 4 election: Requested vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:49.494923 26446 leader_election.cc:302] T 23dcaac2d7d14adf820d826f05c1abc6 P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:49.494987 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ba7c4fe9614ab3abb3645c1da3fcd3" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:49.495199 28773 raft_consensus.cc:2781] T 23dcaac2d7d14adf820d826f05c1abc6 P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:54:49.495241 26379 raft_consensus.cc:3032] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 5e9d825f803c40e089b64e319beccf6e [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:54:49.495326 28773 raft_consensus.cc:686] T 23dcaac2d7d14adf820d826f05c1abc6 P 4bc1541b47df446297a369de05163012 [term 3 LEADER]: Becoming Leader. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:49.495329 26379 raft_consensus.cc:729] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 5e9d825f803c40e089b64e319beccf6e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:49.495587 28773 consensus_queue.cc:237] T 23dcaac2d7d14adf820d826f05c1abc6 P 4bc1541b47df446297a369de05163012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.495612 26379 consensus_queue.cc:260] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.495764 26510 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d2287fdff0db43df90ff5580c0430b8c"
dest_uuid: "4bc1541b47df446297a369de05163012"
from {username='slave'} at 127.0.0.1:41462
I20240502 01:54:49.495910 26510 raft_consensus.cc:482] T d2287fdff0db43df90ff5580c0430b8c P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.495923 26379 raft_consensus.cc:3037] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.496021 26510 raft_consensus.cc:3037] T d2287fdff0db43df90ff5580c0430b8c P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.497241 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ba7c4fe9614ab3abb3645c1da3fcd3" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.497499 26643 raft_consensus.cc:3037] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.498494 26510 raft_consensus.cc:504] T d2287fdff0db43df90ff5580c0430b8c P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.498965 26379 raft_consensus.cc:2396] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bc1541b47df446297a369de05163012 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.499125 26510 leader_election.cc:288] T d2287fdff0db43df90ff5580c0430b8c P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 4 election: Requested vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:49.499620 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2287fdff0db43df90ff5580c0430b8c" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:49.499836 26379 raft_consensus.cc:3032] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:54:49.499807 26640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2287fdff0db43df90ff5580c0430b8c" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.499913 26379 raft_consensus.cc:729] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:49.500013 26640 raft_consensus.cc:3037] T d2287fdff0db43df90ff5580c0430b8c P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.500174 26379 consensus_queue.cc:260] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.500474 26510 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d4b8975c68d64c2d804fcc852384afde"
dest_uuid: "4bc1541b47df446297a369de05163012"
from {username='slave'} at 127.0.0.1:41462
I20240502 01:54:49.500511 26379 raft_consensus.cc:3037] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.500684 26510 raft_consensus.cc:482] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.500814 26510 raft_consensus.cc:3037] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.502451 26228 catalog_manager.cc:5484] T beff61f07ecf4ffab508da651a686c8e P 4bc1541b47df446297a369de05163012 reported cstate change: term changed from 2 to 3, leader changed from 5e9d825f803c40e089b64e319beccf6e (127.11.42.65) to 4bc1541b47df446297a369de05163012 (127.11.42.66). New cstate: current_term: 3 leader_uuid: "4bc1541b47df446297a369de05163012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.502839 26640 raft_consensus.cc:2396] T d2287fdff0db43df90ff5580c0430b8c P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bc1541b47df446297a369de05163012 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.503067 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23dcaac2d7d14adf820d826f05c1abc6" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:49.503294 26510 raft_consensus.cc:504] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.503607 26378 raft_consensus.cc:3032] T 23dcaac2d7d14adf820d826f05c1abc6 P 5e9d825f803c40e089b64e319beccf6e [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:54:49.504107 26510 leader_election.cc:288] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 3 election: Requested vote from peers 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:49.504304 26773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4b8975c68d64c2d804fcc852384afde" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:49.504537 26773 raft_consensus.cc:3037] T d4b8975c68d64c2d804fcc852384afde P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.504500 26640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4b8975c68d64c2d804fcc852384afde" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.504684 26640 raft_consensus.cc:3032] T d4b8975c68d64c2d804fcc852384afde P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:54:49.504765 26640 raft_consensus.cc:729] T d4b8975c68d64c2d804fcc852384afde P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:49.504999 26508 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4f9ed0162eb74c4ea422f122da01dce3"
dest_uuid: "4bc1541b47df446297a369de05163012"
from {username='slave'} at 127.0.0.1:41462
I20240502 01:54:49.505204 26508 raft_consensus.cc:482] T 4f9ed0162eb74c4ea422f122da01dce3 P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.505131 26640 consensus_queue.cc:260] T d4b8975c68d64c2d804fcc852384afde P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.505288 26508 raft_consensus.cc:3037] T 4f9ed0162eb74c4ea422f122da01dce3 P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.505568 26640 raft_consensus.cc:3037] T d4b8975c68d64c2d804fcc852384afde P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.507517 26773 raft_consensus.cc:2454] T d4b8975c68d64c2d804fcc852384afde P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 3.
I20240502 01:54:49.507848 26508 raft_consensus.cc:504] T 4f9ed0162eb74c4ea422f122da01dce3 P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.508261 26447 leader_election.cc:302] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 4bc1541b47df446297a369de05163012; no voters:
I20240502 01:54:49.508322 26640 raft_consensus.cc:2454] T d4b8975c68d64c2d804fcc852384afde P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 3.
I20240502 01:54:49.508502 26508 leader_election.cc:288] T 4f9ed0162eb74c4ea422f122da01dce3 P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 3 election: Requested vote from peers 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959), 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085)
I20240502 01:54:49.508793 28765 raft_consensus.cc:2781] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:54:49.508977 28765 raft_consensus.cc:686] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012 [term 3 LEADER]: Becoming Leader. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:49.509245 28765 consensus_queue.cc:237] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.509898 26643 raft_consensus.cc:2396] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bc1541b47df446297a369de05163012 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.510761 26446 leader_election.cc:302] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bc1541b47df446297a369de05163012; no voters: 5e9d825f803c40e089b64e319beccf6e, 83c50029f7a94cfca6d1ee68e0728aa2
I20240502 01:54:49.511202 28765 raft_consensus.cc:2726] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 01:54:49.511262 26773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9ed0162eb74c4ea422f122da01dce3" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:49.511499 26773 raft_consensus.cc:3037] T 4f9ed0162eb74c4ea422f122da01dce3 P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.514258 26773 raft_consensus.cc:2454] T 4f9ed0162eb74c4ea422f122da01dce3 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 3.
I20240502 01:54:49.509050 26377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9ed0162eb74c4ea422f122da01dce3" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:49.514673 26447 leader_election.cc:302] T 4f9ed0162eb74c4ea422f122da01dce3 P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 4bc1541b47df446297a369de05163012; no voters:
I20240502 01:54:49.514724 26377 raft_consensus.cc:3032] T 4f9ed0162eb74c4ea422f122da01dce3 P 5e9d825f803c40e089b64e319beccf6e [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:54:49.514816 26377 raft_consensus.cc:729] T 4f9ed0162eb74c4ea422f122da01dce3 P 5e9d825f803c40e089b64e319beccf6e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:49.515038 28765 raft_consensus.cc:2781] T 4f9ed0162eb74c4ea422f122da01dce3 P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:54:49.515082 26377 consensus_queue.cc:260] T 4f9ed0162eb74c4ea422f122da01dce3 P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.515326 28765 raft_consensus.cc:686] T 4f9ed0162eb74c4ea422f122da01dce3 P 4bc1541b47df446297a369de05163012 [term 3 LEADER]: Becoming Leader. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:49.515499 26377 raft_consensus.cc:3037] T 4f9ed0162eb74c4ea422f122da01dce3 P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.515576 28765 consensus_queue.cc:237] T 4f9ed0162eb74c4ea422f122da01dce3 P 4bc1541b47df446297a369de05163012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.518318 26377 raft_consensus.cc:2396] T 4f9ed0162eb74c4ea422f122da01dce3 P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bc1541b47df446297a369de05163012 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.518644 26378 raft_consensus.cc:729] T 23dcaac2d7d14adf820d826f05c1abc6 P 5e9d825f803c40e089b64e319beccf6e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:49.518889 26378 consensus_queue.cc:260] T 23dcaac2d7d14adf820d826f05c1abc6 P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.519169 26378 raft_consensus.cc:3037] T 23dcaac2d7d14adf820d826f05c1abc6 P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.521519 26508 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "37e818dc0b834eb185e27fcb3cb972e1"
dest_uuid: "4bc1541b47df446297a369de05163012"
from {username='slave'} at 127.0.0.1:41462
I20240502 01:54:49.503840 26228 catalog_manager.cc:5484] T 1c47a74a6dc54d5eb0b47818d816b7fd P 4bc1541b47df446297a369de05163012 reported cstate change: term changed from 2 to 3, leader changed from 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67) to 4bc1541b47df446297a369de05163012 (127.11.42.66). New cstate: current_term: 3 leader_uuid: "4bc1541b47df446297a369de05163012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:49.521968 26379 raft_consensus.cc:2396] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bc1541b47df446297a369de05163012 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.522194 26508 raft_consensus.cc:482] T 37e818dc0b834eb185e27fcb3cb972e1 P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.522428 26228 catalog_manager.cc:5484] T d9dd99b54b7c4d19a80c229bf99d7976 P 4bc1541b47df446297a369de05163012 reported cstate change: term changed from 2 to 3, leader changed from 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67) to 4bc1541b47df446297a369de05163012 (127.11.42.66). New cstate: current_term: 3 leader_uuid: "4bc1541b47df446297a369de05163012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.522413 26446 leader_election.cc:302] T d2287fdff0db43df90ff5580c0430b8c P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bc1541b47df446297a369de05163012; no voters: 5e9d825f803c40e089b64e319beccf6e, 83c50029f7a94cfca6d1ee68e0728aa2
I20240502 01:54:49.522931 26508 raft_consensus.cc:3037] T 37e818dc0b834eb185e27fcb3cb972e1 P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.523196 28765 raft_consensus.cc:2726] T d2287fdff0db43df90ff5580c0430b8c P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 01:54:49.524947 26378 raft_consensus.cc:2396] T 23dcaac2d7d14adf820d826f05c1abc6 P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bc1541b47df446297a369de05163012 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.527256 26508 raft_consensus.cc:504] T 37e818dc0b834eb185e27fcb3cb972e1 P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.528419 26773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e818dc0b834eb185e27fcb3cb972e1" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:49.528635 26773 raft_consensus.cc:3037] T 37e818dc0b834eb185e27fcb3cb972e1 P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.528769 26508 leader_election.cc:288] T 37e818dc0b834eb185e27fcb3cb972e1 P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 2 election: Requested vote from peers 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959), 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085)
I20240502 01:54:49.529218 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e818dc0b834eb185e27fcb3cb972e1" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:49.529464 26379 raft_consensus.cc:3032] T 37e818dc0b834eb185e27fcb3cb972e1 P 5e9d825f803c40e089b64e319beccf6e [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:49.529579 26379 raft_consensus.cc:729] T 37e818dc0b834eb185e27fcb3cb972e1 P 5e9d825f803c40e089b64e319beccf6e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:49.529837 26379 consensus_queue.cc:260] T 37e818dc0b834eb185e27fcb3cb972e1 P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.530171 26379 raft_consensus.cc:3037] T 37e818dc0b834eb185e27fcb3cb972e1 P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.530860 26508 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dc82c30359e84bddb4bcfe1b3b13ced7"
dest_uuid: "4bc1541b47df446297a369de05163012"
from {username='slave'} at 127.0.0.1:41462
I20240502 01:54:49.530941 26773 raft_consensus.cc:2454] T 37e818dc0b834eb185e27fcb3cb972e1 P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 2.
I20240502 01:54:49.531035 26508 raft_consensus.cc:482] T dc82c30359e84bddb4bcfe1b3b13ced7 P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.531134 26508 raft_consensus.cc:3037] T dc82c30359e84bddb4bcfe1b3b13ced7 P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.531322 26447 leader_election.cc:302] T 37e818dc0b834eb185e27fcb3cb972e1 P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 4bc1541b47df446297a369de05163012; no voters:
I20240502 01:54:49.532025 28773 raft_consensus.cc:2781] T 37e818dc0b834eb185e27fcb3cb972e1 P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:49.532172 28773 raft_consensus.cc:686] T 37e818dc0b834eb185e27fcb3cb972e1 P 4bc1541b47df446297a369de05163012 [term 2 LEADER]: Becoming Leader. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:49.532426 28773 consensus_queue.cc:237] T 37e818dc0b834eb185e27fcb3cb972e1 P 4bc1541b47df446297a369de05163012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.532928 26379 raft_consensus.cc:2454] T 37e818dc0b834eb185e27fcb3cb972e1 P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 2.
I20240502 01:54:49.533634 26508 raft_consensus.cc:504] T dc82c30359e84bddb4bcfe1b3b13ced7 P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.534658 26772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc82c30359e84bddb4bcfe1b3b13ced7" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:49.534684 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc82c30359e84bddb4bcfe1b3b13ced7" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.534843 26643 raft_consensus.cc:3032] T dc82c30359e84bddb4bcfe1b3b13ced7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:49.534875 26772 raft_consensus.cc:3037] T dc82c30359e84bddb4bcfe1b3b13ced7 P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.534955 26643 raft_consensus.cc:729] T dc82c30359e84bddb4bcfe1b3b13ced7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:49.535213 26643 consensus_queue.cc:260] T dc82c30359e84bddb4bcfe1b3b13ced7 P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.535584 26643 raft_consensus.cc:3037] T dc82c30359e84bddb4bcfe1b3b13ced7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.537571 26772 raft_consensus.cc:2454] T dc82c30359e84bddb4bcfe1b3b13ced7 P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 2.
I20240502 01:54:49.538115 26643 raft_consensus.cc:2454] T dc82c30359e84bddb4bcfe1b3b13ced7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 2.
I20240502 01:54:49.538529 26447 leader_election.cc:302] T dc82c30359e84bddb4bcfe1b3b13ced7 P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 4bc1541b47df446297a369de05163012; no voters:
I20240502 01:54:49.538810 26508 leader_election.cc:288] T dc82c30359e84bddb4bcfe1b3b13ced7 P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 2 election: Requested vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:49.538770 28765 raft_consensus.cc:2781] T dc82c30359e84bddb4bcfe1b3b13ced7 P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:49.539047 28765 raft_consensus.cc:686] T dc82c30359e84bddb4bcfe1b3b13ced7 P 4bc1541b47df446297a369de05163012 [term 2 LEADER]: Becoming Leader. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:49.539319 28765 consensus_queue.cc:237] T dc82c30359e84bddb4bcfe1b3b13ced7 P 4bc1541b47df446297a369de05163012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.540153 26510 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "99a4b7d022e1461a8233952ab47ef5d3"
dest_uuid: "4bc1541b47df446297a369de05163012"
from {username='slave'} at 127.0.0.1:41462
I20240502 01:54:49.540387 26510 raft_consensus.cc:482] T 99a4b7d022e1461a8233952ab47ef5d3 P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.540575 26510 raft_consensus.cc:3037] T 99a4b7d022e1461a8233952ab47ef5d3 P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.543406 26510 raft_consensus.cc:504] T 99a4b7d022e1461a8233952ab47ef5d3 P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.544135 26510 leader_election.cc:288] T 99a4b7d022e1461a8233952ab47ef5d3 P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 4 election: Requested vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085)
I20240502 01:54:49.544745 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a4b7d022e1461a8233952ab47ef5d3" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.544940 26643 raft_consensus.cc:3037] T 99a4b7d022e1461a8233952ab47ef5d3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.545085 26511 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "89107a545fcc48aead1e199da1f78311"
dest_uuid: "4bc1541b47df446297a369de05163012"
from {username='slave'} at 127.0.0.1:41462
I20240502 01:54:49.545281 26511 raft_consensus.cc:482] T 89107a545fcc48aead1e199da1f78311 P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.545390 26511 raft_consensus.cc:3037] T 89107a545fcc48aead1e199da1f78311 P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.547451 26509 raft_consensus.cc:1261] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Refusing update from remote peer 83c50029f7a94cfca6d1ee68e0728aa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:54:49.547706 26643 raft_consensus.cc:2396] T 99a4b7d022e1461a8233952ab47ef5d3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bc1541b47df446297a369de05163012 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.547925 26511 raft_consensus.cc:504] T 89107a545fcc48aead1e199da1f78311 P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.548061 28028 consensus_queue.cc:1035] T 22d4681165d1488db2c45ec83cb49cb4 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:49.548089 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a4b7d022e1461a8233952ab47ef5d3" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:49.548266 26378 raft_consensus.cc:3032] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:54:49.548352 26378 raft_consensus.cc:729] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:49.548480 26511 leader_election.cc:288] T 89107a545fcc48aead1e199da1f78311 P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 2 election: Requested vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:49.548619 26378 consensus_queue.cc:260] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.548918 26378 raft_consensus.cc:3037] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.549906 26509 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2fd06590221d414b9e896e02b1be6f3e"
dest_uuid: "4bc1541b47df446297a369de05163012"
from {username='slave'} at 127.0.0.1:41462
I20240502 01:54:49.550086 26509 raft_consensus.cc:482] T 2fd06590221d414b9e896e02b1be6f3e P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.550182 26509 raft_consensus.cc:3037] T 2fd06590221d414b9e896e02b1be6f3e P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.550174 26773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89107a545fcc48aead1e199da1f78311" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:49.550369 26773 raft_consensus.cc:3037] T 89107a545fcc48aead1e199da1f78311 P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.551776 26378 raft_consensus.cc:2396] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bc1541b47df446297a369de05163012 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.552184 26446 leader_election.cc:302] T 99a4b7d022e1461a8233952ab47ef5d3 P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bc1541b47df446297a369de05163012; no voters: 5e9d825f803c40e089b64e319beccf6e, 83c50029f7a94cfca6d1ee68e0728aa2
I20240502 01:54:49.552664 28767 raft_consensus.cc:2726] T 99a4b7d022e1461a8233952ab47ef5d3 P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 01:54:49.552707 26509 raft_consensus.cc:504] T 2fd06590221d414b9e896e02b1be6f3e P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.552855 26773 raft_consensus.cc:2454] T 89107a545fcc48aead1e199da1f78311 P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 2.
I20240502 01:54:49.553237 26447 leader_election.cc:302] T 89107a545fcc48aead1e199da1f78311 P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 4bc1541b47df446297a369de05163012; no voters:
I20240502 01:54:49.553539 28767 raft_consensus.cc:2781] T 89107a545fcc48aead1e199da1f78311 P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:49.553669 28767 raft_consensus.cc:686] T 89107a545fcc48aead1e199da1f78311 P 4bc1541b47df446297a369de05163012 [term 2 LEADER]: Becoming Leader. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:49.553910 28767 consensus_queue.cc:237] T 89107a545fcc48aead1e199da1f78311 P 4bc1541b47df446297a369de05163012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.555486 26509 leader_election.cc:288] T 2fd06590221d414b9e896e02b1be6f3e P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 4 election: Requested vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:49.555912 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89107a545fcc48aead1e199da1f78311" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.556174 26643 raft_consensus.cc:3032] T 89107a545fcc48aead1e199da1f78311 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:49.556308 26643 raft_consensus.cc:729] T 89107a545fcc48aead1e199da1f78311 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:49.556593 26509 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1ce4dd22bb314eb7855126e3fe1785f6"
dest_uuid: "4bc1541b47df446297a369de05163012"
from {username='slave'} at 127.0.0.1:41462
I20240502 01:54:49.556762 26509 raft_consensus.cc:482] T 1ce4dd22bb314eb7855126e3fe1785f6 P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.556870 26509 raft_consensus.cc:3037] T 1ce4dd22bb314eb7855126e3fe1785f6 P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.556655 26643 consensus_queue.cc:260] T 89107a545fcc48aead1e199da1f78311 P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.557312 26643 raft_consensus.cc:3037] T 89107a545fcc48aead1e199da1f78311 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.559549 26640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fd06590221d414b9e896e02b1be6f3e" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.559788 26640 raft_consensus.cc:3037] T 2fd06590221d414b9e896e02b1be6f3e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.560675 26509 raft_consensus.cc:504] T 1ce4dd22bb314eb7855126e3fe1785f6 P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.561795 26772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ce4dd22bb314eb7855126e3fe1785f6" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:49.562036 26772 raft_consensus.cc:3037] T 1ce4dd22bb314eb7855126e3fe1785f6 P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.563069 26509 leader_election.cc:288] T 1ce4dd22bb314eb7855126e3fe1785f6 P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 3 election: Requested vote from peers 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959), 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085)
I20240502 01:54:49.563966 26507 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b5ab243d9f1c4b568d38ed03027adb44"
dest_uuid: "4bc1541b47df446297a369de05163012"
from {username='slave'} at 127.0.0.1:41462
I20240502 01:54:49.564168 26507 raft_consensus.cc:482] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.564275 26507 raft_consensus.cc:3037] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.564713 26772 raft_consensus.cc:2454] T 1ce4dd22bb314eb7855126e3fe1785f6 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 3.
I20240502 01:54:49.565333 26447 leader_election.cc:302] T 1ce4dd22bb314eb7855126e3fe1785f6 P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 4bc1541b47df446297a369de05163012; no voters:
I20240502 01:54:49.565765 28765 raft_consensus.cc:2781] T 1ce4dd22bb314eb7855126e3fe1785f6 P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:54:49.566040 28765 raft_consensus.cc:686] T 1ce4dd22bb314eb7855126e3fe1785f6 P 4bc1541b47df446297a369de05163012 [term 3 LEADER]: Becoming Leader. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:49.566370 28765 consensus_queue.cc:237] T 1ce4dd22bb314eb7855126e3fe1785f6 P 4bc1541b47df446297a369de05163012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.567003 26507 raft_consensus.cc:504] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.567667 26507 leader_election.cc:288] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 3 election: Requested vote from peers 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959), 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085)
I20240502 01:54:49.568279 26377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ab243d9f1c4b568d38ed03027adb44" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:49.568547 26640 raft_consensus.cc:2396] T 2fd06590221d414b9e896e02b1be6f3e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bc1541b47df446297a369de05163012 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.568543 26377 raft_consensus.cc:3032] T b5ab243d9f1c4b568d38ed03027adb44 P 5e9d825f803c40e089b64e319beccf6e [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:54:49.568624 26377 raft_consensus.cc:729] T b5ab243d9f1c4b568d38ed03027adb44 P 5e9d825f803c40e089b64e319beccf6e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:49.568897 26507 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c9e4788d50d04d129e8d03125073c6f2"
dest_uuid: "4bc1541b47df446297a369de05163012"
from {username='slave'} at 127.0.0.1:41462
I20240502 01:54:49.568924 26377 consensus_queue.cc:260] T b5ab243d9f1c4b568d38ed03027adb44 P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.569109 26507 raft_consensus.cc:482] T c9e4788d50d04d129e8d03125073c6f2 P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.569216 26507 raft_consensus.cc:3037] T c9e4788d50d04d129e8d03125073c6f2 P 4bc1541b47df446297a369de05163012 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.569218 26377 raft_consensus.cc:3037] T b5ab243d9f1c4b568d38ed03027adb44 P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.569346 26643 raft_consensus.cc:2454] T 89107a545fcc48aead1e199da1f78311 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 2.
I20240502 01:54:49.571028 26773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ab243d9f1c4b568d38ed03027adb44" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:49.571512 26773 raft_consensus.cc:3037] T b5ab243d9f1c4b568d38ed03027adb44 P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.571761 26507 raft_consensus.cc:504] T c9e4788d50d04d129e8d03125073c6f2 P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.572050 26377 raft_consensus.cc:2396] T b5ab243d9f1c4b568d38ed03027adb44 P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bc1541b47df446297a369de05163012 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.572703 26507 leader_election.cc:288] T c9e4788d50d04d129e8d03125073c6f2 P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 2 election: Requested vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:49.572665 26376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e4788d50d04d129e8d03125073c6f2" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:49.572876 26376 raft_consensus.cc:3032] T c9e4788d50d04d129e8d03125073c6f2 P 5e9d825f803c40e089b64e319beccf6e [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:49.572964 26376 raft_consensus.cc:729] T c9e4788d50d04d129e8d03125073c6f2 P 5e9d825f803c40e089b64e319beccf6e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:49.573154 26772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e4788d50d04d129e8d03125073c6f2" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:49.573246 26376 consensus_queue.cc:260] T c9e4788d50d04d129e8d03125073c6f2 P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.573501 26772 raft_consensus.cc:3037] T c9e4788d50d04d129e8d03125073c6f2 P 401942254c4c43a1ba02c4ae68d17320 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.573568 26508 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "22d4681165d1488db2c45ec83cb49cb4"
dest_uuid: "4bc1541b47df446297a369de05163012"
from {username='slave'} at 127.0.0.1:41462
I20240502 01:54:49.573647 26376 raft_consensus.cc:3037] T c9e4788d50d04d129e8d03125073c6f2 P 5e9d825f803c40e089b64e319beccf6e [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:49.576378 26376 raft_consensus.cc:2454] T c9e4788d50d04d129e8d03125073c6f2 P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 2.
I20240502 01:54:49.576722 26772 raft_consensus.cc:2454] T c9e4788d50d04d129e8d03125073c6f2 P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 2.
I20240502 01:54:49.576897 26446 leader_election.cc:302] T c9e4788d50d04d129e8d03125073c6f2 P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:49.577246 28770 raft_consensus.cc:2781] T c9e4788d50d04d129e8d03125073c6f2 P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:49.577412 28770 raft_consensus.cc:686] T c9e4788d50d04d129e8d03125073c6f2 P 4bc1541b47df446297a369de05163012 [term 2 LEADER]: Becoming Leader. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:49.577705 28770 consensus_queue.cc:237] T c9e4788d50d04d129e8d03125073c6f2 P 4bc1541b47df446297a369de05163012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.580964 26773 raft_consensus.cc:2454] T b5ab243d9f1c4b568d38ed03027adb44 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 3.
I20240502 01:54:49.582522 26447 leader_election.cc:302] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 4bc1541b47df446297a369de05163012; no voters: 5e9d825f803c40e089b64e319beccf6e
I20240502 01:54:49.582984 28765 raft_consensus.cc:2781] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:54:49.583290 28765 raft_consensus.cc:686] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012 [term 3 LEADER]: Becoming Leader. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:49.583595 28765 consensus_queue.cc:237] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.584717 26508 raft_consensus.cc:482] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.584861 26508 raft_consensus.cc:3037] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.585173 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ce4dd22bb314eb7855126e3fe1785f6" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:49.585382 26379 raft_consensus.cc:3032] T 1ce4dd22bb314eb7855126e3fe1785f6 P 5e9d825f803c40e089b64e319beccf6e [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:54:49.585458 26379 raft_consensus.cc:729] T 1ce4dd22bb314eb7855126e3fe1785f6 P 5e9d825f803c40e089b64e319beccf6e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:49.585695 26379 consensus_queue.cc:260] T 1ce4dd22bb314eb7855126e3fe1785f6 P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.585958 26379 raft_consensus.cc:3037] T 1ce4dd22bb314eb7855126e3fe1785f6 P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.587623 26508 raft_consensus.cc:504] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.588181 26511 raft_consensus.cc:1226] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Rejecting Update request from peer 83c50029f7a94cfca6d1ee68e0728aa2 for earlier term 2. Current term is 3. Ops: []
I20240502 01:54:49.588368 26379 raft_consensus.cc:2396] T 1ce4dd22bb314eb7855126e3fe1785f6 P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bc1541b47df446297a369de05163012 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.588707 28755 consensus_queue.cc:1046] T 22d4681165d1488db2c45ec83cb49cb4 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:49.588953 28755 raft_consensus.cc:3032] T 22d4681165d1488db2c45ec83cb49cb4 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:54:49.589083 28755 raft_consensus.cc:729] T 22d4681165d1488db2c45ec83cb49cb4 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:49.589344 26508 leader_election.cc:288] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 3 election: Requested vote from peers 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:49.589315 28755 consensus_queue.cc:260] T 22d4681165d1488db2c45ec83cb49cb4 P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.589632 28755 raft_consensus.cc:3037] T 22d4681165d1488db2c45ec83cb49cb4 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.590000 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fd06590221d414b9e896e02b1be6f3e" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:49.590170 26378 raft_consensus.cc:3032] T 2fd06590221d414b9e896e02b1be6f3e P 5e9d825f803c40e089b64e319beccf6e [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:54:49.590276 26378 raft_consensus.cc:729] T 2fd06590221d414b9e896e02b1be6f3e P 5e9d825f803c40e089b64e319beccf6e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:49.590180 26772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22d4681165d1488db2c45ec83cb49cb4" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:49.590498 26772 raft_consensus.cc:3037] T 22d4681165d1488db2c45ec83cb49cb4 P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.590515 26378 consensus_queue.cc:260] T 2fd06590221d414b9e896e02b1be6f3e P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.590806 26378 raft_consensus.cc:3037] T 2fd06590221d414b9e896e02b1be6f3e P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.590837 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22d4681165d1488db2c45ec83cb49cb4" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.592284 26234 catalog_manager.cc:5484] T 37e818dc0b834eb185e27fcb3cb972e1 P 4bc1541b47df446297a369de05163012 reported cstate change: term changed from 1 to 2, leader changed from 5e9d825f803c40e089b64e319beccf6e (127.11.42.65) to 4bc1541b47df446297a369de05163012 (127.11.42.66). New cstate: current_term: 2 leader_uuid: "4bc1541b47df446297a369de05163012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.592917 26234 catalog_manager.cc:5484] T 23dcaac2d7d14adf820d826f05c1abc6 P 4bc1541b47df446297a369de05163012 reported cstate change: term changed from 2 to 3, leader changed from 5e9d825f803c40e089b64e319beccf6e (127.11.42.65) to 4bc1541b47df446297a369de05163012 (127.11.42.66). New cstate: current_term: 3 leader_uuid: "4bc1541b47df446297a369de05163012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.593423 26234 catalog_manager.cc:5484] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012 reported cstate change: term changed from 2 to 3, leader changed from 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67) to 4bc1541b47df446297a369de05163012 (127.11.42.66). New cstate: current_term: 3 leader_uuid: "4bc1541b47df446297a369de05163012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.593849 26234 catalog_manager.cc:5484] T dc82c30359e84bddb4bcfe1b3b13ced7 P 4bc1541b47df446297a369de05163012 reported cstate change: term changed from 1 to 2, leader changed from 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67) to 4bc1541b47df446297a369de05163012 (127.11.42.66). New cstate: current_term: 2 leader_uuid: "4bc1541b47df446297a369de05163012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.594259 26234 catalog_manager.cc:5484] T 4f9ed0162eb74c4ea422f122da01dce3 P 4bc1541b47df446297a369de05163012 reported cstate change: term changed from 2 to 3, leader changed from 5e9d825f803c40e089b64e319beccf6e (127.11.42.65) to 4bc1541b47df446297a369de05163012 (127.11.42.66). New cstate: current_term: 3 leader_uuid: "4bc1541b47df446297a369de05163012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.594982 26772 raft_consensus.cc:2454] T 22d4681165d1488db2c45ec83cb49cb4 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 3.
I20240502 01:54:49.595373 26447 leader_election.cc:302] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 4bc1541b47df446297a369de05163012; no voters:
I20240502 01:54:49.595639 28773 raft_consensus.cc:2781] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:54:49.595772 28773 raft_consensus.cc:686] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012 [term 3 LEADER]: Becoming Leader. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:49.596024 28773 consensus_queue.cc:237] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.603425 26378 raft_consensus.cc:2396] T 2fd06590221d414b9e896e02b1be6f3e P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bc1541b47df446297a369de05163012 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.603843 26446 leader_election.cc:302] T 2fd06590221d414b9e896e02b1be6f3e P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bc1541b47df446297a369de05163012; no voters: 5e9d825f803c40e089b64e319beccf6e, 83c50029f7a94cfca6d1ee68e0728aa2
I20240502 01:54:49.604440 28773 raft_consensus.cc:2726] T 2fd06590221d414b9e896e02b1be6f3e P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 01:54:49.605568 26643 raft_consensus.cc:2454] T 22d4681165d1488db2c45ec83cb49cb4 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 3.
I20240502 01:54:49.609717 26228 catalog_manager.cc:5484] T c9e4788d50d04d129e8d03125073c6f2 P 4bc1541b47df446297a369de05163012 reported cstate change: term changed from 1 to 2, leader changed from 5e9d825f803c40e089b64e319beccf6e (127.11.42.65) to 4bc1541b47df446297a369de05163012 (127.11.42.66). New cstate: current_term: 2 leader_uuid: "4bc1541b47df446297a369de05163012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.610260 26773 raft_consensus.cc:1226] T 22d4681165d1488db2c45ec83cb49cb4 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Rejecting Update request from peer 83c50029f7a94cfca6d1ee68e0728aa2 for earlier term 2. Current term is 3. Ops: []
I20240502 01:54:49.611148 26228 catalog_manager.cc:5484] T 89107a545fcc48aead1e199da1f78311 P 4bc1541b47df446297a369de05163012 reported cstate change: term changed from 1 to 2, leader changed from 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67) to 4bc1541b47df446297a369de05163012 (127.11.42.66). New cstate: current_term: 2 leader_uuid: "4bc1541b47df446297a369de05163012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:49.611853 26228 catalog_manager.cc:5484] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012 reported cstate change: term changed from 2 to 3, leader changed from 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67) to 4bc1541b47df446297a369de05163012 (127.11.42.66). New cstate: current_term: 3 leader_uuid: "4bc1541b47df446297a369de05163012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.613612 26228 catalog_manager.cc:5484] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012 reported cstate change: term changed from 2 to 3, leader changed from 5e9d825f803c40e089b64e319beccf6e (127.11.42.65) to 4bc1541b47df446297a369de05163012 (127.11.42.66). New cstate: current_term: 3 leader_uuid: "4bc1541b47df446297a369de05163012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.614226 26228 catalog_manager.cc:5484] T 1ce4dd22bb314eb7855126e3fe1785f6 P 4bc1541b47df446297a369de05163012 reported cstate change: term changed from 2 to 3, leader changed from 5e9d825f803c40e089b64e319beccf6e (127.11.42.65) to 4bc1541b47df446297a369de05163012 (127.11.42.66). New cstate: current_term: 3 leader_uuid: "4bc1541b47df446297a369de05163012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.631870 26773 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8cf6da25f535446b9515444e093cae7f"
dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
from {username='slave'} at 127.0.0.1:43296
I20240502 01:54:49.632107 26773 raft_consensus.cc:482] T 8cf6da25f535446b9515444e093cae7f P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.632249 26773 raft_consensus.cc:3037] T 8cf6da25f535446b9515444e093cae7f P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.634447 26773 raft_consensus.cc:504] T 8cf6da25f535446b9515444e093cae7f P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.635052 26773 leader_election.cc:288] T 8cf6da25f535446b9515444e093cae7f P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 3 election: Requested vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:49.635514 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cf6da25f535446b9515444e093cae7f" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:49.635550 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cf6da25f535446b9515444e093cae7f" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:49.635718 26378 raft_consensus.cc:3037] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.635749 26511 raft_consensus.cc:3032] T 8cf6da25f535446b9515444e093cae7f P 4bc1541b47df446297a369de05163012 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:54:49.635843 26511 raft_consensus.cc:729] T 8cf6da25f535446b9515444e093cae7f P 4bc1541b47df446297a369de05163012 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:49.636132 26511 consensus_queue.cc:260] T 8cf6da25f535446b9515444e093cae7f P 4bc1541b47df446297a369de05163012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.636473 26511 raft_consensus.cc:3037] T 8cf6da25f535446b9515444e093cae7f P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.637742 26378 raft_consensus.cc:2454] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 401942254c4c43a1ba02c4ae68d17320 in term 3.
I20240502 01:54:49.638170 26708 leader_election.cc:302] T 8cf6da25f535446b9515444e093cae7f P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:49.638460 27670 raft_consensus.cc:2781] T 8cf6da25f535446b9515444e093cae7f P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:54:49.638780 26511 raft_consensus.cc:2396] T 8cf6da25f535446b9515444e093cae7f P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 401942254c4c43a1ba02c4ae68d17320 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.638911 27670 raft_consensus.cc:686] T 8cf6da25f535446b9515444e093cae7f P 401942254c4c43a1ba02c4ae68d17320 [term 3 LEADER]: Becoming Leader. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Running, Role: LEADER
I20240502 01:54:49.639219 27670 consensus_queue.cc:237] T 8cf6da25f535446b9515444e093cae7f P 401942254c4c43a1ba02c4ae68d17320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.640049 26773 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ef474f44fe5b4a06b2c8dda12b1bc496"
dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
from {username='slave'} at 127.0.0.1:43296
I20240502 01:54:49.640241 26773 raft_consensus.cc:482] T ef474f44fe5b4a06b2c8dda12b1bc496 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.640367 26773 raft_consensus.cc:3037] T ef474f44fe5b4a06b2c8dda12b1bc496 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.643503 26773 raft_consensus.cc:504] T ef474f44fe5b4a06b2c8dda12b1bc496 P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.644165 26773 leader_election.cc:288] T ef474f44fe5b4a06b2c8dda12b1bc496 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 4 election: Requested vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085)
I20240502 01:54:49.644703 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef474f44fe5b4a06b2c8dda12b1bc496" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:49.644915 26378 raft_consensus.cc:3032] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:54:49.644992 26378 raft_consensus.cc:729] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:49.645293 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef474f44fe5b4a06b2c8dda12b1bc496" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.645316 26378 consensus_queue.cc:260] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.645587 26378 raft_consensus.cc:3037] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.645552 26643 raft_consensus.cc:3037] T ef474f44fe5b4a06b2c8dda12b1bc496 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.646739 26228 catalog_manager.cc:5484] T 8cf6da25f535446b9515444e093cae7f P 401942254c4c43a1ba02c4ae68d17320 reported cstate change: term changed from 2 to 3, leader changed from 4bc1541b47df446297a369de05163012 (127.11.42.66) to 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68). New cstate: current_term: 3 leader_uuid: "401942254c4c43a1ba02c4ae68d17320" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.648321 26643 raft_consensus.cc:2396] T ef474f44fe5b4a06b2c8dda12b1bc496 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 401942254c4c43a1ba02c4ae68d17320 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.648368 26378 raft_consensus.cc:2396] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 401942254c4c43a1ba02c4ae68d17320 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.649621 26708 leader_election.cc:302] T ef474f44fe5b4a06b2c8dda12b1bc496 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320; no voters: 5e9d825f803c40e089b64e319beccf6e, 83c50029f7a94cfca6d1ee68e0728aa2
I20240502 01:54:49.650110 27670 raft_consensus.cc:2726] T ef474f44fe5b4a06b2c8dda12b1bc496 P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 01:54:49.650482 26773 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "14a747ac14354fdab2a2993594d3398f"
dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
from {username='slave'} at 127.0.0.1:43296
I20240502 01:54:49.650683 26773 raft_consensus.cc:482] T 14a747ac14354fdab2a2993594d3398f P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.650828 26773 raft_consensus.cc:3037] T 14a747ac14354fdab2a2993594d3398f P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.653779 26773 raft_consensus.cc:504] T 14a747ac14354fdab2a2993594d3398f P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.654414 26773 leader_election.cc:288] T 14a747ac14354fdab2a2993594d3398f P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 4 election: Requested vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:49.654789 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a747ac14354fdab2a2993594d3398f" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.654893 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a747ac14354fdab2a2993594d3398f" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:49.655031 26643 raft_consensus.cc:3037] T 14a747ac14354fdab2a2993594d3398f P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.655113 26511 raft_consensus.cc:3032] T 14a747ac14354fdab2a2993594d3398f P 4bc1541b47df446297a369de05163012 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:54:49.655195 26511 raft_consensus.cc:729] T 14a747ac14354fdab2a2993594d3398f P 4bc1541b47df446297a369de05163012 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:49.655501 26511 consensus_queue.cc:260] T 14a747ac14354fdab2a2993594d3398f P 4bc1541b47df446297a369de05163012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.655812 26511 raft_consensus.cc:3037] T 14a747ac14354fdab2a2993594d3398f P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.657124 26773 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4254942d7ddc4c408114ae8d4b932a1a"
dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
from {username='slave'} at 127.0.0.1:43296
I20240502 01:54:49.657310 26773 raft_consensus.cc:482] T 4254942d7ddc4c408114ae8d4b932a1a P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.657470 26773 raft_consensus.cc:3037] T 4254942d7ddc4c408114ae8d4b932a1a P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.657837 26643 raft_consensus.cc:2396] T 14a747ac14354fdab2a2993594d3398f P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 401942254c4c43a1ba02c4ae68d17320 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.658425 26511 raft_consensus.cc:2396] T 14a747ac14354fdab2a2993594d3398f P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 401942254c4c43a1ba02c4ae68d17320 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.659054 26709 leader_election.cc:302] T 14a747ac14354fdab2a2993594d3398f P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320; no voters: 4bc1541b47df446297a369de05163012, 83c50029f7a94cfca6d1ee68e0728aa2
I20240502 01:54:49.659279 27670 raft_consensus.cc:2726] T 14a747ac14354fdab2a2993594d3398f P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 01:54:49.660426 26773 raft_consensus.cc:504] T 4254942d7ddc4c408114ae8d4b932a1a P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.661101 26773 leader_election.cc:288] T 4254942d7ddc4c408114ae8d4b932a1a P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 3 election: Requested vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085)
I20240502 01:54:49.661849 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4254942d7ddc4c408114ae8d4b932a1a" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:49.662127 26378 raft_consensus.cc:3032] T 4254942d7ddc4c408114ae8d4b932a1a P 5e9d825f803c40e089b64e319beccf6e [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:54:49.662132 26773 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d4b8975c68d64c2d804fcc852384afde"
dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
from {username='slave'} at 127.0.0.1:43296
I20240502 01:54:49.662166 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4254942d7ddc4c408114ae8d4b932a1a" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.662261 26378 raft_consensus.cc:729] T 4254942d7ddc4c408114ae8d4b932a1a P 5e9d825f803c40e089b64e319beccf6e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:49.662384 26643 raft_consensus.cc:3037] T 4254942d7ddc4c408114ae8d4b932a1a P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.662401 26773 raft_consensus.cc:482] T d4b8975c68d64c2d804fcc852384afde P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.662586 26773 raft_consensus.cc:3037] T d4b8975c68d64c2d804fcc852384afde P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.662611 26378 consensus_queue.cc:260] T 4254942d7ddc4c408114ae8d4b932a1a P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.662901 26378 raft_consensus.cc:3037] T 4254942d7ddc4c408114ae8d4b932a1a P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.665159 26643 raft_consensus.cc:2454] T 4254942d7ddc4c408114ae8d4b932a1a P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 401942254c4c43a1ba02c4ae68d17320 in term 3.
I20240502 01:54:49.665158 26773 raft_consensus.cc:504] T d4b8975c68d64c2d804fcc852384afde P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.665611 26708 leader_election.cc:302] T 4254942d7ddc4c408114ae8d4b932a1a P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:49.665918 27670 raft_consensus.cc:2781] T 4254942d7ddc4c408114ae8d4b932a1a P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:54:49.666090 27670 raft_consensus.cc:686] T 4254942d7ddc4c408114ae8d4b932a1a P 401942254c4c43a1ba02c4ae68d17320 [term 3 LEADER]: Becoming Leader. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Running, Role: LEADER
I20240502 01:54:49.666085 26773 leader_election.cc:288] T d4b8975c68d64c2d804fcc852384afde P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 4 election: Requested vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:49.666420 27670 consensus_queue.cc:237] T 4254942d7ddc4c408114ae8d4b932a1a P 401942254c4c43a1ba02c4ae68d17320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.666509 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4b8975c68d64c2d804fcc852384afde" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.666662 26643 raft_consensus.cc:3037] T d4b8975c68d64c2d804fcc852384afde P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.666864 26378 raft_consensus.cc:2396] T 4254942d7ddc4c408114ae8d4b932a1a P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 401942254c4c43a1ba02c4ae68d17320 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.667668 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4b8975c68d64c2d804fcc852384afde" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:49.667910 26511 raft_consensus.cc:3032] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:54:49.668012 26511 raft_consensus.cc:729] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:49.668309 26511 consensus_queue.cc:260] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.668618 26511 raft_consensus.cc:3037] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.669490 26643 raft_consensus.cc:2454] T d4b8975c68d64c2d804fcc852384afde P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 401942254c4c43a1ba02c4ae68d17320 in term 4.
I20240502 01:54:49.670224 26773 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4f9ed0162eb74c4ea422f122da01dce3"
dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
from {username='slave'} at 127.0.0.1:43296
I20240502 01:54:49.670699 26708 leader_election.cc:302] T d4b8975c68d64c2d804fcc852384afde P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:49.670858 26773 raft_consensus.cc:482] T 4f9ed0162eb74c4ea422f122da01dce3 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.671058 26511 raft_consensus.cc:2396] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 401942254c4c43a1ba02c4ae68d17320 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 01:54:49.671093 26773 raft_consensus.cc:3037] T 4f9ed0162eb74c4ea422f122da01dce3 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.672950 26773 raft_consensus.cc:504] T 4f9ed0162eb74c4ea422f122da01dce3 P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.674270 26773 leader_election.cc:288] T 4f9ed0162eb74c4ea422f122da01dce3 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 4 election: Requested vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085)
I20240502 01:54:49.675575 26773 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c56cb13e76384d5f9a0e45be2f65c085"
dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
from {username='slave'} at 127.0.0.1:43296
I20240502 01:54:49.675844 26773 raft_consensus.cc:482] T c56cb13e76384d5f9a0e45be2f65c085 P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.676014 26773 raft_consensus.cc:3037] T c56cb13e76384d5f9a0e45be2f65c085 P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.677309 28847 raft_consensus.cc:2781] T d4b8975c68d64c2d804fcc852384afde P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 01:54:49.679203 26773 raft_consensus.cc:504] T c56cb13e76384d5f9a0e45be2f65c085 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.680297 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c56cb13e76384d5f9a0e45be2f65c085" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.680507 26643 raft_consensus.cc:3037] T c56cb13e76384d5f9a0e45be2f65c085 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.680759 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9ed0162eb74c4ea422f122da01dce3" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:49.681283 26511 raft_consensus.cc:3032] T 4f9ed0162eb74c4ea422f122da01dce3 P 4bc1541b47df446297a369de05163012 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:54:49.683012 26228 catalog_manager.cc:5484] T 4254942d7ddc4c408114ae8d4b932a1a P 401942254c4c43a1ba02c4ae68d17320 reported cstate change: term changed from 2 to 3, leader changed from 5e9d825f803c40e089b64e319beccf6e (127.11.42.65) to 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68). New cstate: current_term: 3 leader_uuid: "401942254c4c43a1ba02c4ae68d17320" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.683272 26643 raft_consensus.cc:2454] T c56cb13e76384d5f9a0e45be2f65c085 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 401942254c4c43a1ba02c4ae68d17320 in term 3.
I20240502 01:54:49.683704 26708 leader_election.cc:302] T c56cb13e76384d5f9a0e45be2f65c085 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:49.684229 26773 leader_election.cc:288] T c56cb13e76384d5f9a0e45be2f65c085 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 3 election: Requested vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085)
I20240502 01:54:49.684551 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c56cb13e76384d5f9a0e45be2f65c085" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:49.684772 26379 raft_consensus.cc:3032] T c56cb13e76384d5f9a0e45be2f65c085 P 5e9d825f803c40e089b64e319beccf6e [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:54:49.684846 26379 raft_consensus.cc:729] T c56cb13e76384d5f9a0e45be2f65c085 P 5e9d825f803c40e089b64e319beccf6e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:49.685004 27670 raft_consensus.cc:2781] T c56cb13e76384d5f9a0e45be2f65c085 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:54:49.685206 27670 raft_consensus.cc:686] T c56cb13e76384d5f9a0e45be2f65c085 P 401942254c4c43a1ba02c4ae68d17320 [term 3 LEADER]: Becoming Leader. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Running, Role: LEADER
I20240502 01:54:49.685144 26379 consensus_queue.cc:260] T c56cb13e76384d5f9a0e45be2f65c085 P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.685433 26379 raft_consensus.cc:3037] T c56cb13e76384d5f9a0e45be2f65c085 P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.685699 27670 consensus_queue.cc:237] T c56cb13e76384d5f9a0e45be2f65c085 P 401942254c4c43a1ba02c4ae68d17320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.688443 26379 raft_consensus.cc:2396] T c56cb13e76384d5f9a0e45be2f65c085 P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 401942254c4c43a1ba02c4ae68d17320 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.689419 26511 raft_consensus.cc:729] T 4f9ed0162eb74c4ea422f122da01dce3 P 4bc1541b47df446297a369de05163012 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:49.689468 26773 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d9dd99b54b7c4d19a80c229bf99d7976"
dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
from {username='slave'} at 127.0.0.1:43296
I20240502 01:54:49.689677 26773 raft_consensus.cc:482] T d9dd99b54b7c4d19a80c229bf99d7976 P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.689777 26773 raft_consensus.cc:3037] T d9dd99b54b7c4d19a80c229bf99d7976 P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.689769 26511 consensus_queue.cc:260] T 4f9ed0162eb74c4ea422f122da01dce3 P 4bc1541b47df446297a369de05163012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.690057 26511 raft_consensus.cc:3037] T 4f9ed0162eb74c4ea422f122da01dce3 P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.691653 26773 raft_consensus.cc:504] T d9dd99b54b7c4d19a80c229bf99d7976 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.692173 26773 leader_election.cc:288] T d9dd99b54b7c4d19a80c229bf99d7976 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 3 election: Requested vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:49.692585 26508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9dd99b54b7c4d19a80c229bf99d7976" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:49.692826 26508 raft_consensus.cc:2379] T d9dd99b54b7c4d19a80c229bf99d7976 P 4bc1541b47df446297a369de05163012 [term 3 LEADER]: Leader election vote request: Denying vote to candidate 401942254c4c43a1ba02c4ae68d17320 in current term 3: Already voted for candidate 4bc1541b47df446297a369de05163012 in this term.
I20240502 01:54:49.692977 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9ed0162eb74c4ea422f122da01dce3" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:49.693140 26773 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1c47a74a6dc54d5eb0b47818d816b7fd"
dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
from {username='slave'} at 127.0.0.1:43296
I20240502 01:54:49.693559 26773 raft_consensus.cc:482] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.694001 26773 raft_consensus.cc:3037] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.694036 26378 raft_consensus.cc:3037] T 4f9ed0162eb74c4ea422f122da01dce3 P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.696257 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9dd99b54b7c4d19a80c229bf99d7976" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.697284 26643 raft_consensus.cc:2379] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 401942254c4c43a1ba02c4ae68d17320 in current term 3: Already voted for candidate 4bc1541b47df446297a369de05163012 in this term.
I20240502 01:54:49.698525 26708 leader_election.cc:302] T d9dd99b54b7c4d19a80c229bf99d7976 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320; no voters: 4bc1541b47df446297a369de05163012, 83c50029f7a94cfca6d1ee68e0728aa2
I20240502 01:54:49.698967 27670 raft_consensus.cc:2726] T d9dd99b54b7c4d19a80c229bf99d7976 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240502 01:54:49.699476 26773 raft_consensus.cc:504] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.700453 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c47a74a6dc54d5eb0b47818d816b7fd" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.700639 26643 raft_consensus.cc:3037] T 1c47a74a6dc54d5eb0b47818d816b7fd P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.700932 26508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c47a74a6dc54d5eb0b47818d816b7fd" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:49.701180 26508 raft_consensus.cc:3032] T 1c47a74a6dc54d5eb0b47818d816b7fd P 4bc1541b47df446297a369de05163012 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:54:49.701257 26508 raft_consensus.cc:729] T 1c47a74a6dc54d5eb0b47818d816b7fd P 4bc1541b47df446297a369de05163012 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:49.701506 26508 consensus_queue.cc:260] T 1c47a74a6dc54d5eb0b47818d816b7fd P 4bc1541b47df446297a369de05163012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.701812 26508 raft_consensus.cc:3037] T 1c47a74a6dc54d5eb0b47818d816b7fd P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.703169 26643 raft_consensus.cc:2454] T 1c47a74a6dc54d5eb0b47818d816b7fd P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 401942254c4c43a1ba02c4ae68d17320 in term 4.
I20240502 01:54:49.704321 26508 raft_consensus.cc:2396] T 1c47a74a6dc54d5eb0b47818d816b7fd P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 401942254c4c43a1ba02c4ae68d17320 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 01:54:49.704897 26708 leader_election.cc:302] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 83c50029f7a94cfca6d1ee68e0728aa2; no voters: 4bc1541b47df446297a369de05163012
I20240502 01:54:49.705183 27670 raft_consensus.cc:2781] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 01:54:49.705185 26773 leader_election.cc:288] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 4 election: Requested vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:49.705382 27670 raft_consensus.cc:686] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320 [term 4 LEADER]: Becoming Leader. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Running, Role: LEADER
I20240502 01:54:49.705610 27670 consensus_queue.cc:237] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.706310 26773 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "245ca06c39774fb5b0f522550a63ac5b"
dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
from {username='slave'} at 127.0.0.1:43296
I20240502 01:54:49.706801 26773 raft_consensus.cc:482] T 245ca06c39774fb5b0f522550a63ac5b P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.706907 26773 raft_consensus.cc:3037] T 245ca06c39774fb5b0f522550a63ac5b P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.707190 26511 raft_consensus.cc:2396] T 4f9ed0162eb74c4ea422f122da01dce3 P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 401942254c4c43a1ba02c4ae68d17320 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.708770 26773 raft_consensus.cc:504] T 245ca06c39774fb5b0f522550a63ac5b P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.709424 26378 raft_consensus.cc:2396] T 4f9ed0162eb74c4ea422f122da01dce3 P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 401942254c4c43a1ba02c4ae68d17320 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.709980 26708 leader_election.cc:302] T 4f9ed0162eb74c4ea422f122da01dce3 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320; no voters: 4bc1541b47df446297a369de05163012, 5e9d825f803c40e089b64e319beccf6e
I20240502 01:54:49.710455 27670 raft_consensus.cc:2726] T 4f9ed0162eb74c4ea422f122da01dce3 P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 01:54:49.710433 28847 raft_consensus.cc:686] T d4b8975c68d64c2d804fcc852384afde P 401942254c4c43a1ba02c4ae68d17320 [term 4 LEADER]: Becoming Leader. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Running, Role: LEADER
I20240502 01:54:49.710745 28847 consensus_queue.cc:237] T d4b8975c68d64c2d804fcc852384afde P 401942254c4c43a1ba02c4ae68d17320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.711202 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "245ca06c39774fb5b0f522550a63ac5b" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:49.711412 26773 leader_election.cc:288] T 245ca06c39774fb5b0f522550a63ac5b P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 3 election: Requested vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:49.711467 26511 raft_consensus.cc:3032] T 245ca06c39774fb5b0f522550a63ac5b P 4bc1541b47df446297a369de05163012 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:54:49.711612 26511 raft_consensus.cc:729] T 245ca06c39774fb5b0f522550a63ac5b P 4bc1541b47df446297a369de05163012 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:49.711248 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "245ca06c39774fb5b0f522550a63ac5b" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:49.712102 26511 consensus_queue.cc:260] T 245ca06c39774fb5b0f522550a63ac5b P 4bc1541b47df446297a369de05163012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.712282 26378 raft_consensus.cc:3037] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.712448 26511 raft_consensus.cc:3037] T 245ca06c39774fb5b0f522550a63ac5b P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.713066 26772 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "daef5905fc19473a8728ccd2627270af"
dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
from {username='slave'} at 127.0.0.1:43296
I20240502 01:54:49.713245 26772 raft_consensus.cc:482] T daef5905fc19473a8728ccd2627270af P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.713342 26772 raft_consensus.cc:3037] T daef5905fc19473a8728ccd2627270af P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.714927 26378 raft_consensus.cc:2454] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 401942254c4c43a1ba02c4ae68d17320 in term 3.
I20240502 01:54:49.715740 26772 raft_consensus.cc:504] T daef5905fc19473a8728ccd2627270af P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.716208 26772 leader_election.cc:288] T daef5905fc19473a8728ccd2627270af P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 4 election: Requested vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:49.716790 26511 raft_consensus.cc:2396] T 245ca06c39774fb5b0f522550a63ac5b P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 401942254c4c43a1ba02c4ae68d17320 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.717264 26708 leader_election.cc:302] T 245ca06c39774fb5b0f522550a63ac5b P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:49.717510 27670 raft_consensus.cc:2781] T 245ca06c39774fb5b0f522550a63ac5b P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:54:49.717680 27670 raft_consensus.cc:686] T 245ca06c39774fb5b0f522550a63ac5b P 401942254c4c43a1ba02c4ae68d17320 [term 3 LEADER]: Becoming Leader. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Running, Role: LEADER
I20240502 01:54:49.717921 27670 consensus_queue.cc:237] T 245ca06c39774fb5b0f522550a63ac5b P 401942254c4c43a1ba02c4ae68d17320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.718701 26228 catalog_manager.cc:5484] T d4b8975c68d64c2d804fcc852384afde P 401942254c4c43a1ba02c4ae68d17320 reported cstate change: term changed from 3 to 4, leader changed from 4bc1541b47df446297a369de05163012 (127.11.42.66) to 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68). New cstate: current_term: 4 leader_uuid: "401942254c4c43a1ba02c4ae68d17320" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.719930 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daef5905fc19473a8728ccd2627270af" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.719916 26773 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "53b7377c0d144e07a4501c2d626f6494"
dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
from {username='slave'} at 127.0.0.1:43296
I20240502 01:54:49.720161 26643 raft_consensus.cc:3037] T daef5905fc19473a8728ccd2627270af P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.720304 26773 raft_consensus.cc:482] T 53b7377c0d144e07a4501c2d626f6494 P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.720470 26773 raft_consensus.cc:3037] T 53b7377c0d144e07a4501c2d626f6494 P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.723074 26643 raft_consensus.cc:2396] T daef5905fc19473a8728ccd2627270af P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 401942254c4c43a1ba02c4ae68d17320 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.722994 26773 raft_consensus.cc:504] T 53b7377c0d144e07a4501c2d626f6494 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.723753 26773 leader_election.cc:288] T 53b7377c0d144e07a4501c2d626f6494 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 3 election: Requested vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:49.723927 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daef5905fc19473a8728ccd2627270af" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:49.724247 26378 raft_consensus.cc:3032] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:54:49.724431 26378 raft_consensus.cc:729] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:49.724516 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b7377c0d144e07a4501c2d626f6494" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:49.724709 26379 raft_consensus.cc:3032] T 53b7377c0d144e07a4501c2d626f6494 P 5e9d825f803c40e089b64e319beccf6e [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:54:49.724799 26379 raft_consensus.cc:729] T 53b7377c0d144e07a4501c2d626f6494 P 5e9d825f803c40e089b64e319beccf6e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:49.725060 26379 consensus_queue.cc:260] T 53b7377c0d144e07a4501c2d626f6494 P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.725317 26379 raft_consensus.cc:3037] T 53b7377c0d144e07a4501c2d626f6494 P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.725613 26378 consensus_queue.cc:260] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.725967 26378 raft_consensus.cc:3037] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.726558 26774 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0ddd395eec1146bc8a20292d81be4868"
dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
from {username='slave'} at 127.0.0.1:43296
I20240502 01:54:49.726713 26774 raft_consensus.cc:482] T 0ddd395eec1146bc8a20292d81be4868 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.726827 26774 raft_consensus.cc:3037] T 0ddd395eec1146bc8a20292d81be4868 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.728168 26379 raft_consensus.cc:2396] T 53b7377c0d144e07a4501c2d626f6494 P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 401942254c4c43a1ba02c4ae68d17320 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.728770 26640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b7377c0d144e07a4501c2d626f6494" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.728945 26640 raft_consensus.cc:3037] T 53b7377c0d144e07a4501c2d626f6494 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.729074 26378 raft_consensus.cc:2396] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 401942254c4c43a1ba02c4ae68d17320 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.729331 26774 raft_consensus.cc:504] T 0ddd395eec1146bc8a20292d81be4868 P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.729859 26774 leader_election.cc:288] T 0ddd395eec1146bc8a20292d81be4868 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 4 election: Requested vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:49.731192 26640 raft_consensus.cc:2454] T 53b7377c0d144e07a4501c2d626f6494 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 401942254c4c43a1ba02c4ae68d17320 in term 3.
I20240502 01:54:49.731791 26708 leader_election.cc:302] T 53b7377c0d144e07a4501c2d626f6494 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:49.731940 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ddd395eec1146bc8a20292d81be4868" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:49.732081 28857 raft_consensus.cc:2781] T 53b7377c0d144e07a4501c2d626f6494 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:54:49.732133 26378 raft_consensus.cc:3032] T 0ddd395eec1146bc8a20292d81be4868 P 5e9d825f803c40e089b64e319beccf6e [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:54:49.732147 26774 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "89107a545fcc48aead1e199da1f78311"
dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
from {username='slave'} at 127.0.0.1:43296
I20240502 01:54:49.732206 26378 raft_consensus.cc:729] T 0ddd395eec1146bc8a20292d81be4868 P 5e9d825f803c40e089b64e319beccf6e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:49.732247 28857 raft_consensus.cc:686] T 53b7377c0d144e07a4501c2d626f6494 P 401942254c4c43a1ba02c4ae68d17320 [term 3 LEADER]: Becoming Leader. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Running, Role: LEADER
I20240502 01:54:49.732326 26774 raft_consensus.cc:482] T 89107a545fcc48aead1e199da1f78311 P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.732340 26708 leader_election.cc:302] T daef5905fc19473a8728ccd2627270af P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320; no voters: 5e9d825f803c40e089b64e319beccf6e, 83c50029f7a94cfca6d1ee68e0728aa2
I20240502 01:54:49.732421 26774 raft_consensus.cc:3037] T 89107a545fcc48aead1e199da1f78311 P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.732465 26378 consensus_queue.cc:260] T 0ddd395eec1146bc8a20292d81be4868 P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.732717 26378 raft_consensus.cc:3037] T 0ddd395eec1146bc8a20292d81be4868 P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.732594 28857 consensus_queue.cc:237] T 53b7377c0d144e07a4501c2d626f6494 P 401942254c4c43a1ba02c4ae68d17320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.732838 27670 raft_consensus.cc:2726] T daef5905fc19473a8728ccd2627270af P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 01:54:49.733966 26640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ddd395eec1146bc8a20292d81be4868" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.734154 26640 raft_consensus.cc:3037] T 0ddd395eec1146bc8a20292d81be4868 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.735741 26378 raft_consensus.cc:2396] T 0ddd395eec1146bc8a20292d81be4868 P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 401942254c4c43a1ba02c4ae68d17320 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.736235 26774 raft_consensus.cc:504] T 89107a545fcc48aead1e199da1f78311 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.736974 26774 leader_election.cc:288] T 89107a545fcc48aead1e199da1f78311 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 3 election: Requested vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:49.737375 26642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89107a545fcc48aead1e199da1f78311" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.737455 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89107a545fcc48aead1e199da1f78311" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:49.737550 26642 raft_consensus.cc:3037] T 89107a545fcc48aead1e199da1f78311 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.737660 26511 raft_consensus.cc:3032] T 89107a545fcc48aead1e199da1f78311 P 4bc1541b47df446297a369de05163012 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:54:49.737737 26511 raft_consensus.cc:729] T 89107a545fcc48aead1e199da1f78311 P 4bc1541b47df446297a369de05163012 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:49.738042 26511 consensus_queue.cc:260] T 89107a545fcc48aead1e199da1f78311 P 4bc1541b47df446297a369de05163012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.738344 26511 raft_consensus.cc:3037] T 89107a545fcc48aead1e199da1f78311 P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.738875 26774 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "37e818dc0b834eb185e27fcb3cb972e1"
dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
from {username='slave'} at 127.0.0.1:43296
I20240502 01:54:49.739022 26774 raft_consensus.cc:482] T 37e818dc0b834eb185e27fcb3cb972e1 P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.739115 26774 raft_consensus.cc:3037] T 37e818dc0b834eb185e27fcb3cb972e1 P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.740124 26642 raft_consensus.cc:2454] T 89107a545fcc48aead1e199da1f78311 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 401942254c4c43a1ba02c4ae68d17320 in term 3.
I20240502 01:54:49.740792 26708 leader_election.cc:302] T 89107a545fcc48aead1e199da1f78311 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:49.741073 28847 raft_consensus.cc:2781] T 89107a545fcc48aead1e199da1f78311 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:54:49.741236 28847 raft_consensus.cc:686] T 89107a545fcc48aead1e199da1f78311 P 401942254c4c43a1ba02c4ae68d17320 [term 3 LEADER]: Becoming Leader. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Running, Role: LEADER
I20240502 01:54:49.741174 26511 raft_consensus.cc:2396] T 89107a545fcc48aead1e199da1f78311 P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 401942254c4c43a1ba02c4ae68d17320 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.741473 28847 consensus_queue.cc:237] T 89107a545fcc48aead1e199da1f78311 P 401942254c4c43a1ba02c4ae68d17320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.742139 26640 raft_consensus.cc:2396] T 0ddd395eec1146bc8a20292d81be4868 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 401942254c4c43a1ba02c4ae68d17320 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.742606 26643 raft_consensus.cc:1261] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Refusing update from remote peer 5e9d825f803c40e089b64e319beccf6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:54:49.742623 26708 leader_election.cc:302] T 0ddd395eec1146bc8a20292d81be4868 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320; no voters: 5e9d825f803c40e089b64e319beccf6e, 83c50029f7a94cfca6d1ee68e0728aa2
I20240502 01:54:49.743014 27670 raft_consensus.cc:2726] T 0ddd395eec1146bc8a20292d81be4868 P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 01:54:49.743667 28191 consensus_queue.cc:1035] T 3d751cdd58eb4cef8039fd4c50855e99 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:49.750440 26773 raft_consensus.cc:1261] T 3d751cdd58eb4cef8039fd4c50855e99 P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Refusing update from remote peer 5e9d825f803c40e089b64e319beccf6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:54:49.750986 28191 consensus_queue.cc:1035] T 3d751cdd58eb4cef8039fd4c50855e99 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:49.741535 26774 raft_consensus.cc:504] T 37e818dc0b834eb185e27fcb3cb972e1 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.761847 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e818dc0b834eb185e27fcb3cb972e1" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:49.761878 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e818dc0b834eb185e27fcb3cb972e1" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:49.762074 26511 raft_consensus.cc:3032] T 37e818dc0b834eb185e27fcb3cb972e1 P 4bc1541b47df446297a369de05163012 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:54:49.762109 26379 raft_consensus.cc:3037] T 37e818dc0b834eb185e27fcb3cb972e1 P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.762149 26511 raft_consensus.cc:729] T 37e818dc0b834eb185e27fcb3cb972e1 P 4bc1541b47df446297a369de05163012 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:49.762444 26511 consensus_queue.cc:260] T 37e818dc0b834eb185e27fcb3cb972e1 P 4bc1541b47df446297a369de05163012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.762706 26511 raft_consensus.cc:3037] T 37e818dc0b834eb185e27fcb3cb972e1 P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.764927 26379 raft_consensus.cc:2454] T 37e818dc0b834eb185e27fcb3cb972e1 P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 401942254c4c43a1ba02c4ae68d17320 in term 3.
I20240502 01:54:49.765436 26708 leader_election.cc:302] T 37e818dc0b834eb185e27fcb3cb972e1 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:49.765662 26511 raft_consensus.cc:2396] T 37e818dc0b834eb185e27fcb3cb972e1 P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 401942254c4c43a1ba02c4ae68d17320 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.765755 27670 raft_consensus.cc:2781] T 37e818dc0b834eb185e27fcb3cb972e1 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:54:49.781877 26774 leader_election.cc:288] T 37e818dc0b834eb185e27fcb3cb972e1 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 3 election: Requested vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085)
I20240502 01:54:49.783190 26774 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dc82c30359e84bddb4bcfe1b3b13ced7"
dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
from {username='slave'} at 127.0.0.1:43296
I20240502 01:54:49.783313 27670 raft_consensus.cc:686] T 37e818dc0b834eb185e27fcb3cb972e1 P 401942254c4c43a1ba02c4ae68d17320 [term 3 LEADER]: Becoming Leader. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Running, Role: LEADER
I20240502 01:54:49.783452 26774 raft_consensus.cc:482] T dc82c30359e84bddb4bcfe1b3b13ced7 P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.783591 26774 raft_consensus.cc:3037] T dc82c30359e84bddb4bcfe1b3b13ced7 P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.783661 27670 consensus_queue.cc:237] T 37e818dc0b834eb185e27fcb3cb972e1 P 401942254c4c43a1ba02c4ae68d17320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.786092 26774 raft_consensus.cc:504] T dc82c30359e84bddb4bcfe1b3b13ced7 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.786834 26774 leader_election.cc:288] T dc82c30359e84bddb4bcfe1b3b13ced7 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 3 election: Requested vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:49.787421 26772 raft_consensus.cc:1226] T 53b7377c0d144e07a4501c2d626f6494 P 401942254c4c43a1ba02c4ae68d17320 [term 3 LEADER]: Rejecting Update request from peer 5e9d825f803c40e089b64e319beccf6e for earlier term 2. Current term is 3. Ops: []
I20240502 01:54:49.788786 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc82c30359e84bddb4bcfe1b3b13ced7" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.789059 26643 raft_consensus.cc:3037] T dc82c30359e84bddb4bcfe1b3b13ced7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.789285 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc82c30359e84bddb4bcfe1b3b13ced7" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:49.789479 26511 raft_consensus.cc:3032] T dc82c30359e84bddb4bcfe1b3b13ced7 P 4bc1541b47df446297a369de05163012 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:54:49.789549 26511 raft_consensus.cc:729] T dc82c30359e84bddb4bcfe1b3b13ced7 P 4bc1541b47df446297a369de05163012 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:49.789844 26511 consensus_queue.cc:260] T dc82c30359e84bddb4bcfe1b3b13ced7 P 4bc1541b47df446297a369de05163012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.790110 26511 raft_consensus.cc:3037] T dc82c30359e84bddb4bcfe1b3b13ced7 P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.791790 26643 raft_consensus.cc:2454] T dc82c30359e84bddb4bcfe1b3b13ced7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 401942254c4c43a1ba02c4ae68d17320 in term 3.
I20240502 01:54:49.792304 26708 leader_election.cc:302] T dc82c30359e84bddb4bcfe1b3b13ced7 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:49.792671 26511 raft_consensus.cc:2396] T dc82c30359e84bddb4bcfe1b3b13ced7 P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 401942254c4c43a1ba02c4ae68d17320 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.792658 28857 raft_consensus.cc:2781] T dc82c30359e84bddb4bcfe1b3b13ced7 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:54:49.792827 28857 raft_consensus.cc:686] T dc82c30359e84bddb4bcfe1b3b13ced7 P 401942254c4c43a1ba02c4ae68d17320 [term 3 LEADER]: Becoming Leader. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Running, Role: LEADER
I20240502 01:54:49.793097 28857 consensus_queue.cc:237] T dc82c30359e84bddb4bcfe1b3b13ced7 P 401942254c4c43a1ba02c4ae68d17320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.794127 26772 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c26358e3969143fcbf2f466241b360a7"
dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
from {username='slave'} at 127.0.0.1:43296
I20240502 01:54:49.794343 26772 raft_consensus.cc:482] T c26358e3969143fcbf2f466241b360a7 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.794459 26772 raft_consensus.cc:3037] T c26358e3969143fcbf2f466241b360a7 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.794751 26228 catalog_manager.cc:5484] T c56cb13e76384d5f9a0e45be2f65c085 P 401942254c4c43a1ba02c4ae68d17320 reported cstate change: term changed from 2 to 3, leader changed from 5e9d825f803c40e089b64e319beccf6e (127.11.42.65) to 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68). New cstate: current_term: 3 leader_uuid: "401942254c4c43a1ba02c4ae68d17320" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:49.795536 26228 catalog_manager.cc:5484] T 245ca06c39774fb5b0f522550a63ac5b P 401942254c4c43a1ba02c4ae68d17320 reported cstate change: term changed from 2 to 3, leader changed from 4bc1541b47df446297a369de05163012 (127.11.42.66) to 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68). New cstate: current_term: 3 leader_uuid: "401942254c4c43a1ba02c4ae68d17320" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.796130 26228 catalog_manager.cc:5484] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320 reported cstate change: term changed from 3 to 4, leader changed from 4bc1541b47df446297a369de05163012 (127.11.42.66) to 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68). New cstate: current_term: 4 leader_uuid: "401942254c4c43a1ba02c4ae68d17320" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.796672 26228 catalog_manager.cc:5484] T 53b7377c0d144e07a4501c2d626f6494 P 401942254c4c43a1ba02c4ae68d17320 reported cstate change: term changed from 2 to 3, leader changed from 5e9d825f803c40e089b64e319beccf6e (127.11.42.65) to 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68). New cstate: current_term: 3 leader_uuid: "401942254c4c43a1ba02c4ae68d17320" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.797365 26643 raft_consensus.cc:1261] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Refusing update from remote peer 5e9d825f803c40e089b64e319beccf6e: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 01:54:49.798228 26643 pending_rounds.cc:76] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2: Aborting all ops after (but not including) 3
I20240502 01:54:49.798389 26643 pending_rounds.cc:97] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 01:54:49.798774 26772 raft_consensus.cc:504] T c26358e3969143fcbf2f466241b360a7 P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.799067 26643 raft_consensus.cc:2866] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 01:54:49.799912 28191 consensus_queue.cc:1035] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:49.808236 26774 raft_consensus.cc:1226] T c26358e3969143fcbf2f466241b360a7 P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Rejecting Update request from peer 5e9d825f803c40e089b64e319beccf6e for earlier term 3. Current term is 4. Ops: []
I20240502 01:54:49.809468 28794 consensus_queue.cc:1046] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:49.809849 28794 raft_consensus.cc:3032] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:54:49.809909 26772 leader_election.cc:288] T c26358e3969143fcbf2f466241b360a7 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 4 election: Requested vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085)
I20240502 01:54:49.809955 28794 raft_consensus.cc:729] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:49.810213 28794 consensus_queue.cc:260] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.810592 28794 raft_consensus.cc:3037] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.813207 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c26358e3969143fcbf2f466241b360a7" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:49.814146 26379 raft_consensus.cc:2396] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 401942254c4c43a1ba02c4ae68d17320 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.815973 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c26358e3969143fcbf2f466241b360a7" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.816187 26643 raft_consensus.cc:3037] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.818127 26772 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c9e4788d50d04d129e8d03125073c6f2"
dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
from {username='slave'} at 127.0.0.1:43296
I20240502 01:54:49.818318 26772 raft_consensus.cc:482] T c9e4788d50d04d129e8d03125073c6f2 P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.818496 26772 raft_consensus.cc:3037] T c9e4788d50d04d129e8d03125073c6f2 P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.821491 26643 raft_consensus.cc:2396] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 401942254c4c43a1ba02c4ae68d17320 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.821991 26708 leader_election.cc:302] T c26358e3969143fcbf2f466241b360a7 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320; no voters: 5e9d825f803c40e089b64e319beccf6e, 83c50029f7a94cfca6d1ee68e0728aa2
I20240502 01:54:49.822438 28847 raft_consensus.cc:2726] T c26358e3969143fcbf2f466241b360a7 P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 01:54:49.823338 26772 raft_consensus.cc:504] T c9e4788d50d04d129e8d03125073c6f2 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.824183 26772 leader_election.cc:288] T c9e4788d50d04d129e8d03125073c6f2 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 3 election: Requested vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085)
I20240502 01:54:49.825102 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e4788d50d04d129e8d03125073c6f2" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:49.825201 26774 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "22d4681165d1488db2c45ec83cb49cb4"
dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
from {username='slave'} at 127.0.0.1:43296
I20240502 01:54:49.825294 26379 raft_consensus.cc:3037] T c9e4788d50d04d129e8d03125073c6f2 P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.825413 26774 raft_consensus.cc:482] T 22d4681165d1488db2c45ec83cb49cb4 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.825516 26774 raft_consensus.cc:3037] T 22d4681165d1488db2c45ec83cb49cb4 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.828143 26379 raft_consensus.cc:2454] T c9e4788d50d04d129e8d03125073c6f2 P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 401942254c4c43a1ba02c4ae68d17320 in term 3.
I20240502 01:54:49.828626 26774 raft_consensus.cc:504] T 22d4681165d1488db2c45ec83cb49cb4 P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.828763 26708 leader_election.cc:302] T c9e4788d50d04d129e8d03125073c6f2 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:49.829111 28847 raft_consensus.cc:2781] T c9e4788d50d04d129e8d03125073c6f2 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:54:49.829270 28847 raft_consensus.cc:686] T c9e4788d50d04d129e8d03125073c6f2 P 401942254c4c43a1ba02c4ae68d17320 [term 3 LEADER]: Becoming Leader. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Running, Role: LEADER
I20240502 01:54:49.829442 26774 leader_election.cc:288] T 22d4681165d1488db2c45ec83cb49cb4 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 4 election: Requested vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:49.829526 28847 consensus_queue.cc:237] T c9e4788d50d04d129e8d03125073c6f2 P 401942254c4c43a1ba02c4ae68d17320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.832388 26772 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3d751cdd58eb4cef8039fd4c50855e99"
dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
from {username='slave'} at 127.0.0.1:43296
I20240502 01:54:49.832571 26772 raft_consensus.cc:482] T 3d751cdd58eb4cef8039fd4c50855e99 P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.832660 26772 raft_consensus.cc:3037] T 3d751cdd58eb4cef8039fd4c50855e99 P 401942254c4c43a1ba02c4ae68d17320 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.835335 26772 raft_consensus.cc:504] T 3d751cdd58eb4cef8039fd4c50855e99 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.835899 26772 leader_election.cc:288] T 3d751cdd58eb4cef8039fd4c50855e99 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 3 election: Requested vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085)
I20240502 01:54:49.836755 26508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22d4681165d1488db2c45ec83cb49cb4" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:49.836865 26772 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1ce4dd22bb314eb7855126e3fe1785f6"
dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
from {username='slave'} at 127.0.0.1:43296
I20240502 01:54:49.837028 26772 raft_consensus.cc:482] T 1ce4dd22bb314eb7855126e3fe1785f6 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.837051 26508 raft_consensus.cc:3032] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:54:49.837137 26772 raft_consensus.cc:3037] T 1ce4dd22bb314eb7855126e3fe1785f6 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.837143 26508 raft_consensus.cc:729] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:49.837433 26508 consensus_queue.cc:260] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.837635 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d751cdd58eb4cef8039fd4c50855e99" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:49.837706 26508 raft_consensus.cc:3037] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.837837 26378 raft_consensus.cc:3032] T 3d751cdd58eb4cef8039fd4c50855e99 P 5e9d825f803c40e089b64e319beccf6e [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:54:49.837919 26378 raft_consensus.cc:729] T 3d751cdd58eb4cef8039fd4c50855e99 P 5e9d825f803c40e089b64e319beccf6e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:49.838171 26378 consensus_queue.cc:260] T 3d751cdd58eb4cef8039fd4c50855e99 P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.838506 26378 raft_consensus.cc:3037] T 3d751cdd58eb4cef8039fd4c50855e99 P 5e9d825f803c40e089b64e319beccf6e [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.840341 26508 raft_consensus.cc:2396] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 401942254c4c43a1ba02c4ae68d17320 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.841743 26640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d751cdd58eb4cef8039fd4c50855e99" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.841979 26640 raft_consensus.cc:3037] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.843202 26228 catalog_manager.cc:5484] T 89107a545fcc48aead1e199da1f78311 P 401942254c4c43a1ba02c4ae68d17320 reported cstate change: term changed from 2 to 3, leader changed from 4bc1541b47df446297a369de05163012 (127.11.42.66) to 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68). New cstate: current_term: 3 leader_uuid: "401942254c4c43a1ba02c4ae68d17320" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.843827 26228 catalog_manager.cc:5484] T dc82c30359e84bddb4bcfe1b3b13ced7 P 401942254c4c43a1ba02c4ae68d17320 reported cstate change: term changed from 2 to 3, leader changed from 4bc1541b47df446297a369de05163012 (127.11.42.66) to 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68). New cstate: current_term: 3 leader_uuid: "401942254c4c43a1ba02c4ae68d17320" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:49.844290 26228 catalog_manager.cc:5484] T 37e818dc0b834eb185e27fcb3cb972e1 P 401942254c4c43a1ba02c4ae68d17320 reported cstate change: term changed from 2 to 3, leader changed from 4bc1541b47df446297a369de05163012 (127.11.42.66) to 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68). New cstate: current_term: 3 leader_uuid: "401942254c4c43a1ba02c4ae68d17320" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.844573 26640 raft_consensus.cc:2454] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 401942254c4c43a1ba02c4ae68d17320 in term 3.
I20240502 01:54:49.841743 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22d4681165d1488db2c45ec83cb49cb4" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:49.844848 26643 raft_consensus.cc:3037] T 22d4681165d1488db2c45ec83cb49cb4 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.841001 26378 raft_consensus.cc:2454] T 3d751cdd58eb4cef8039fd4c50855e99 P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 401942254c4c43a1ba02c4ae68d17320 in term 3.
I20240502 01:54:49.845696 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e4788d50d04d129e8d03125073c6f2" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:49.845876 26511 raft_consensus.cc:3032] T c9e4788d50d04d129e8d03125073c6f2 P 4bc1541b47df446297a369de05163012 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:54:49.845945 26511 raft_consensus.cc:729] T c9e4788d50d04d129e8d03125073c6f2 P 4bc1541b47df446297a369de05163012 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:49.846187 26511 consensus_queue.cc:260] T c9e4788d50d04d129e8d03125073c6f2 P 4bc1541b47df446297a369de05163012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.846475 26511 raft_consensus.cc:3037] T c9e4788d50d04d129e8d03125073c6f2 P 4bc1541b47df446297a369de05163012 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:49.847400 26643 raft_consensus.cc:2396] T 22d4681165d1488db2c45ec83cb49cb4 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 401942254c4c43a1ba02c4ae68d17320 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.848937 26511 raft_consensus.cc:2396] T c9e4788d50d04d129e8d03125073c6f2 P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 401942254c4c43a1ba02c4ae68d17320 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.850364 26708 leader_election.cc:302] T 3d751cdd58eb4cef8039fd4c50855e99 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:49.850509 26772 raft_consensus.cc:504] T 1ce4dd22bb314eb7855126e3fe1785f6 P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.850762 28847 raft_consensus.cc:2781] T 3d751cdd58eb4cef8039fd4c50855e99 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:54:49.850905 28847 raft_consensus.cc:686] T 3d751cdd58eb4cef8039fd4c50855e99 P 401942254c4c43a1ba02c4ae68d17320 [term 3 LEADER]: Becoming Leader. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Running, Role: LEADER
I20240502 01:54:49.850965 26708 leader_election.cc:302] T 22d4681165d1488db2c45ec83cb49cb4 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320; no voters: 4bc1541b47df446297a369de05163012, 83c50029f7a94cfca6d1ee68e0728aa2
I20240502 01:54:49.851194 28847 consensus_queue.cc:237] T 3d751cdd58eb4cef8039fd4c50855e99 P 401942254c4c43a1ba02c4ae68d17320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.851436 27670 raft_consensus.cc:2726] T 22d4681165d1488db2c45ec83cb49cb4 P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 01:54:49.851644 26772 leader_election.cc:288] T 1ce4dd22bb314eb7855126e3fe1785f6 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 4 election: Requested vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085)
I20240502 01:54:49.851991 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ce4dd22bb314eb7855126e3fe1785f6" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:49.852180 26379 raft_consensus.cc:3037] T 1ce4dd22bb314eb7855126e3fe1785f6 P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.852578 26774 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b5ab243d9f1c4b568d38ed03027adb44"
dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
from {username='slave'} at 127.0.0.1:43296
I20240502 01:54:49.852746 26774 raft_consensus.cc:482] T b5ab243d9f1c4b568d38ed03027adb44 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.852864 26774 raft_consensus.cc:3037] T b5ab243d9f1c4b568d38ed03027adb44 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.855062 26379 raft_consensus.cc:2396] T 1ce4dd22bb314eb7855126e3fe1785f6 P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 401942254c4c43a1ba02c4ae68d17320 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.855679 26774 raft_consensus.cc:504] T b5ab243d9f1c4b568d38ed03027adb44 P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.856506 26774 leader_election.cc:288] T b5ab243d9f1c4b568d38ed03027adb44 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 4 election: Requested vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085)
I20240502 01:54:49.856568 26234 catalog_manager.cc:5484] T 3d751cdd58eb4cef8039fd4c50855e99 P 401942254c4c43a1ba02c4ae68d17320 reported cstate change: term changed from 2 to 3, leader changed from 5e9d825f803c40e089b64e319beccf6e (127.11.42.65) to 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68). New cstate: current_term: 3 leader_uuid: "401942254c4c43a1ba02c4ae68d17320" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:49.857913 26774 raft_consensus.cc:1261] T d9dd99b54b7c4d19a80c229bf99d7976 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Refusing update from remote peer 4bc1541b47df446297a369de05163012: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240502 01:54:49.857668 26234 catalog_manager.cc:5484] T c9e4788d50d04d129e8d03125073c6f2 P 401942254c4c43a1ba02c4ae68d17320 reported cstate change: term changed from 2 to 3, leader changed from 4bc1541b47df446297a369de05163012 (127.11.42.66) to 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68). New cstate: current_term: 3 leader_uuid: "401942254c4c43a1ba02c4ae68d17320" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:49.858996 26508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ab243d9f1c4b568d38ed03027adb44" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:49.859225 26508 raft_consensus.cc:3032] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:54:49.859314 26508 raft_consensus.cc:729] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:49.859380 28770 consensus_queue.cc:1035] T d9dd99b54b7c4d19a80c229bf99d7976 P 4bc1541b47df446297a369de05163012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:49.859493 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ce4dd22bb314eb7855126e3fe1785f6" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:49.859596 26508 consensus_queue.cc:260] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.859740 26511 raft_consensus.cc:3032] T 1ce4dd22bb314eb7855126e3fe1785f6 P 4bc1541b47df446297a369de05163012 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:54:49.859819 26511 raft_consensus.cc:729] T 1ce4dd22bb314eb7855126e3fe1785f6 P 4bc1541b47df446297a369de05163012 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:49.859898 26508 raft_consensus.cc:3037] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.860219 26511 consensus_queue.cc:260] T 1ce4dd22bb314eb7855126e3fe1785f6 P 4bc1541b47df446297a369de05163012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.860497 26511 raft_consensus.cc:3037] T 1ce4dd22bb314eb7855126e3fe1785f6 P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.861075 28745 heartbeater.cc:503] Master 127.11.42.126:36275 was elected leader, sending a full tablet report...
I20240502 01:54:49.861670 26378 raft_consensus.cc:1261] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Refusing update from remote peer 4bc1541b47df446297a369de05163012: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 01:54:49.861791 26378 pending_rounds.cc:76] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e: Aborting all ops after (but not including) 3
I20240502 01:54:49.861864 26378 pending_rounds.cc:97] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 01:54:49.862001 26378 raft_consensus.cc:2866] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 01:54:49.862640 28770 consensus_queue.cc:1035] T 9ead21d62e564a37a8c3d5a452621d13 P 4bc1541b47df446297a369de05163012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:49.863359 26511 raft_consensus.cc:2396] T 1ce4dd22bb314eb7855126e3fe1785f6 P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 401942254c4c43a1ba02c4ae68d17320 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.871588 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ab243d9f1c4b568d38ed03027adb44" candidate_uuid: "401942254c4c43a1ba02c4ae68d17320" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:49.871825 26378 raft_consensus.cc:3037] T b5ab243d9f1c4b568d38ed03027adb44 P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.872464 26508 raft_consensus.cc:2396] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 401942254c4c43a1ba02c4ae68d17320 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.874001 26643 raft_consensus.cc:1261] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Refusing update from remote peer 4bc1541b47df446297a369de05163012: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 01:54:49.874828 28162 consensus_queue.cc:1035] T 9ead21d62e564a37a8c3d5a452621d13 P 4bc1541b47df446297a369de05163012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:49.878885 26642 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cf09a605f32845a3aaad81d79063070b"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:49.879093 26642 raft_consensus.cc:482] T cf09a605f32845a3aaad81d79063070b P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.879072 26708 leader_election.cc:302] T 1ce4dd22bb314eb7855126e3fe1785f6 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320; no voters: 4bc1541b47df446297a369de05163012, 5e9d825f803c40e089b64e319beccf6e
I20240502 01:54:49.879191 26642 raft_consensus.cc:3037] T cf09a605f32845a3aaad81d79063070b P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:49.879415 28857 raft_consensus.cc:2726] T 1ce4dd22bb314eb7855126e3fe1785f6 P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 01:54:49.881901 26642 raft_consensus.cc:504] T cf09a605f32845a3aaad81d79063070b P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.883749 26642 leader_election.cc:288] T cf09a605f32845a3aaad81d79063070b P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 5 election: Requested vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:49.885618 26641 raft_consensus.cc:1261] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Refusing update from remote peer 4bc1541b47df446297a369de05163012: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240502 01:54:49.885921 26378 raft_consensus.cc:2396] T b5ab243d9f1c4b568d38ed03027adb44 P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 401942254c4c43a1ba02c4ae68d17320 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.886534 28770 consensus_queue.cc:1035] T d9dd99b54b7c4d19a80c229bf99d7976 P 4bc1541b47df446297a369de05163012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:49.886669 26708 leader_election.cc:302] T b5ab243d9f1c4b568d38ed03027adb44 P 401942254c4c43a1ba02c4ae68d17320 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320; no voters: 4bc1541b47df446297a369de05163012, 5e9d825f803c40e089b64e319beccf6e
I20240502 01:54:49.886940 28857 raft_consensus.cc:2726] T b5ab243d9f1c4b568d38ed03027adb44 P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 01:54:49.887147 26642 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4254942d7ddc4c408114ae8d4b932a1a"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:49.887364 26642 raft_consensus.cc:482] T 4254942d7ddc4c408114ae8d4b932a1a P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.887502 26642 raft_consensus.cc:3037] T 4254942d7ddc4c408114ae8d4b932a1a P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.890187 26642 raft_consensus.cc:504] T 4254942d7ddc4c408114ae8d4b932a1a P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.890733 26508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf09a605f32845a3aaad81d79063070b" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:49.890971 26508 raft_consensus.cc:3037] T cf09a605f32845a3aaad81d79063070b P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:49.891647 26772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4254942d7ddc4c408114ae8d4b932a1a" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:49.891939 26772 raft_consensus.cc:3032] T 4254942d7ddc4c408114ae8d4b932a1a P 401942254c4c43a1ba02c4ae68d17320 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:54:49.891902 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4254942d7ddc4c408114ae8d4b932a1a" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:49.892102 26378 raft_consensus.cc:3037] T 4254942d7ddc4c408114ae8d4b932a1a P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.892102 26772 raft_consensus.cc:729] T 4254942d7ddc4c408114ae8d4b932a1a P 401942254c4c43a1ba02c4ae68d17320 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Running, Role: LEADER
I20240502 01:54:49.892562 26772 consensus_queue.cc:260] T 4254942d7ddc4c408114ae8d4b932a1a P 401942254c4c43a1ba02c4ae68d17320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:49.892951 26772 raft_consensus.cc:3037] T 4254942d7ddc4c408114ae8d4b932a1a P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.894840 26508 raft_consensus.cc:2454] T cf09a605f32845a3aaad81d79063070b P 4bc1541b47df446297a369de05163012 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 5.
I20240502 01:54:49.895042 26378 raft_consensus.cc:2396] T 4254942d7ddc4c408114ae8d4b932a1a P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 83c50029f7a94cfca6d1ee68e0728aa2 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.895306 26578 leader_election.cc:302] T cf09a605f32845a3aaad81d79063070b P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:49.895588 28755 raft_consensus.cc:2781] T cf09a605f32845a3aaad81d79063070b P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 01:54:49.895691 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf09a605f32845a3aaad81d79063070b" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:49.895766 28755 raft_consensus.cc:686] T cf09a605f32845a3aaad81d79063070b P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 LEADER]: Becoming Leader. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:49.895864 26379 raft_consensus.cc:3037] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:49.896034 28755 consensus_queue.cc:237] T cf09a605f32845a3aaad81d79063070b P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.896569 26642 leader_election.cc:288] T 4254942d7ddc4c408114ae8d4b932a1a P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 4 election: Requested vote from peers 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959), 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085)
I20240502 01:54:49.897889 26643 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d4b8975c68d64c2d804fcc852384afde"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:49.898083 26643 raft_consensus.cc:482] T d4b8975c68d64c2d804fcc852384afde P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.898173 26643 raft_consensus.cc:3037] T d4b8975c68d64c2d804fcc852384afde P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:49.899252 26772 raft_consensus.cc:2396] T 4254942d7ddc4c408114ae8d4b932a1a P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 83c50029f7a94cfca6d1ee68e0728aa2 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.899765 26578 leader_election.cc:302] T 4254942d7ddc4c408114ae8d4b932a1a P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83c50029f7a94cfca6d1ee68e0728aa2; no voters: 401942254c4c43a1ba02c4ae68d17320, 5e9d825f803c40e089b64e319beccf6e
I20240502 01:54:49.900274 28762 raft_consensus.cc:2726] T 4254942d7ddc4c408114ae8d4b932a1a P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 01:54:49.900732 26643 raft_consensus.cc:504] T d4b8975c68d64c2d804fcc852384afde P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.901918 26643 leader_election.cc:288] T d4b8975c68d64c2d804fcc852384afde P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 5 election: Requested vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:49.901988 26379 raft_consensus.cc:2396] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 83c50029f7a94cfca6d1ee68e0728aa2 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 01:54:49.902271 26772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4b8975c68d64c2d804fcc852384afde" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:49.902269 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4b8975c68d64c2d804fcc852384afde" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:49.902480 26511 raft_consensus.cc:3037] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:49.902539 26772 raft_consensus.cc:3032] T d4b8975c68d64c2d804fcc852384afde P 401942254c4c43a1ba02c4ae68d17320 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 01:54:49.902693 26772 raft_consensus.cc:729] T d4b8975c68d64c2d804fcc852384afde P 401942254c4c43a1ba02c4ae68d17320 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Running, Role: LEADER
I20240502 01:54:49.903050 26772 consensus_queue.cc:260] T d4b8975c68d64c2d804fcc852384afde P 401942254c4c43a1ba02c4ae68d17320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.903501 26772 raft_consensus.cc:3037] T d4b8975c68d64c2d804fcc852384afde P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:49.904654 26643 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ef474f44fe5b4a06b2c8dda12b1bc496"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:49.904843 26643 raft_consensus.cc:482] T ef474f44fe5b4a06b2c8dda12b1bc496 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.904994 26643 raft_consensus.cc:3037] T ef474f44fe5b4a06b2c8dda12b1bc496 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:49.909467 26643 raft_consensus.cc:504] T ef474f44fe5b4a06b2c8dda12b1bc496 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.910274 26643 leader_election.cc:288] T ef474f44fe5b4a06b2c8dda12b1bc496 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 5 election: Requested vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:49.911063 26511 raft_consensus.cc:2396] T d4b8975c68d64c2d804fcc852384afde P 4bc1541b47df446297a369de05163012 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 83c50029f7a94cfca6d1ee68e0728aa2 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 01:54:49.910945 26234 catalog_manager.cc:5484] T cf09a605f32845a3aaad81d79063070b P 83c50029f7a94cfca6d1ee68e0728aa2 reported cstate change: term changed from 3 to 5, leader changed from 5e9d825f803c40e089b64e319beccf6e (127.11.42.65) to 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67). New cstate: current_term: 5 leader_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.911608 26774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef474f44fe5b4a06b2c8dda12b1bc496" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:49.911790 26774 raft_consensus.cc:3037] T ef474f44fe5b4a06b2c8dda12b1bc496 P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:49.913179 26772 raft_consensus.cc:2396] T d4b8975c68d64c2d804fcc852384afde P 401942254c4c43a1ba02c4ae68d17320 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 83c50029f7a94cfca6d1ee68e0728aa2 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 01:54:49.914530 26774 raft_consensus.cc:2454] T ef474f44fe5b4a06b2c8dda12b1bc496 P 401942254c4c43a1ba02c4ae68d17320 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 5.
I20240502 01:54:49.915083 26578 leader_election.cc:302] T ef474f44fe5b4a06b2c8dda12b1bc496 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:49.915357 28762 raft_consensus.cc:2781] T ef474f44fe5b4a06b2c8dda12b1bc496 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 01:54:49.915511 28762 raft_consensus.cc:686] T ef474f44fe5b4a06b2c8dda12b1bc496 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 LEADER]: Becoming Leader. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:49.915753 28762 consensus_queue.cc:237] T ef474f44fe5b4a06b2c8dda12b1bc496 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.916759 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef474f44fe5b4a06b2c8dda12b1bc496" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:49.917133 26379 raft_consensus.cc:3037] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:49.922669 26379 raft_consensus.cc:2396] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 83c50029f7a94cfca6d1ee68e0728aa2 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 01:54:49.927232 26643 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "14a747ac14354fdab2a2993594d3398f"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:49.927435 26643 raft_consensus.cc:482] T 14a747ac14354fdab2a2993594d3398f P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.927536 26643 raft_consensus.cc:3037] T 14a747ac14354fdab2a2993594d3398f P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:49.928901 26234 catalog_manager.cc:5484] T ef474f44fe5b4a06b2c8dda12b1bc496 P 83c50029f7a94cfca6d1ee68e0728aa2 reported cstate change: term changed from 3 to 5, leader changed from 5e9d825f803c40e089b64e319beccf6e (127.11.42.65) to 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67). New cstate: current_term: 5 leader_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.929522 26578 leader_election.cc:302] T d4b8975c68d64c2d804fcc852384afde P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83c50029f7a94cfca6d1ee68e0728aa2; no voters: 401942254c4c43a1ba02c4ae68d17320, 4bc1541b47df446297a369de05163012
I20240502 01:54:49.929917 28755 raft_consensus.cc:2726] T d4b8975c68d64c2d804fcc852384afde P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240502 01:54:49.930364 26643 raft_consensus.cc:504] T 14a747ac14354fdab2a2993594d3398f P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.931627 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a747ac14354fdab2a2993594d3398f" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:49.931839 26511 raft_consensus.cc:3037] T 14a747ac14354fdab2a2993594d3398f P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:49.932811 26774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a747ac14354fdab2a2993594d3398f" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:49.933085 26774 raft_consensus.cc:3037] T 14a747ac14354fdab2a2993594d3398f P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:49.936041 26774 raft_consensus.cc:2454] T 14a747ac14354fdab2a2993594d3398f P 401942254c4c43a1ba02c4ae68d17320 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 5.
I20240502 01:54:49.936548 26578 leader_election.cc:302] T 14a747ac14354fdab2a2993594d3398f P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:49.936836 28755 raft_consensus.cc:2781] T 14a747ac14354fdab2a2993594d3398f P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 01:54:49.937042 28755 raft_consensus.cc:686] T 14a747ac14354fdab2a2993594d3398f P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 LEADER]: Becoming Leader. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:49.937098 26643 leader_election.cc:288] T 14a747ac14354fdab2a2993594d3398f P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 5 election: Requested vote from peers 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:49.937326 28755 consensus_queue.cc:237] T 14a747ac14354fdab2a2993594d3398f P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.937887 26511 raft_consensus.cc:2396] T 14a747ac14354fdab2a2993594d3398f P 4bc1541b47df446297a369de05163012 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 83c50029f7a94cfca6d1ee68e0728aa2 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 01:54:49.938391 26643 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3d751cdd58eb4cef8039fd4c50855e99"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:49.938558 26643 raft_consensus.cc:482] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.938683 26643 raft_consensus.cc:3037] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.940949 26236 catalog_manager.cc:5484] T 14a747ac14354fdab2a2993594d3398f P 83c50029f7a94cfca6d1ee68e0728aa2 reported cstate change: term changed from 3 to 5, leader changed from 4bc1541b47df446297a369de05163012 (127.11.42.66) to 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67). New cstate: current_term: 5 leader_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.942428 26643 raft_consensus.cc:504] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.943681 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d751cdd58eb4cef8039fd4c50855e99" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:49.943884 26379 raft_consensus.cc:3037] T 3d751cdd58eb4cef8039fd4c50855e99 P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.945217 26774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d751cdd58eb4cef8039fd4c50855e99" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:49.945426 26774 raft_consensus.cc:3032] T 3d751cdd58eb4cef8039fd4c50855e99 P 401942254c4c43a1ba02c4ae68d17320 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:54:49.945533 26774 raft_consensus.cc:729] T 3d751cdd58eb4cef8039fd4c50855e99 P 401942254c4c43a1ba02c4ae68d17320 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Running, Role: LEADER
I20240502 01:54:49.945829 26774 consensus_queue.cc:260] T 3d751cdd58eb4cef8039fd4c50855e99 P 401942254c4c43a1ba02c4ae68d17320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.946131 26774 raft_consensus.cc:3037] T 3d751cdd58eb4cef8039fd4c50855e99 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.948734 26774 raft_consensus.cc:2396] T 3d751cdd58eb4cef8039fd4c50855e99 P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 83c50029f7a94cfca6d1ee68e0728aa2 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 01:54:49.950742 26379 raft_consensus.cc:2454] T 3d751cdd58eb4cef8039fd4c50855e99 P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 4.
I20240502 01:54:49.950913 26643 leader_election.cc:288] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 4 election: Requested vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:49.951210 26577 leader_election.cc:302] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e9d825f803c40e089b64e319beccf6e, 83c50029f7a94cfca6d1ee68e0728aa2; no voters: 401942254c4c43a1ba02c4ae68d17320
I20240502 01:54:49.951738 28762 raft_consensus.cc:2781] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 01:54:49.951881 28762 raft_consensus.cc:686] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 LEADER]: Becoming Leader. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:49.952152 28762 consensus_queue.cc:237] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.953789 26643 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9ead21d62e564a37a8c3d5a452621d13"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:49.954943 26643 raft_consensus.cc:482] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.955106 26643 raft_consensus.cc:3037] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.954877 26236 catalog_manager.cc:5484] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2 reported cstate change: term changed from 3 to 4, leader changed from 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68) to 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67). New cstate: current_term: 4 leader_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.959373 26643 raft_consensus.cc:504] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.960000 26643 leader_election.cc:288] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 4 election: Requested vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085)
I20240502 01:54:49.960480 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ead21d62e564a37a8c3d5a452621d13" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:49.960480 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ead21d62e564a37a8c3d5a452621d13" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:49.960702 26379 raft_consensus.cc:3037] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.960702 26511 raft_consensus.cc:3032] T 9ead21d62e564a37a8c3d5a452621d13 P 4bc1541b47df446297a369de05163012 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:54:49.960789 26511 raft_consensus.cc:729] T 9ead21d62e564a37a8c3d5a452621d13 P 4bc1541b47df446297a369de05163012 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:49.960832 26643 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1c47a74a6dc54d5eb0b47818d816b7fd"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:49.960995 26643 raft_consensus.cc:482] T 1c47a74a6dc54d5eb0b47818d816b7fd P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.961128 26643 raft_consensus.cc:3037] T 1c47a74a6dc54d5eb0b47818d816b7fd P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:49.961107 26511 consensus_queue.cc:260] T 9ead21d62e564a37a8c3d5a452621d13 P 4bc1541b47df446297a369de05163012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.961474 26511 raft_consensus.cc:3037] T 9ead21d62e564a37a8c3d5a452621d13 P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.963022 26643 raft_consensus.cc:504] T 1c47a74a6dc54d5eb0b47818d816b7fd P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.963280 26379 raft_consensus.cc:2454] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 4.
I20240502 01:54:49.963557 26643 leader_election.cc:288] T 1c47a74a6dc54d5eb0b47818d816b7fd P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 5 election: Requested vote from peers 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:49.963728 26577 leader_election.cc:302] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e9d825f803c40e089b64e319beccf6e, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:49.964000 28762 raft_consensus.cc:2781] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 01:54:49.964057 26508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c47a74a6dc54d5eb0b47818d816b7fd" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:49.964159 28762 raft_consensus.cc:686] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 LEADER]: Becoming Leader. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:49.964227 26508 raft_consensus.cc:3037] T 1c47a74a6dc54d5eb0b47818d816b7fd P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:49.964430 28762 consensus_queue.cc:237] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:49.964632 26643 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "daef5905fc19473a8728ccd2627270af"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:49.964681 26774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c47a74a6dc54d5eb0b47818d816b7fd" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:49.964814 26643 raft_consensus.cc:482] T daef5905fc19473a8728ccd2627270af P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.964854 26774 raft_consensus.cc:3032] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 01:54:49.964912 26643 raft_consensus.cc:3037] T daef5905fc19473a8728ccd2627270af P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:49.964943 26774 raft_consensus.cc:729] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Running, Role: LEADER
I20240502 01:54:49.965237 26774 consensus_queue.cc:260] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:49.965539 26774 raft_consensus.cc:3037] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:49.966668 26508 raft_consensus.cc:2396] T 1c47a74a6dc54d5eb0b47818d816b7fd P 4bc1541b47df446297a369de05163012 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 83c50029f7a94cfca6d1ee68e0728aa2 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 01:54:49.967509 26511 raft_consensus.cc:2454] T 9ead21d62e564a37a8c3d5a452621d13 P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 4.
I20240502 01:54:49.967486 26236 catalog_manager.cc:5484] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2 reported cstate change: term changed from 3 to 4, leader changed from 4bc1541b47df446297a369de05163012 (127.11.42.66) to 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67). New cstate: current_term: 4 leader_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:49.968472 26643 raft_consensus.cc:504] T daef5905fc19473a8728ccd2627270af P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.968660 26774 raft_consensus.cc:2396] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 83c50029f7a94cfca6d1ee68e0728aa2 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 01:54:49.969050 26643 leader_election.cc:288] T daef5905fc19473a8728ccd2627270af P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 5 election: Requested vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:49.969111 26578 leader_election.cc:302] T 1c47a74a6dc54d5eb0b47818d816b7fd P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83c50029f7a94cfca6d1ee68e0728aa2; no voters: 401942254c4c43a1ba02c4ae68d17320, 4bc1541b47df446297a369de05163012
I20240502 01:54:49.969797 26772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daef5905fc19473a8728ccd2627270af" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:49.969988 26772 raft_consensus.cc:3037] T daef5905fc19473a8728ccd2627270af P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:49.970712 28762 raft_consensus.cc:2726] T 1c47a74a6dc54d5eb0b47818d816b7fd P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240502 01:54:49.971531 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daef5905fc19473a8728ccd2627270af" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:49.971699 26379 raft_consensus.cc:3037] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:49.972708 26772 raft_consensus.cc:2454] T daef5905fc19473a8728ccd2627270af P 401942254c4c43a1ba02c4ae68d17320 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 5.
I20240502 01:54:49.973887 26379 raft_consensus.cc:2396] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 83c50029f7a94cfca6d1ee68e0728aa2 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 01:54:49.974084 26578 leader_election.cc:302] T daef5905fc19473a8728ccd2627270af P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:49.974402 28762 raft_consensus.cc:2781] T daef5905fc19473a8728ccd2627270af P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 01:54:49.974563 28762 raft_consensus.cc:686] T daef5905fc19473a8728ccd2627270af P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 LEADER]: Becoming Leader. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:49.974777 28762 consensus_queue.cc:237] T daef5905fc19473a8728ccd2627270af P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.975445 26643 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c56cb13e76384d5f9a0e45be2f65c085"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:49.976346 26643 raft_consensus.cc:482] T c56cb13e76384d5f9a0e45be2f65c085 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.976480 26643 raft_consensus.cc:3037] T c56cb13e76384d5f9a0e45be2f65c085 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.978055 26236 catalog_manager.cc:5484] T daef5905fc19473a8728ccd2627270af P 83c50029f7a94cfca6d1ee68e0728aa2 reported cstate change: term changed from 3 to 5, leader changed from 5e9d825f803c40e089b64e319beccf6e (127.11.42.65) to 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67). New cstate: current_term: 5 leader_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:49.979070 26643 raft_consensus.cc:504] T c56cb13e76384d5f9a0e45be2f65c085 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.980700 26772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c56cb13e76384d5f9a0e45be2f65c085" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:49.980894 26772 raft_consensus.cc:3032] T c56cb13e76384d5f9a0e45be2f65c085 P 401942254c4c43a1ba02c4ae68d17320 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:54:49.980990 26772 raft_consensus.cc:729] T c56cb13e76384d5f9a0e45be2f65c085 P 401942254c4c43a1ba02c4ae68d17320 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Running, Role: LEADER
I20240502 01:54:49.981102 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c56cb13e76384d5f9a0e45be2f65c085" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:49.981317 26379 raft_consensus.cc:3037] T c56cb13e76384d5f9a0e45be2f65c085 P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.981256 26772 consensus_queue.cc:260] T c56cb13e76384d5f9a0e45be2f65c085 P 401942254c4c43a1ba02c4ae68d17320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.981552 26772 raft_consensus.cc:3037] T c56cb13e76384d5f9a0e45be2f65c085 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.983465 26379 raft_consensus.cc:2396] T c56cb13e76384d5f9a0e45be2f65c085 P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 83c50029f7a94cfca6d1ee68e0728aa2 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.984333 26772 raft_consensus.cc:2396] T c56cb13e76384d5f9a0e45be2f65c085 P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 83c50029f7a94cfca6d1ee68e0728aa2 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:49.984745 26578 leader_election.cc:302] T c56cb13e76384d5f9a0e45be2f65c085 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83c50029f7a94cfca6d1ee68e0728aa2; no voters: 401942254c4c43a1ba02c4ae68d17320, 5e9d825f803c40e089b64e319beccf6e
I20240502 01:54:49.985394 26643 leader_election.cc:288] T c56cb13e76384d5f9a0e45be2f65c085 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 4 election: Requested vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:49.985661 28762 raft_consensus.cc:2726] T c56cb13e76384d5f9a0e45be2f65c085 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 01:54:49.986543 26643 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d9dd99b54b7c4d19a80c229bf99d7976"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:49.986733 26643 raft_consensus.cc:482] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:49.986822 26643 raft_consensus.cc:3037] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.989567 28680 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:49.990672 26643 raft_consensus.cc:504] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.991628 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9dd99b54b7c4d19a80c229bf99d7976" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:49.991878 26511 raft_consensus.cc:3032] T d9dd99b54b7c4d19a80c229bf99d7976 P 4bc1541b47df446297a369de05163012 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:54:49.991813 26772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9dd99b54b7c4d19a80c229bf99d7976" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:49.991955 26511 raft_consensus.cc:729] T d9dd99b54b7c4d19a80c229bf99d7976 P 4bc1541b47df446297a369de05163012 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:49.992000 26772 raft_consensus.cc:3037] T d9dd99b54b7c4d19a80c229bf99d7976 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.992223 26511 consensus_queue.cc:260] T d9dd99b54b7c4d19a80c229bf99d7976 P 4bc1541b47df446297a369de05163012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.992586 26511 raft_consensus.cc:3037] T d9dd99b54b7c4d19a80c229bf99d7976 P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:49.996023 26772 raft_consensus.cc:2454] T d9dd99b54b7c4d19a80c229bf99d7976 P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 4.
I20240502 01:54:49.996435 26578 leader_election.cc:302] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:49.996702 28762 raft_consensus.cc:2781] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 01:54:49.996874 28762 raft_consensus.cc:686] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 LEADER]: Becoming Leader. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:49.997180 28762 consensus_queue.cc:237] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:49.997296 26643 leader_election.cc:288] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 4 election: Requested vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:49.996711 26511 raft_consensus.cc:2454] T d9dd99b54b7c4d19a80c229bf99d7976 P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 4.
I20240502 01:54:50.000262 26236 catalog_manager.cc:5484] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 reported cstate change: term changed from 3 to 4, leader changed from 4bc1541b47df446297a369de05163012 (127.11.42.66) to 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67). New cstate: current_term: 4 leader_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:50.013482 26752 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:54:50.017670 26640 raft_consensus.cc:1261] T 23dcaac2d7d14adf820d826f05c1abc6 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Refusing update from remote peer 4bc1541b47df446297a369de05163012: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 01:54:50.018333 26643 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "23dcaac2d7d14adf820d826f05c1abc6"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:50.018520 26643 raft_consensus.cc:482] T 23dcaac2d7d14adf820d826f05c1abc6 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:50.018637 26643 raft_consensus.cc:3037] T 23dcaac2d7d14adf820d826f05c1abc6 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:50.020037 26379 raft_consensus.cc:1261] T beff61f07ecf4ffab508da651a686c8e P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Refusing update from remote peer 4bc1541b47df446297a369de05163012: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 01:54:50.020151 26379 pending_rounds.cc:76] T beff61f07ecf4ffab508da651a686c8e P 5e9d825f803c40e089b64e319beccf6e: Aborting all ops after (but not including) 3
I20240502 01:54:50.020222 26379 pending_rounds.cc:97] T beff61f07ecf4ffab508da651a686c8e P 5e9d825f803c40e089b64e319beccf6e: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 01:54:50.020282 26379 raft_consensus.cc:2866] T beff61f07ecf4ffab508da651a686c8e P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 01:54:50.021873 26643 raft_consensus.cc:504] T 23dcaac2d7d14adf820d826f05c1abc6 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:50.022099 28770 consensus_queue.cc:1035] T 23dcaac2d7d14adf820d826f05c1abc6 P 4bc1541b47df446297a369de05163012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:50.023022 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23dcaac2d7d14adf820d826f05c1abc6" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:50.023139 26643 leader_election.cc:288] T 23dcaac2d7d14adf820d826f05c1abc6 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 4 election: Requested vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:50.023237 26379 raft_consensus.cc:3037] T 23dcaac2d7d14adf820d826f05c1abc6 P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:50.026329 26640 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "beff61f07ecf4ffab508da651a686c8e"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:50.026520 26640 raft_consensus.cc:482] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:50.026634 26640 raft_consensus.cc:3037] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:50.029004 26640 raft_consensus.cc:504] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:50.030172 26508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beff61f07ecf4ffab508da651a686c8e" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:50.030428 26508 raft_consensus.cc:3032] T beff61f07ecf4ffab508da651a686c8e P 4bc1541b47df446297a369de05163012 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:54:50.030505 26508 raft_consensus.cc:729] T beff61f07ecf4ffab508da651a686c8e P 4bc1541b47df446297a369de05163012 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:50.030782 26508 consensus_queue.cc:260] T beff61f07ecf4ffab508da651a686c8e P 4bc1541b47df446297a369de05163012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:50.031085 26508 raft_consensus.cc:3037] T beff61f07ecf4ffab508da651a686c8e P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:50.033309 26643 raft_consensus.cc:1226] T 23dcaac2d7d14adf820d826f05c1abc6 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Rejecting Update request from peer 4bc1541b47df446297a369de05163012 for earlier term 3. Current term is 4. Ops: [3.4-3.4]
I20240502 01:54:50.034495 26508 raft_consensus.cc:2396] T beff61f07ecf4ffab508da651a686c8e P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 83c50029f7a94cfca6d1ee68e0728aa2 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:50.038491 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beff61f07ecf4ffab508da651a686c8e" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:50.038736 26378 raft_consensus.cc:3037] T beff61f07ecf4ffab508da651a686c8e P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:50.039158 26379 raft_consensus.cc:2396] T 23dcaac2d7d14adf820d826f05c1abc6 P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 83c50029f7a94cfca6d1ee68e0728aa2 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:50.039453 26640 leader_election.cc:288] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 4 election: Requested vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085)
I20240502 01:54:50.039814 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23dcaac2d7d14adf820d826f05c1abc6" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:50.040243 26511 raft_consensus.cc:3032] T 23dcaac2d7d14adf820d826f05c1abc6 P 4bc1541b47df446297a369de05163012 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:54:50.040343 26511 raft_consensus.cc:729] T 23dcaac2d7d14adf820d826f05c1abc6 P 4bc1541b47df446297a369de05163012 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:50.040638 26511 consensus_queue.cc:260] T 23dcaac2d7d14adf820d826f05c1abc6 P 4bc1541b47df446297a369de05163012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:50.040927 26511 raft_consensus.cc:3037] T 23dcaac2d7d14adf820d826f05c1abc6 P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:50.040799 26640 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b4ba7c4fe9614ab3abb3645c1da3fcd3"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:50.041386 26378 raft_consensus.cc:2454] T beff61f07ecf4ffab508da651a686c8e P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 4.
I20240502 01:54:50.041555 26640 raft_consensus.cc:482] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:50.041719 26640 raft_consensus.cc:3037] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.043747 26511 raft_consensus.cc:2396] T 23dcaac2d7d14adf820d826f05c1abc6 P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 83c50029f7a94cfca6d1ee68e0728aa2 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:50.043790 26577 leader_election.cc:302] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e9d825f803c40e089b64e319beccf6e, 83c50029f7a94cfca6d1ee68e0728aa2; no voters: 4bc1541b47df446297a369de05163012
I20240502 01:54:50.044286 28762 raft_consensus.cc:2781] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 01:54:50.044466 28762 raft_consensus.cc:686] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 LEADER]: Becoming Leader. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:50.044731 28762 consensus_queue.cc:237] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:50.049669 26640 raft_consensus.cc:504] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:50.050272 26640 leader_election.cc:288] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 5 election: Requested vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:50.050760 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ba7c4fe9614ab3abb3645c1da3fcd3" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:50.050966 26511 raft_consensus.cc:3037] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.051396 26643 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d2287fdff0db43df90ff5580c0430b8c"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:50.051563 26643 raft_consensus.cc:482] T d2287fdff0db43df90ff5580c0430b8c P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:50.051666 26643 raft_consensus.cc:3037] T d2287fdff0db43df90ff5580c0430b8c P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.054131 26577 leader_election.cc:302] T 23dcaac2d7d14adf820d826f05c1abc6 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83c50029f7a94cfca6d1ee68e0728aa2; no voters: 4bc1541b47df446297a369de05163012, 5e9d825f803c40e089b64e319beccf6e
I20240502 01:54:50.054278 26643 raft_consensus.cc:504] T d2287fdff0db43df90ff5580c0430b8c P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:50.054719 26511 raft_consensus.cc:2454] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 4bc1541b47df446297a369de05163012 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 5.
I20240502 01:54:50.054900 26643 leader_election.cc:288] T d2287fdff0db43df90ff5580c0430b8c P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 5 election: Requested vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:50.055147 28755 raft_consensus.cc:2726] T 23dcaac2d7d14adf820d826f05c1abc6 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 01:54:50.055171 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ba7c4fe9614ab3abb3645c1da3fcd3" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:50.055446 26379 raft_consensus.cc:3037] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.056293 26578 leader_election.cc:302] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:50.056375 26359 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:50.056596 28755 raft_consensus.cc:2781] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 01:54:50.056756 28755 raft_consensus.cc:686] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 LEADER]: Becoming Leader. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:50.057003 28755 consensus_queue.cc:237] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:50.058511 26643 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "53b7377c0d144e07a4501c2d626f6494"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:50.058734 26379 raft_consensus.cc:2396] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 83c50029f7a94cfca6d1ee68e0728aa2 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 01:54:50.058727 26643 raft_consensus.cc:482] T 53b7377c0d144e07a4501c2d626f6494 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:50.058880 26643 raft_consensus.cc:3037] T 53b7377c0d144e07a4501c2d626f6494 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:50.059115 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2287fdff0db43df90ff5580c0430b8c" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:50.059340 26511 raft_consensus.cc:3037] T d2287fdff0db43df90ff5580c0430b8c P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.062204 26511 raft_consensus.cc:2454] T d2287fdff0db43df90ff5580c0430b8c P 4bc1541b47df446297a369de05163012 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 5.
I20240502 01:54:50.062666 26578 leader_election.cc:302] T d2287fdff0db43df90ff5580c0430b8c P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:50.062985 28772 raft_consensus.cc:2781] T d2287fdff0db43df90ff5580c0430b8c P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 01:54:50.063135 28772 raft_consensus.cc:686] T d2287fdff0db43df90ff5580c0430b8c P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 LEADER]: Becoming Leader. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:50.063438 28772 consensus_queue.cc:237] T d2287fdff0db43df90ff5580c0430b8c P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:50.063701 26643 raft_consensus.cc:504] T 53b7377c0d144e07a4501c2d626f6494 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:50.064224 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2287fdff0db43df90ff5580c0430b8c" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:50.064419 26378 raft_consensus.cc:3037] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.065773 26772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b7377c0d144e07a4501c2d626f6494" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:50.066026 26772 raft_consensus.cc:3032] T 53b7377c0d144e07a4501c2d626f6494 P 401942254c4c43a1ba02c4ae68d17320 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:54:50.066113 26772 raft_consensus.cc:729] T 53b7377c0d144e07a4501c2d626f6494 P 401942254c4c43a1ba02c4ae68d17320 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Running, Role: LEADER
I20240502 01:54:50.066392 26772 consensus_queue.cc:260] T 53b7377c0d144e07a4501c2d626f6494 P 401942254c4c43a1ba02c4ae68d17320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:50.066685 26772 raft_consensus.cc:3037] T 53b7377c0d144e07a4501c2d626f6494 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:50.069238 26643 leader_election.cc:288] T 53b7377c0d144e07a4501c2d626f6494 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 4 election: Requested vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:50.069316 26377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b7377c0d144e07a4501c2d626f6494" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:50.069518 26377 raft_consensus.cc:3037] T 53b7377c0d144e07a4501c2d626f6494 P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:50.069545 26772 raft_consensus.cc:2396] T 53b7377c0d144e07a4501c2d626f6494 P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 83c50029f7a94cfca6d1ee68e0728aa2 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:50.070911 26640 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "99a4b7d022e1461a8233952ab47ef5d3"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:50.071094 26640 raft_consensus.cc:482] T 99a4b7d022e1461a8233952ab47ef5d3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:50.071200 26640 raft_consensus.cc:3037] T 99a4b7d022e1461a8233952ab47ef5d3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.072031 26377 raft_consensus.cc:2396] T 53b7377c0d144e07a4501c2d626f6494 P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 83c50029f7a94cfca6d1ee68e0728aa2 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:50.073980 26640 raft_consensus.cc:504] T 99a4b7d022e1461a8233952ab47ef5d3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:50.074453 26577 leader_election.cc:302] T 53b7377c0d144e07a4501c2d626f6494 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83c50029f7a94cfca6d1ee68e0728aa2; no voters: 401942254c4c43a1ba02c4ae68d17320, 5e9d825f803c40e089b64e319beccf6e
I20240502 01:54:50.075057 28772 raft_consensus.cc:2726] T 53b7377c0d144e07a4501c2d626f6494 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 01:54:50.077468 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a4b7d022e1461a8233952ab47ef5d3" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:50.078066 26640 leader_election.cc:288] T 99a4b7d022e1461a8233952ab47ef5d3 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 5 election: Requested vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085)
I20240502 01:54:50.078327 26511 raft_consensus.cc:3037] T 99a4b7d022e1461a8233952ab47ef5d3 P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.079190 26377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a4b7d022e1461a8233952ab47ef5d3" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:50.079449 26377 raft_consensus.cc:3037] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.080372 26236 catalog_manager.cc:5484] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2 reported cstate change: term changed from 3 to 4, leader changed from 4bc1541b47df446297a369de05163012 (127.11.42.66) to 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67). New cstate: current_term: 4 leader_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:50.081895 26377 raft_consensus.cc:2396] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 83c50029f7a94cfca6d1ee68e0728aa2 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 01:54:50.082633 26378 raft_consensus.cc:2396] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 83c50029f7a94cfca6d1ee68e0728aa2 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 01:54:50.085434 26643 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0ddd395eec1146bc8a20292d81be4868"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:50.085620 26643 raft_consensus.cc:482] T 0ddd395eec1146bc8a20292d81be4868 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:50.085762 26643 raft_consensus.cc:3037] T 0ddd395eec1146bc8a20292d81be4868 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.088622 26643 raft_consensus.cc:504] T 0ddd395eec1146bc8a20292d81be4868 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:50.090618 26236 catalog_manager.cc:5484] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 83c50029f7a94cfca6d1ee68e0728aa2 reported cstate change: term changed from 3 to 5, leader changed from 5e9d825f803c40e089b64e319beccf6e (127.11.42.65) to 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67). New cstate: current_term: 5 leader_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:50.091199 26236 catalog_manager.cc:5484] T d2287fdff0db43df90ff5580c0430b8c P 83c50029f7a94cfca6d1ee68e0728aa2 reported cstate change: term changed from 3 to 5, leader changed from 5e9d825f803c40e089b64e319beccf6e (127.11.42.65) to 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67). New cstate: current_term: 5 leader_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:50.096211 26377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ddd395eec1146bc8a20292d81be4868" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:50.096526 26377 raft_consensus.cc:3037] T 0ddd395eec1146bc8a20292d81be4868 P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.096526 26772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ddd395eec1146bc8a20292d81be4868" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:50.096787 26772 raft_consensus.cc:3037] T 0ddd395eec1146bc8a20292d81be4868 P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.099443 26772 raft_consensus.cc:2454] T 0ddd395eec1146bc8a20292d81be4868 P 401942254c4c43a1ba02c4ae68d17320 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 5.
I20240502 01:54:50.099972 26578 leader_election.cc:302] T 0ddd395eec1146bc8a20292d81be4868 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:50.100298 28755 raft_consensus.cc:2781] T 0ddd395eec1146bc8a20292d81be4868 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 01:54:50.100440 28755 raft_consensus.cc:686] T 0ddd395eec1146bc8a20292d81be4868 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 LEADER]: Becoming Leader. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:50.100723 28755 consensus_queue.cc:237] T 0ddd395eec1146bc8a20292d81be4868 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:50.103722 26643 leader_election.cc:288] T 0ddd395eec1146bc8a20292d81be4868 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 5 election: Requested vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:50.079209 26489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:50.099519 26377 raft_consensus.cc:2396] T 0ddd395eec1146bc8a20292d81be4868 P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 83c50029f7a94cfca6d1ee68e0728aa2 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 01:54:50.111786 26511 raft_consensus.cc:2454] T 99a4b7d022e1461a8233952ab47ef5d3 P 4bc1541b47df446297a369de05163012 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 5.
I20240502 01:54:50.112250 26578 leader_election.cc:302] T 99a4b7d022e1461a8233952ab47ef5d3 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4bc1541b47df446297a369de05163012, 83c50029f7a94cfca6d1ee68e0728aa2; no voters: 5e9d825f803c40e089b64e319beccf6e
I20240502 01:54:50.112668 28755 raft_consensus.cc:2781] T 99a4b7d022e1461a8233952ab47ef5d3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 01:54:50.112913 28755 raft_consensus.cc:686] T 99a4b7d022e1461a8233952ab47ef5d3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 LEADER]: Becoming Leader. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:50.113183 28755 consensus_queue.cc:237] T 99a4b7d022e1461a8233952ab47ef5d3 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:50.113224 26236 catalog_manager.cc:5484] T 0ddd395eec1146bc8a20292d81be4868 P 83c50029f7a94cfca6d1ee68e0728aa2 reported cstate change: term changed from 3 to 5, leader changed from 5e9d825f803c40e089b64e319beccf6e (127.11.42.65) to 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67). New cstate: current_term: 5 leader_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:50.118089 26640 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2fd06590221d414b9e896e02b1be6f3e"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:50.118335 26640 raft_consensus.cc:482] T 2fd06590221d414b9e896e02b1be6f3e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:50.118528 26640 raft_consensus.cc:3037] T 2fd06590221d414b9e896e02b1be6f3e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.121419 26640 raft_consensus.cc:504] T 2fd06590221d414b9e896e02b1be6f3e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:50.122443 26377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fd06590221d414b9e896e02b1be6f3e" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:50.122715 26377 raft_consensus.cc:3037] T 2fd06590221d414b9e896e02b1be6f3e P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.124419 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fd06590221d414b9e896e02b1be6f3e" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:50.124650 26511 raft_consensus.cc:3037] T 2fd06590221d414b9e896e02b1be6f3e P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.126183 26640 leader_election.cc:288] T 2fd06590221d414b9e896e02b1be6f3e P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 5 election: Requested vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:50.128690 26511 raft_consensus.cc:2454] T 2fd06590221d414b9e896e02b1be6f3e P 4bc1541b47df446297a369de05163012 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 5.
I20240502 01:54:50.129174 26578 leader_election.cc:302] T 2fd06590221d414b9e896e02b1be6f3e P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:50.129496 28762 raft_consensus.cc:2781] T 2fd06590221d414b9e896e02b1be6f3e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 01:54:50.129634 28762 raft_consensus.cc:686] T 2fd06590221d414b9e896e02b1be6f3e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 LEADER]: Becoming Leader. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:50.129869 28762 consensus_queue.cc:237] T 2fd06590221d414b9e896e02b1be6f3e P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:50.135190 28548 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:50.139248 26236 catalog_manager.cc:5484] T 99a4b7d022e1461a8233952ab47ef5d3 P 83c50029f7a94cfca6d1ee68e0728aa2 reported cstate change: term changed from 3 to 5, leader changed from 5e9d825f803c40e089b64e319beccf6e (127.11.42.65) to 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67). New cstate: current_term: 5 leader_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:50.145170 26640 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "89107a545fcc48aead1e199da1f78311"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:50.145370 26640 raft_consensus.cc:482] T 89107a545fcc48aead1e199da1f78311 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:50.145489 26640 raft_consensus.cc:3037] T 89107a545fcc48aead1e199da1f78311 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:50.145148 26236 catalog_manager.cc:5484] T 2fd06590221d414b9e896e02b1be6f3e P 83c50029f7a94cfca6d1ee68e0728aa2 reported cstate change: term changed from 3 to 5, leader changed from 5e9d825f803c40e089b64e319beccf6e (127.11.42.65) to 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67). New cstate: current_term: 5 leader_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:50.148362 26640 raft_consensus.cc:504] T 89107a545fcc48aead1e199da1f78311 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:50.149497 26772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89107a545fcc48aead1e199da1f78311" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:50.149703 26772 raft_consensus.cc:3032] T 89107a545fcc48aead1e199da1f78311 P 401942254c4c43a1ba02c4ae68d17320 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:54:50.149799 26772 raft_consensus.cc:729] T 89107a545fcc48aead1e199da1f78311 P 401942254c4c43a1ba02c4ae68d17320 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Running, Role: LEADER
I20240502 01:54:50.150048 26772 consensus_queue.cc:260] T 89107a545fcc48aead1e199da1f78311 P 401942254c4c43a1ba02c4ae68d17320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:50.150349 26772 raft_consensus.cc:3037] T 89107a545fcc48aead1e199da1f78311 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:50.153182 26772 raft_consensus.cc:2396] T 89107a545fcc48aead1e199da1f78311 P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 83c50029f7a94cfca6d1ee68e0728aa2 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:50.159337 26377 raft_consensus.cc:2396] T 2fd06590221d414b9e896e02b1be6f3e P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 83c50029f7a94cfca6d1ee68e0728aa2 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 01:54:50.165343 26623 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 01:54:50.168284 28417 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:50.169402 26640 leader_election.cc:288] T 89107a545fcc48aead1e199da1f78311 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 4 election: Requested vote from peers 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:50.170892 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89107a545fcc48aead1e199da1f78311" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:50.171128 26511 raft_consensus.cc:3037] T 89107a545fcc48aead1e199da1f78311 P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:50.173921 26511 raft_consensus.cc:2396] T 89107a545fcc48aead1e199da1f78311 P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 83c50029f7a94cfca6d1ee68e0728aa2 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:50.178896 26578 leader_election.cc:302] T 89107a545fcc48aead1e199da1f78311 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83c50029f7a94cfca6d1ee68e0728aa2; no voters: 401942254c4c43a1ba02c4ae68d17320, 4bc1541b47df446297a369de05163012
I20240502 01:54:50.179227 28755 raft_consensus.cc:2726] T 89107a545fcc48aead1e199da1f78311 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 01:54:50.187355 26377 raft_consensus.cc:1261] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Refusing update from remote peer 401942254c4c43a1ba02c4ae68d17320: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 01:54:50.188041 28857 consensus_queue.cc:1035] T 245ca06c39774fb5b0f522550a63ac5b P 401942254c4c43a1ba02c4ae68d17320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:50.189139 26640 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dc82c30359e84bddb4bcfe1b3b13ced7"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:50.189350 26640 raft_consensus.cc:482] T dc82c30359e84bddb4bcfe1b3b13ced7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:50.189460 26640 raft_consensus.cc:3037] T dc82c30359e84bddb4bcfe1b3b13ced7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:50.192420 26640 raft_consensus.cc:504] T dc82c30359e84bddb4bcfe1b3b13ced7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:50.193722 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc82c30359e84bddb4bcfe1b3b13ced7" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:50.193941 26511 raft_consensus.cc:3037] T dc82c30359e84bddb4bcfe1b3b13ced7 P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:50.199108 26511 raft_consensus.cc:2396] T dc82c30359e84bddb4bcfe1b3b13ced7 P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 83c50029f7a94cfca6d1ee68e0728aa2 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:50.199822 26378 raft_consensus.cc:1261] T 37e818dc0b834eb185e27fcb3cb972e1 P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Refusing update from remote peer 401942254c4c43a1ba02c4ae68d17320: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 01:54:50.200408 28857 consensus_queue.cc:1035] T 37e818dc0b834eb185e27fcb3cb972e1 P 401942254c4c43a1ba02c4ae68d17320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:50.200765 28286 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:50.201033 26640 leader_election.cc:288] T dc82c30359e84bddb4bcfe1b3b13ced7 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 4 election: Requested vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:50.201776 26772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc82c30359e84bddb4bcfe1b3b13ced7" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:50.201978 26772 raft_consensus.cc:3032] T dc82c30359e84bddb4bcfe1b3b13ced7 P 401942254c4c43a1ba02c4ae68d17320 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:54:50.202076 26772 raft_consensus.cc:729] T dc82c30359e84bddb4bcfe1b3b13ced7 P 401942254c4c43a1ba02c4ae68d17320 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Running, Role: LEADER
I20240502 01:54:50.202342 26772 consensus_queue.cc:260] T dc82c30359e84bddb4bcfe1b3b13ced7 P 401942254c4c43a1ba02c4ae68d17320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:50.202637 26772 raft_consensus.cc:3037] T dc82c30359e84bddb4bcfe1b3b13ced7 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:50.207978 26772 raft_consensus.cc:2396] T dc82c30359e84bddb4bcfe1b3b13ced7 P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 83c50029f7a94cfca6d1ee68e0728aa2 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:50.208758 26578 leader_election.cc:302] T dc82c30359e84bddb4bcfe1b3b13ced7 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83c50029f7a94cfca6d1ee68e0728aa2; no voters: 401942254c4c43a1ba02c4ae68d17320, 4bc1541b47df446297a369de05163012
I20240502 01:54:50.209210 28755 raft_consensus.cc:2726] T dc82c30359e84bddb4bcfe1b3b13ced7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 01:54:50.213625 26640 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c26358e3969143fcbf2f466241b360a7"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:50.213857 26640 raft_consensus.cc:482] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:50.213991 26640 raft_consensus.cc:3037] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.219898 26511 raft_consensus.cc:1261] T 245ca06c39774fb5b0f522550a63ac5b P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Refusing update from remote peer 401942254c4c43a1ba02c4ae68d17320: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 01:54:50.219236 26640 raft_consensus.cc:504] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:50.220257 26511 pending_rounds.cc:76] T 245ca06c39774fb5b0f522550a63ac5b P 4bc1541b47df446297a369de05163012: Aborting all ops after (but not including) 3
I20240502 01:54:50.220942 26640 leader_election.cc:288] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 5 election: Requested vote from peers 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959), 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085)
I20240502 01:54:50.221207 26511 pending_rounds.cc:97] T 245ca06c39774fb5b0f522550a63ac5b P 4bc1541b47df446297a369de05163012: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 01:54:50.221263 26772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c26358e3969143fcbf2f466241b360a7" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:50.221367 26377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c26358e3969143fcbf2f466241b360a7" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:50.221501 26772 raft_consensus.cc:3037] T c26358e3969143fcbf2f466241b360a7 P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.221534 26511 raft_consensus.cc:2866] T 245ca06c39774fb5b0f522550a63ac5b P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 01:54:50.221550 26377 raft_consensus.cc:3037] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.222196 28871 consensus_queue.cc:1035] T 245ca06c39774fb5b0f522550a63ac5b P 401942254c4c43a1ba02c4ae68d17320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:50.223829 26640 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "22d4681165d1488db2c45ec83cb49cb4"
dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
from {username='slave'} at 127.0.0.1:51260
I20240502 01:54:50.224030 26640 raft_consensus.cc:482] T 22d4681165d1488db2c45ec83cb49cb4 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:50.224129 26640 raft_consensus.cc:3037] T 22d4681165d1488db2c45ec83cb49cb4 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.224295 26377 raft_consensus.cc:2454] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 5.
I20240502 01:54:50.225107 26577 leader_election.cc:302] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e9d825f803c40e089b64e319beccf6e, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:50.225489 28755 raft_consensus.cc:2781] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 01:54:50.225636 28755 raft_consensus.cc:686] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 LEADER]: Becoming Leader. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:50.225914 28755 consensus_queue.cc:237] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:50.227089 26640 raft_consensus.cc:504] T 22d4681165d1488db2c45ec83cb49cb4 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:50.228124 26640 leader_election.cc:288] T 22d4681165d1488db2c45ec83cb49cb4 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 5 election: Requested vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:50.229048 26508 raft_consensus.cc:1261] T 37e818dc0b834eb185e27fcb3cb972e1 P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Refusing update from remote peer 401942254c4c43a1ba02c4ae68d17320: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 01:54:50.229166 26508 pending_rounds.cc:76] T 37e818dc0b834eb185e27fcb3cb972e1 P 4bc1541b47df446297a369de05163012: Aborting all ops after (but not including) 3
I20240502 01:54:50.229097 26774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22d4681165d1488db2c45ec83cb49cb4" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:50.229218 26508 pending_rounds.cc:97] T 37e818dc0b834eb185e27fcb3cb972e1 P 4bc1541b47df446297a369de05163012: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 01:54:50.229266 26774 raft_consensus.cc:3037] T 22d4681165d1488db2c45ec83cb49cb4 P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.229293 26508 raft_consensus.cc:2866] T 37e818dc0b834eb185e27fcb3cb972e1 P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 01:54:50.229914 26508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22d4681165d1488db2c45ec83cb49cb4" candidate_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:50.230108 26508 raft_consensus.cc:3037] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.231840 26774 raft_consensus.cc:2454] T 22d4681165d1488db2c45ec83cb49cb4 P 401942254c4c43a1ba02c4ae68d17320 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 5.
I20240502 01:54:50.233388 26578 leader_election.cc:302] T 22d4681165d1488db2c45ec83cb49cb4 P 83c50029f7a94cfca6d1ee68e0728aa2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:50.233669 28762 raft_consensus.cc:2781] T 22d4681165d1488db2c45ec83cb49cb4 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 01:54:50.233785 28762 raft_consensus.cc:686] T 22d4681165d1488db2c45ec83cb49cb4 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 LEADER]: Becoming Leader. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:50.233809 26772 raft_consensus.cc:2454] T c26358e3969143fcbf2f466241b360a7 P 401942254c4c43a1ba02c4ae68d17320 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c50029f7a94cfca6d1ee68e0728aa2 in term 5.
I20240502 01:54:50.234040 28762 consensus_queue.cc:237] T 22d4681165d1488db2c45ec83cb49cb4 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:50.234906 27670 consensus_queue.cc:1035] T 37e818dc0b834eb185e27fcb3cb972e1 P 401942254c4c43a1ba02c4ae68d17320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:50.240275 26508 raft_consensus.cc:2396] T 22d4681165d1488db2c45ec83cb49cb4 P 4bc1541b47df446297a369de05163012 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 83c50029f7a94cfca6d1ee68e0728aa2 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 01:54:50.250479 26511 raft_consensus.cc:1261] T 8cf6da25f535446b9515444e093cae7f P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Refusing update from remote peer 401942254c4c43a1ba02c4ae68d17320: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 01:54:50.250600 26511 pending_rounds.cc:76] T 8cf6da25f535446b9515444e093cae7f P 4bc1541b47df446297a369de05163012: Aborting all ops after (but not including) 3
I20240502 01:54:50.250658 26511 pending_rounds.cc:97] T 8cf6da25f535446b9515444e093cae7f P 4bc1541b47df446297a369de05163012: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 01:54:50.250734 26511 raft_consensus.cc:2866] T 8cf6da25f535446b9515444e093cae7f P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 01:54:50.251274 28847 consensus_queue.cc:1035] T 8cf6da25f535446b9515444e093cae7f P 401942254c4c43a1ba02c4ae68d17320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:50.256038 26377 raft_consensus.cc:1261] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Refusing update from remote peer 401942254c4c43a1ba02c4ae68d17320: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 01:54:50.255345 26236 catalog_manager.cc:5484] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2 reported cstate change: term changed from 3 to 5, leader changed from 5e9d825f803c40e089b64e319beccf6e (127.11.42.65) to 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67). New cstate: current_term: 5 leader_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:50.256891 28857 consensus_queue.cc:1035] T 8cf6da25f535446b9515444e093cae7f P 401942254c4c43a1ba02c4ae68d17320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:50.258370 26377 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cf09a605f32845a3aaad81d79063070b"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:50.258553 26377 raft_consensus.cc:482] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:50.258670 26377 raft_consensus.cc:3037] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Advancing to term 6
I20240502 01:54:50.263716 26377 raft_consensus.cc:504] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:50.266014 26228 catalog_manager.cc:5484] T 22d4681165d1488db2c45ec83cb49cb4 P 83c50029f7a94cfca6d1ee68e0728aa2 reported cstate change: term changed from 3 to 5, leader changed from 4bc1541b47df446297a369de05163012 (127.11.42.66) to 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67). New cstate: current_term: 5 leader_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:50.267314 26377 leader_election.cc:288] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 6 election: Requested vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:50.267382 26640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf09a605f32845a3aaad81d79063070b" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:50.267573 26640 raft_consensus.cc:3032] T cf09a605f32845a3aaad81d79063070b P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 LEADER]: Stepping down as leader of term 5
I20240502 01:54:50.267669 26640 raft_consensus.cc:729] T cf09a605f32845a3aaad81d79063070b P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:50.267915 26640 consensus_queue.cc:260] T cf09a605f32845a3aaad81d79063070b P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:50.268224 26640 raft_consensus.cc:3037] T cf09a605f32845a3aaad81d79063070b P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Advancing to term 6
I20240502 01:54:50.270944 26640 raft_consensus.cc:2396] T cf09a605f32845a3aaad81d79063070b P 83c50029f7a94cfca6d1ee68e0728aa2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e9d825f803c40e089b64e319beccf6e for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 01:54:50.272194 26508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf09a605f32845a3aaad81d79063070b" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:50.272502 26508 raft_consensus.cc:3037] T cf09a605f32845a3aaad81d79063070b P 4bc1541b47df446297a369de05163012 [term 5 FOLLOWER]: Advancing to term 6
I20240502 01:54:50.273965 26378 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c56cb13e76384d5f9a0e45be2f65c085"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:50.274152 26378 raft_consensus.cc:482] T c56cb13e76384d5f9a0e45be2f65c085 P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:50.274286 26378 raft_consensus.cc:3037] T c56cb13e76384d5f9a0e45be2f65c085 P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.277419 26378 raft_consensus.cc:504] T c56cb13e76384d5f9a0e45be2f65c085 P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:50.278539 26640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c56cb13e76384d5f9a0e45be2f65c085" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:50.278600 26772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c56cb13e76384d5f9a0e45be2f65c085" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:50.278831 26772 raft_consensus.cc:3037] T c56cb13e76384d5f9a0e45be2f65c085 P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.278851 26640 raft_consensus.cc:3037] T c56cb13e76384d5f9a0e45be2f65c085 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.279331 26378 leader_election.cc:288] T c56cb13e76384d5f9a0e45be2f65c085 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 5 election: Requested vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:50.281457 26508 raft_consensus.cc:2454] T cf09a605f32845a3aaad81d79063070b P 4bc1541b47df446297a369de05163012 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 6.
I20240502 01:54:50.281543 26772 raft_consensus.cc:2396] T c56cb13e76384d5f9a0e45be2f65c085 P 401942254c4c43a1ba02c4ae68d17320 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e9d825f803c40e089b64e319beccf6e for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 01:54:50.281904 26314 leader_election.cc:302] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4bc1541b47df446297a369de05163012, 5e9d825f803c40e089b64e319beccf6e; no voters: 83c50029f7a94cfca6d1ee68e0728aa2
I20240502 01:54:50.282884 28794 raft_consensus.cc:2781] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [term 6 FOLLOWER]: Leader election won for term 6
I20240502 01:54:50.283046 28794 raft_consensus.cc:686] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [term 6 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:50.283308 28794 consensus_queue.cc:237] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:50.286643 26228 catalog_manager.cc:5484] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 5 to 6, leader changed from 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67) to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 6 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:50.287155 26378 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8cf6da25f535446b9515444e093cae7f"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:50.287416 26378 raft_consensus.cc:482] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:50.288329 26378 raft_consensus.cc:3037] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:50.289105 26640 raft_consensus.cc:2454] T c56cb13e76384d5f9a0e45be2f65c085 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 5.
I20240502 01:54:50.289602 26313 leader_election.cc:302] T c56cb13e76384d5f9a0e45be2f65c085 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e9d825f803c40e089b64e319beccf6e, 83c50029f7a94cfca6d1ee68e0728aa2; no voters: 401942254c4c43a1ba02c4ae68d17320
I20240502 01:54:50.290036 28794 raft_consensus.cc:2781] T c56cb13e76384d5f9a0e45be2f65c085 P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Leader election won for term 5
I20240502 01:54:50.290161 28794 raft_consensus.cc:686] T c56cb13e76384d5f9a0e45be2f65c085 P 5e9d825f803c40e089b64e319beccf6e [term 5 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:50.290390 28794 consensus_queue.cc:237] T c56cb13e76384d5f9a0e45be2f65c085 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:50.294997 26378 raft_consensus.cc:504] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:50.295523 26378 leader_election.cc:288] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 4 election: Requested vote from peers 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:50.296329 26772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cf6da25f535446b9515444e093cae7f" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:50.296525 26772 raft_consensus.cc:3032] T 8cf6da25f535446b9515444e093cae7f P 401942254c4c43a1ba02c4ae68d17320 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:54:50.296515 26377 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ef474f44fe5b4a06b2c8dda12b1bc496"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:50.296615 26772 raft_consensus.cc:729] T 8cf6da25f535446b9515444e093cae7f P 401942254c4c43a1ba02c4ae68d17320 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Running, Role: LEADER
I20240502 01:54:50.296696 26377 raft_consensus.cc:482] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:50.296795 26377 raft_consensus.cc:3037] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Advancing to term 6
I20240502 01:54:50.296897 26772 consensus_queue.cc:260] T 8cf6da25f535446b9515444e093cae7f P 401942254c4c43a1ba02c4ae68d17320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:50.297324 26772 raft_consensus.cc:3037] T 8cf6da25f535446b9515444e093cae7f P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:50.299335 26377 raft_consensus.cc:504] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:50.300060 26377 leader_election.cc:288] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 6 election: Requested vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:50.300159 26508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cf6da25f535446b9515444e093cae7f" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:50.300408 26508 raft_consensus.cc:3037] T 8cf6da25f535446b9515444e093cae7f P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:50.300957 26640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef474f44fe5b4a06b2c8dda12b1bc496" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:50.301079 26377 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "beff61f07ecf4ffab508da651a686c8e"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:50.301182 26640 raft_consensus.cc:3032] T ef474f44fe5b4a06b2c8dda12b1bc496 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 LEADER]: Stepping down as leader of term 5
I20240502 01:54:50.301254 26640 raft_consensus.cc:729] T ef474f44fe5b4a06b2c8dda12b1bc496 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:50.301250 26377 raft_consensus.cc:482] T beff61f07ecf4ffab508da651a686c8e P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:50.301371 26377 raft_consensus.cc:3037] T beff61f07ecf4ffab508da651a686c8e P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.302101 26640 consensus_queue.cc:260] T ef474f44fe5b4a06b2c8dda12b1bc496 P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:50.302386 26640 raft_consensus.cc:3037] T ef474f44fe5b4a06b2c8dda12b1bc496 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Advancing to term 6
I20240502 01:54:50.303078 26508 raft_consensus.cc:2454] T 8cf6da25f535446b9515444e093cae7f P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 4.
I20240502 01:54:50.303465 26314 leader_election.cc:302] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:50.304116 28795 raft_consensus.cc:2781] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Leader election won for term 4
I20240502 01:54:50.304497 28795 raft_consensus.cc:686] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e [term 4 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:50.304553 26774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef474f44fe5b4a06b2c8dda12b1bc496" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:50.304762 26774 raft_consensus.cc:3037] T ef474f44fe5b4a06b2c8dda12b1bc496 P 401942254c4c43a1ba02c4ae68d17320 [term 5 FOLLOWER]: Advancing to term 6
I20240502 01:54:50.304773 28795 consensus_queue.cc:237] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:50.306147 26377 raft_consensus.cc:504] T beff61f07ecf4ffab508da651a686c8e P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:50.307241 26774 raft_consensus.cc:2454] T ef474f44fe5b4a06b2c8dda12b1bc496 P 401942254c4c43a1ba02c4ae68d17320 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 6.
I20240502 01:54:50.307583 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beff61f07ecf4ffab508da651a686c8e" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:50.307808 26643 raft_consensus.cc:3032] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 01:54:50.307899 26643 raft_consensus.cc:729] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:50.308166 26643 consensus_queue.cc:260] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:50.308470 26643 raft_consensus.cc:3037] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.308633 26377 leader_election.cc:288] T beff61f07ecf4ffab508da651a686c8e P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 5 election: Requested vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:50.309490 26377 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4254942d7ddc4c408114ae8d4b932a1a"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:50.309677 26377 raft_consensus.cc:482] T 4254942d7ddc4c408114ae8d4b932a1a P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:50.309780 26377 raft_consensus.cc:3037] T 4254942d7ddc4c408114ae8d4b932a1a P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.309947 26507 raft_consensus.cc:1261] T c9e4788d50d04d129e8d03125073c6f2 P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Refusing update from remote peer 401942254c4c43a1ba02c4ae68d17320: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 01:54:50.310030 26507 pending_rounds.cc:76] T c9e4788d50d04d129e8d03125073c6f2 P 4bc1541b47df446297a369de05163012: Aborting all ops after (but not including) 3
I20240502 01:54:50.310091 26507 pending_rounds.cc:97] T c9e4788d50d04d129e8d03125073c6f2 P 4bc1541b47df446297a369de05163012: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 01:54:50.310142 26507 raft_consensus.cc:2866] T c9e4788d50d04d129e8d03125073c6f2 P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 01:54:50.310859 27670 consensus_queue.cc:1035] T c9e4788d50d04d129e8d03125073c6f2 P 401942254c4c43a1ba02c4ae68d17320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:50.311187 26643 raft_consensus.cc:2396] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e9d825f803c40e089b64e319beccf6e for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:50.312407 26377 raft_consensus.cc:504] T 4254942d7ddc4c408114ae8d4b932a1a P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:50.308455 26640 raft_consensus.cc:2396] T ef474f44fe5b4a06b2c8dda12b1bc496 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e9d825f803c40e089b64e319beccf6e for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 01:54:50.313859 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4254942d7ddc4c408114ae8d4b932a1a" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:50.314056 26643 raft_consensus.cc:3037] T 4254942d7ddc4c408114ae8d4b932a1a P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.315358 26377 leader_election.cc:288] T 4254942d7ddc4c408114ae8d4b932a1a P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 5 election: Requested vote from peers 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:50.315619 26314 leader_election.cc:302] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 5e9d825f803c40e089b64e319beccf6e; no voters: 83c50029f7a94cfca6d1ee68e0728aa2
I20240502 01:54:50.316007 28794 raft_consensus.cc:2781] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [term 6 FOLLOWER]: Leader election won for term 6
I20240502 01:54:50.316169 28794 raft_consensus.cc:686] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [term 6 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:50.316491 28794 consensus_queue.cc:237] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:50.317971 26643 raft_consensus.cc:2454] T 4254942d7ddc4c408114ae8d4b932a1a P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 5.
I20240502 01:54:50.318615 26313 leader_election.cc:302] T 4254942d7ddc4c408114ae8d4b932a1a P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e9d825f803c40e089b64e319beccf6e, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:50.318930 28795 raft_consensus.cc:2781] T 4254942d7ddc4c408114ae8d4b932a1a P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Leader election won for term 5
I20240502 01:54:50.319077 28795 raft_consensus.cc:686] T 4254942d7ddc4c408114ae8d4b932a1a P 5e9d825f803c40e089b64e319beccf6e [term 5 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:50.319288 28795 consensus_queue.cc:237] T 4254942d7ddc4c408114ae8d4b932a1a P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:50.319985 26228 catalog_manager.cc:5484] T c56cb13e76384d5f9a0e45be2f65c085 P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 3 to 5, leader changed from 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68) to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 5 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:50.322853 26378 raft_consensus.cc:1261] T c9e4788d50d04d129e8d03125073c6f2 P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Refusing update from remote peer 401942254c4c43a1ba02c4ae68d17320: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 01:54:50.323544 28871 consensus_queue.cc:1035] T c9e4788d50d04d129e8d03125073c6f2 P 401942254c4c43a1ba02c4ae68d17320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:50.324903 26774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4254942d7ddc4c408114ae8d4b932a1a" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:50.325119 26774 raft_consensus.cc:3037] T 4254942d7ddc4c408114ae8d4b932a1a P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.327940 26774 raft_consensus.cc:2396] T 4254942d7ddc4c408114ae8d4b932a1a P 401942254c4c43a1ba02c4ae68d17320 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e9d825f803c40e089b64e319beccf6e for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 01:54:50.299892 26772 raft_consensus.cc:2454] T 8cf6da25f535446b9515444e093cae7f P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 4.
I20240502 01:54:50.329190 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beff61f07ecf4ffab508da651a686c8e" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:50.329396 26511 raft_consensus.cc:3037] T beff61f07ecf4ffab508da651a686c8e P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.331912 26377 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4f9ed0162eb74c4ea422f122da01dce3"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:50.332409 26377 raft_consensus.cc:482] T 4f9ed0162eb74c4ea422f122da01dce3 P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:50.332751 26377 raft_consensus.cc:3037] T 4f9ed0162eb74c4ea422f122da01dce3 P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.332782 26511 raft_consensus.cc:2396] T beff61f07ecf4ffab508da651a686c8e P 4bc1541b47df446297a369de05163012 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e9d825f803c40e089b64e319beccf6e for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:50.334920 26314 leader_election.cc:302] T beff61f07ecf4ffab508da651a686c8e P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e9d825f803c40e089b64e319beccf6e; no voters: 4bc1541b47df446297a369de05163012, 83c50029f7a94cfca6d1ee68e0728aa2
I20240502 01:54:50.335353 28794 raft_consensus.cc:2726] T beff61f07ecf4ffab508da651a686c8e P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240502 01:54:50.337596 26377 raft_consensus.cc:504] T 4f9ed0162eb74c4ea422f122da01dce3 P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:50.338631 26774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9ed0162eb74c4ea422f122da01dce3" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:50.338907 26774 raft_consensus.cc:3037] T 4f9ed0162eb74c4ea422f122da01dce3 P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.338958 26228 catalog_manager.cc:5484] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 3 to 4, leader changed from 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68) to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 4 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:50.339524 26228 catalog_manager.cc:5484] T 4254942d7ddc4c408114ae8d4b932a1a P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 3 to 5, leader changed from 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68) to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 5 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:50.340041 26228 catalog_manager.cc:5484] T ef474f44fe5b4a06b2c8dda12b1bc496 P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 5 to 6, leader changed from 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67) to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 6 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:50.343327 26774 raft_consensus.cc:2454] T 4f9ed0162eb74c4ea422f122da01dce3 P 401942254c4c43a1ba02c4ae68d17320 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 5.
I20240502 01:54:50.343780 26314 leader_election.cc:302] T 4f9ed0162eb74c4ea422f122da01dce3 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:50.344520 28795 raft_consensus.cc:2781] T 4f9ed0162eb74c4ea422f122da01dce3 P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Leader election won for term 5
I20240502 01:54:50.344668 28795 raft_consensus.cc:686] T 4f9ed0162eb74c4ea422f122da01dce3 P 5e9d825f803c40e089b64e319beccf6e [term 5 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:50.344971 28795 consensus_queue.cc:237] T 4f9ed0162eb74c4ea422f122da01dce3 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:50.346680 26511 raft_consensus.cc:1261] T 14a747ac14354fdab2a2993594d3398f P 4bc1541b47df446297a369de05163012 [term 5 FOLLOWER]: Refusing update from remote peer 83c50029f7a94cfca6d1ee68e0728aa2: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240502 01:54:50.347295 28755 consensus_queue.cc:1035] T 14a747ac14354fdab2a2993594d3398f P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:50.347513 28755 consensus_queue.cc:1225] T 14a747ac14354fdab2a2993594d3398f P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Peer 4bc1541b47df446297a369de05163012 log is divergent from this leader: its last log entry 3.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240502 01:54:50.348546 26511 pending_rounds.cc:76] T 14a747ac14354fdab2a2993594d3398f P 4bc1541b47df446297a369de05163012: Aborting all ops after (but not including) 3
I20240502 01:54:50.348708 26511 pending_rounds.cc:97] T 14a747ac14354fdab2a2993594d3398f P 4bc1541b47df446297a369de05163012: Aborting uncommitted NO_OP operation due to leader change: 3.4
I20240502 01:54:50.348884 26511 raft_consensus.cc:2866] T 14a747ac14354fdab2a2993594d3398f P 4bc1541b47df446297a369de05163012 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 01:54:50.350980 26377 leader_election.cc:288] T 4f9ed0162eb74c4ea422f122da01dce3 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 5 election: Requested vote from peers 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:50.352201 26508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9ed0162eb74c4ea422f122da01dce3" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:50.352388 26508 raft_consensus.cc:3037] T 4f9ed0162eb74c4ea422f122da01dce3 P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.355350 26377 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9ead21d62e564a37a8c3d5a452621d13"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:50.355564 26377 raft_consensus.cc:482] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:50.355751 26377 raft_consensus.cc:3037] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.360338 26774 raft_consensus.cc:1261] T 14a747ac14354fdab2a2993594d3398f P 401942254c4c43a1ba02c4ae68d17320 [term 5 FOLLOWER]: Refusing update from remote peer 83c50029f7a94cfca6d1ee68e0728aa2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240502 01:54:50.360230 26228 catalog_manager.cc:5484] T 4f9ed0162eb74c4ea422f122da01dce3 P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 3 to 5, leader changed from 4bc1541b47df446297a369de05163012 (127.11.42.66) to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 5 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:50.361229 28761 consensus_queue.cc:1035] T 14a747ac14354fdab2a2993594d3398f P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:50.364836 26377 raft_consensus.cc:504] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:50.366883 26377 leader_election.cc:288] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 5 election: Requested vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:50.367869 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ead21d62e564a37a8c3d5a452621d13" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:50.368070 26511 raft_consensus.cc:3037] T 9ead21d62e564a37a8c3d5a452621d13 P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.368808 26377 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b4ba7c4fe9614ab3abb3645c1da3fcd3"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:50.368978 26377 raft_consensus.cc:482] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:50.369112 26377 raft_consensus.cc:3037] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Advancing to term 6
I20240502 01:54:50.369885 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ead21d62e564a37a8c3d5a452621d13" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:50.370081 26643 raft_consensus.cc:3032] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 01:54:50.370167 26643 raft_consensus.cc:729] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:50.370412 26508 raft_consensus.cc:2396] T 4f9ed0162eb74c4ea422f122da01dce3 P 4bc1541b47df446297a369de05163012 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e9d825f803c40e089b64e319beccf6e for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 01:54:50.370431 26643 consensus_queue.cc:260] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:50.370723 26643 raft_consensus.cc:3037] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.371407 26378 raft_consensus.cc:1261] T 3d751cdd58eb4cef8039fd4c50855e99 P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Refusing update from remote peer 83c50029f7a94cfca6d1ee68e0728aa2: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240502 01:54:50.372061 26377 raft_consensus.cc:504] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 5e9d825f803c40e089b64e319beccf6e [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:50.373204 26508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ba7c4fe9614ab3abb3645c1da3fcd3" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:50.373453 26508 raft_consensus.cc:3037] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 4bc1541b47df446297a369de05163012 [term 5 FOLLOWER]: Advancing to term 6
I20240502 01:54:50.375420 28761 consensus_queue.cc:1035] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:50.376163 26508 raft_consensus.cc:2454] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 4bc1541b47df446297a369de05163012 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 6.
I20240502 01:54:50.376576 26314 leader_election.cc:302] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:50.376919 28795 raft_consensus.cc:2781] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 5e9d825f803c40e089b64e319beccf6e [term 6 FOLLOWER]: Leader election won for term 6
I20240502 01:54:50.377123 28795 raft_consensus.cc:686] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 5e9d825f803c40e089b64e319beccf6e [term 6 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:50.377460 28795 consensus_queue.cc:237] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:50.378507 26640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ba7c4fe9614ab3abb3645c1da3fcd3" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:50.378716 26640 raft_consensus.cc:3032] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 LEADER]: Stepping down as leader of term 5
I20240502 01:54:50.378803 26640 raft_consensus.cc:729] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:50.379051 26640 consensus_queue.cc:260] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:50.379292 26640 raft_consensus.cc:3037] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Advancing to term 6
I20240502 01:54:50.381193 26640 raft_consensus.cc:2396] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e9d825f803c40e089b64e319beccf6e for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 01:54:50.382054 26643 raft_consensus.cc:2396] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e9d825f803c40e089b64e319beccf6e for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 01:54:50.383558 26377 leader_election.cc:288] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 6 election: Requested vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:50.383584 26511 raft_consensus.cc:2454] T 9ead21d62e564a37a8c3d5a452621d13 P 4bc1541b47df446297a369de05163012 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 5.
I20240502 01:54:50.385630 26314 leader_election.cc:302] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:50.385954 28794 raft_consensus.cc:2781] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Leader election won for term 5
I20240502 01:54:50.386137 28794 raft_consensus.cc:686] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e [term 5 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:50.388031 28794 consensus_queue.cc:237] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:50.388448 26377 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "53b7377c0d144e07a4501c2d626f6494"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:50.388793 26377 raft_consensus.cc:482] T 53b7377c0d144e07a4501c2d626f6494 P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:50.388912 26377 raft_consensus.cc:3037] T 53b7377c0d144e07a4501c2d626f6494 P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.389976 26228 catalog_manager.cc:5484] T b4ba7c4fe9614ab3abb3645c1da3fcd3 P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 5 to 6, leader changed from 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67) to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 6 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:50.392228 26774 raft_consensus.cc:1261] T 3d751cdd58eb4cef8039fd4c50855e99 P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Refusing update from remote peer 83c50029f7a94cfca6d1ee68e0728aa2: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20240502 01:54:50.392366 26774 pending_rounds.cc:76] T 3d751cdd58eb4cef8039fd4c50855e99 P 401942254c4c43a1ba02c4ae68d17320: Aborting all ops after (but not including) 4
I20240502 01:54:50.392431 26774 pending_rounds.cc:97] T 3d751cdd58eb4cef8039fd4c50855e99 P 401942254c4c43a1ba02c4ae68d17320: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20240502 01:54:50.392534 26774 raft_consensus.cc:2866] T 3d751cdd58eb4cef8039fd4c50855e99 P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 01:54:50.393121 28762 consensus_queue.cc:1035] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:50.400213 26228 catalog_manager.cc:5484] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 4 to 5, leader changed from 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67) to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 5 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:50.401531 26377 raft_consensus.cc:504] T 53b7377c0d144e07a4501c2d626f6494 P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:50.402166 26377 leader_election.cc:288] T 53b7377c0d144e07a4501c2d626f6494 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 5 election: Requested vote from peers 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:50.403016 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b7377c0d144e07a4501c2d626f6494" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:50.403214 26643 raft_consensus.cc:3037] T 53b7377c0d144e07a4501c2d626f6494 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.403415 26774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b7377c0d144e07a4501c2d626f6494" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:50.403509 26377 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d2287fdff0db43df90ff5580c0430b8c"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:50.403627 26774 raft_consensus.cc:3037] T 53b7377c0d144e07a4501c2d626f6494 P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.403673 26377 raft_consensus.cc:482] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:50.403786 26377 raft_consensus.cc:3037] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Advancing to term 6
I20240502 01:54:50.405973 26643 raft_consensus.cc:2454] T 53b7377c0d144e07a4501c2d626f6494 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 5.
I20240502 01:54:50.406476 26774 raft_consensus.cc:2396] T 53b7377c0d144e07a4501c2d626f6494 P 401942254c4c43a1ba02c4ae68d17320 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e9d825f803c40e089b64e319beccf6e for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 01:54:50.406409 26377 raft_consensus.cc:504] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:50.407024 26377 leader_election.cc:288] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 6 election: Requested vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:50.407178 26313 leader_election.cc:302] T 53b7377c0d144e07a4501c2d626f6494 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e9d825f803c40e089b64e319beccf6e, 83c50029f7a94cfca6d1ee68e0728aa2; no voters: 401942254c4c43a1ba02c4ae68d17320
I20240502 01:54:50.407618 28794 raft_consensus.cc:2781] T 53b7377c0d144e07a4501c2d626f6494 P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Leader election won for term 5
I20240502 01:54:50.407595 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2287fdff0db43df90ff5580c0430b8c" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:50.407781 28794 raft_consensus.cc:686] T 53b7377c0d144e07a4501c2d626f6494 P 5e9d825f803c40e089b64e319beccf6e [term 5 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:50.407802 26511 raft_consensus.cc:3037] T d2287fdff0db43df90ff5580c0430b8c P 4bc1541b47df446297a369de05163012 [term 5 FOLLOWER]: Advancing to term 6
I20240502 01:54:50.408094 28794 consensus_queue.cc:237] T 53b7377c0d144e07a4501c2d626f6494 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:50.410370 26511 raft_consensus.cc:2454] T d2287fdff0db43df90ff5580c0430b8c P 4bc1541b47df446297a369de05163012 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 6.
I20240502 01:54:50.410730 26378 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "245ca06c39774fb5b0f522550a63ac5b"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:50.410951 26378 raft_consensus.cc:482] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:50.411077 26378 raft_consensus.cc:3037] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:50.413256 26640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2287fdff0db43df90ff5580c0430b8c" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:50.413486 26640 raft_consensus.cc:3032] T d2287fdff0db43df90ff5580c0430b8c P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 LEADER]: Stepping down as leader of term 5
I20240502 01:54:50.413583 26640 raft_consensus.cc:729] T d2287fdff0db43df90ff5580c0430b8c P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:50.415067 26314 leader_election.cc:302] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:50.413854 26640 consensus_queue.cc:260] T d2287fdff0db43df90ff5580c0430b8c P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:50.415546 26640 raft_consensus.cc:3037] T d2287fdff0db43df90ff5580c0430b8c P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Advancing to term 6
I20240502 01:54:50.416030 28794 raft_consensus.cc:2781] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [term 6 FOLLOWER]: Leader election won for term 6
I20240502 01:54:50.416257 28794 raft_consensus.cc:686] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [term 6 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:50.416507 28794 consensus_queue.cc:237] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:50.416836 26234 catalog_manager.cc:5484] T 53b7377c0d144e07a4501c2d626f6494 P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 3 to 5, leader changed from 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68) to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 5 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:50.418342 26640 raft_consensus.cc:2396] T d2287fdff0db43df90ff5580c0430b8c P 83c50029f7a94cfca6d1ee68e0728aa2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e9d825f803c40e089b64e319beccf6e for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 01:54:50.419637 26378 raft_consensus.cc:504] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:50.420961 26774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "245ca06c39774fb5b0f522550a63ac5b" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:50.421288 26378 leader_election.cc:288] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 4 election: Requested vote from peers 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:50.422634 26377 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "daef5905fc19473a8728ccd2627270af"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:50.422821 26377 raft_consensus.cc:482] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:50.422935 26377 raft_consensus.cc:3037] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Advancing to term 6
I20240502 01:54:50.423538 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "245ca06c39774fb5b0f522550a63ac5b" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:50.423724 26511 raft_consensus.cc:3037] T 245ca06c39774fb5b0f522550a63ac5b P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:50.421278 26774 raft_consensus.cc:3032] T 245ca06c39774fb5b0f522550a63ac5b P 401942254c4c43a1ba02c4ae68d17320 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:54:50.424541 26774 raft_consensus.cc:729] T 245ca06c39774fb5b0f522550a63ac5b P 401942254c4c43a1ba02c4ae68d17320 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Running, Role: LEADER
I20240502 01:54:50.425508 26377 raft_consensus.cc:504] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:50.426035 26379 raft_consensus.cc:1226] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [term 6 FOLLOWER]: Rejecting Update request from peer 83c50029f7a94cfca6d1ee68e0728aa2 for earlier term 5. Current term is 6. Ops: []
I20240502 01:54:50.426174 26377 leader_election.cc:288] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 6 election: Requested vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:50.426820 28755 consensus_queue.cc:1046] T daef5905fc19473a8728ccd2627270af P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:50.427037 28755 consensus_queue.cc:1225] T daef5905fc19473a8728ccd2627270af P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Peer 5e9d825f803c40e089b64e319beccf6e log is divergent from this leader: its last log entry 3.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240502 01:54:50.427141 28755 raft_consensus.cc:3032] T daef5905fc19473a8728ccd2627270af P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 LEADER]: Stepping down as leader of term 5
I20240502 01:54:50.427222 28755 raft_consensus.cc:729] T daef5905fc19473a8728ccd2627270af P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:50.427469 28755 consensus_queue.cc:260] T daef5905fc19473a8728ccd2627270af P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:50.427795 28755 raft_consensus.cc:3037] T daef5905fc19473a8728ccd2627270af P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Advancing to term 6
I20240502 01:54:50.429963 26640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daef5905fc19473a8728ccd2627270af" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:50.430294 26379 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "99a4b7d022e1461a8233952ab47ef5d3"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:50.430490 26379 raft_consensus.cc:482] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:50.430586 26379 raft_consensus.cc:3037] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Advancing to term 6
I20240502 01:54:50.434415 26379 raft_consensus.cc:504] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:50.437868 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a4b7d022e1461a8233952ab47ef5d3" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:50.438067 26643 raft_consensus.cc:3032] T 99a4b7d022e1461a8233952ab47ef5d3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 LEADER]: Stepping down as leader of term 5
I20240502 01:54:50.438154 26643 raft_consensus.cc:729] T 99a4b7d022e1461a8233952ab47ef5d3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:50.438421 26643 consensus_queue.cc:260] T 99a4b7d022e1461a8233952ab47ef5d3 P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:50.438697 26643 raft_consensus.cc:3037] T 99a4b7d022e1461a8233952ab47ef5d3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Advancing to term 6
I20240502 01:54:50.442112 26508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a4b7d022e1461a8233952ab47ef5d3" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:50.442323 26508 raft_consensus.cc:3037] T 99a4b7d022e1461a8233952ab47ef5d3 P 4bc1541b47df446297a369de05163012 [term 5 FOLLOWER]: Advancing to term 6
I20240502 01:54:50.442433 26234 catalog_manager.cc:5484] T d2287fdff0db43df90ff5580c0430b8c P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 5 to 6, leader changed from 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67) to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 6 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:50.445298 26508 raft_consensus.cc:2454] T 99a4b7d022e1461a8233952ab47ef5d3 P 4bc1541b47df446297a369de05163012 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 6.
I20240502 01:54:50.445838 26314 leader_election.cc:302] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:50.446588 26643 raft_consensus.cc:2396] T 99a4b7d022e1461a8233952ab47ef5d3 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e9d825f803c40e089b64e319beccf6e for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 01:54:50.447165 28794 raft_consensus.cc:2781] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [term 6 FOLLOWER]: Leader election won for term 6
I20240502 01:54:50.447335 28794 raft_consensus.cc:686] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [term 6 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:50.447597 28794 consensus_queue.cc:237] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:50.432186 26772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daef5905fc19473a8728ccd2627270af" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:50.448818 26772 raft_consensus.cc:3037] T daef5905fc19473a8728ccd2627270af P 401942254c4c43a1ba02c4ae68d17320 [term 5 FOLLOWER]: Advancing to term 6
I20240502 01:54:50.449486 26508 raft_consensus.cc:1261] T d9dd99b54b7c4d19a80c229bf99d7976 P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Refusing update from remote peer 83c50029f7a94cfca6d1ee68e0728aa2: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240502 01:54:50.450145 28755 consensus_queue.cc:1035] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:50.451678 26772 raft_consensus.cc:2454] T daef5905fc19473a8728ccd2627270af P 401942254c4c43a1ba02c4ae68d17320 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 6.
I20240502 01:54:50.453605 26773 raft_consensus.cc:1261] T d9dd99b54b7c4d19a80c229bf99d7976 P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Refusing update from remote peer 83c50029f7a94cfca6d1ee68e0728aa2: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240502 01:54:50.455294 28772 consensus_queue.cc:1035] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:50.458509 26314 leader_election.cc:302] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:50.459172 28795 raft_consensus.cc:2781] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [term 6 FOLLOWER]: Leader election won for term 6
I20240502 01:54:50.459401 28795 raft_consensus.cc:686] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [term 6 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:50.459681 28795 consensus_queue.cc:237] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:50.462430 26774 consensus_queue.cc:260] T 245ca06c39774fb5b0f522550a63ac5b P 401942254c4c43a1ba02c4ae68d17320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:50.462905 26774 raft_consensus.cc:3037] T 245ca06c39774fb5b0f522550a63ac5b P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:50.465139 26379 leader_election.cc:288] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 6 election: Requested vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:50.466225 26774 raft_consensus.cc:2454] T 245ca06c39774fb5b0f522550a63ac5b P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 4.
I20240502 01:54:50.466485 26379 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0ddd395eec1146bc8a20292d81be4868"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:50.466691 26379 raft_consensus.cc:482] T 0ddd395eec1146bc8a20292d81be4868 P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:50.466733 26314 leader_election.cc:302] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:50.466828 26379 raft_consensus.cc:3037] T 0ddd395eec1146bc8a20292d81be4868 P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Advancing to term 6
I20240502 01:54:50.467159 28795 raft_consensus.cc:2781] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Leader election won for term 4
I20240502 01:54:50.467504 28795 raft_consensus.cc:686] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e [term 4 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:50.467854 28795 consensus_queue.cc:237] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:50.469972 26379 raft_consensus.cc:504] T 0ddd395eec1146bc8a20292d81be4868 P 5e9d825f803c40e089b64e319beccf6e [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:50.471040 26640 raft_consensus.cc:2396] T daef5905fc19473a8728ccd2627270af P 83c50029f7a94cfca6d1ee68e0728aa2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e9d825f803c40e089b64e319beccf6e for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 01:54:50.471820 26379 leader_election.cc:288] T 0ddd395eec1146bc8a20292d81be4868 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 6 election: Requested vote from peers 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:50.472487 26511 raft_consensus.cc:2454] T 245ca06c39774fb5b0f522550a63ac5b P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 4.
I20240502 01:54:50.472743 26774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ddd395eec1146bc8a20292d81be4868" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:50.473881 26379 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2fd06590221d414b9e896e02b1be6f3e"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:50.474076 26379 raft_consensus.cc:482] T 2fd06590221d414b9e896e02b1be6f3e P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:50.474184 26379 raft_consensus.cc:3037] T 2fd06590221d414b9e896e02b1be6f3e P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Advancing to term 6
I20240502 01:54:50.472962 26774 raft_consensus.cc:3037] T 0ddd395eec1146bc8a20292d81be4868 P 401942254c4c43a1ba02c4ae68d17320 [term 5 FOLLOWER]: Advancing to term 6
I20240502 01:54:50.475432 26234 catalog_manager.cc:5484] T 99a4b7d022e1461a8233952ab47ef5d3 P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 5 to 6, leader changed from 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67) to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 6 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:50.476691 26379 raft_consensus.cc:504] T 2fd06590221d414b9e896e02b1be6f3e P 5e9d825f803c40e089b64e319beccf6e [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:50.477272 26379 leader_election.cc:288] T 2fd06590221d414b9e896e02b1be6f3e P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 6 election: Requested vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 4bc1541b47df446297a369de05163012 (127.11.42.66:34003)
I20240502 01:54:50.478552 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fd06590221d414b9e896e02b1be6f3e" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:50.478766 26643 raft_consensus.cc:3032] T 2fd06590221d414b9e896e02b1be6f3e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 LEADER]: Stepping down as leader of term 5
I20240502 01:54:50.478852 26643 raft_consensus.cc:729] T 2fd06590221d414b9e896e02b1be6f3e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:50.479143 26643 consensus_queue.cc:260] T 2fd06590221d414b9e896e02b1be6f3e P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:50.479446 26643 raft_consensus.cc:3037] T 2fd06590221d414b9e896e02b1be6f3e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Advancing to term 6
I20240502 01:54:50.479905 26379 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "23dcaac2d7d14adf820d826f05c1abc6"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:50.480088 26379 raft_consensus.cc:482] T 23dcaac2d7d14adf820d826f05c1abc6 P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:50.480135 26774 raft_consensus.cc:2454] T 0ddd395eec1146bc8a20292d81be4868 P 401942254c4c43a1ba02c4ae68d17320 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 6.
I20240502 01:54:50.480206 26379 raft_consensus.cc:3037] T 23dcaac2d7d14adf820d826f05c1abc6 P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.481365 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fd06590221d414b9e896e02b1be6f3e" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:50.481909 26511 raft_consensus.cc:3037] T 2fd06590221d414b9e896e02b1be6f3e P 4bc1541b47df446297a369de05163012 [term 5 FOLLOWER]: Advancing to term 6
I20240502 01:54:50.482307 26643 raft_consensus.cc:2396] T 2fd06590221d414b9e896e02b1be6f3e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e9d825f803c40e089b64e319beccf6e for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 01:54:50.482578 26640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ddd395eec1146bc8a20292d81be4868" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:50.483023 26314 leader_election.cc:302] T 0ddd395eec1146bc8a20292d81be4868 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:50.483088 26379 raft_consensus.cc:504] T 23dcaac2d7d14adf820d826f05c1abc6 P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:50.483394 26640 raft_consensus.cc:3032] T 0ddd395eec1146bc8a20292d81be4868 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 LEADER]: Stepping down as leader of term 5
I20240502 01:54:50.483997 28794 raft_consensus.cc:2781] T 0ddd395eec1146bc8a20292d81be4868 P 5e9d825f803c40e089b64e319beccf6e [term 6 FOLLOWER]: Leader election won for term 6
I20240502 01:54:50.484176 28794 raft_consensus.cc:686] T 0ddd395eec1146bc8a20292d81be4868 P 5e9d825f803c40e089b64e319beccf6e [term 6 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:50.484473 28794 consensus_queue.cc:237] T 0ddd395eec1146bc8a20292d81be4868 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:50.485195 26508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23dcaac2d7d14adf820d826f05c1abc6" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:50.485414 26508 raft_consensus.cc:3037] T 23dcaac2d7d14adf820d826f05c1abc6 P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.485663 26511 raft_consensus.cc:2454] T 2fd06590221d414b9e896e02b1be6f3e P 4bc1541b47df446297a369de05163012 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 6.
I20240502 01:54:50.485899 26640 raft_consensus.cc:729] T 0ddd395eec1146bc8a20292d81be4868 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:50.486183 26640 consensus_queue.cc:260] T 0ddd395eec1146bc8a20292d81be4868 P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:50.486477 26640 raft_consensus.cc:3037] T 0ddd395eec1146bc8a20292d81be4868 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Advancing to term 6
I20240502 01:54:50.487486 26234 catalog_manager.cc:5484] T daef5905fc19473a8728ccd2627270af P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 5 to 6, leader changed from 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67) to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 6 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:50.488044 26508 raft_consensus.cc:2396] T 23dcaac2d7d14adf820d826f05c1abc6 P 4bc1541b47df446297a369de05163012 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e9d825f803c40e089b64e319beccf6e for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 01:54:50.488158 26234 catalog_manager.cc:5484] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 3 to 4, leader changed from 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68) to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 4 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:50.488602 26314 leader_election.cc:302] T 2fd06590221d414b9e896e02b1be6f3e P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4bc1541b47df446297a369de05163012, 5e9d825f803c40e089b64e319beccf6e; no voters: 83c50029f7a94cfca6d1ee68e0728aa2
I20240502 01:54:50.489154 28795 raft_consensus.cc:2781] T 2fd06590221d414b9e896e02b1be6f3e P 5e9d825f803c40e089b64e319beccf6e [term 6 FOLLOWER]: Leader election won for term 6
I20240502 01:54:50.489321 28795 raft_consensus.cc:686] T 2fd06590221d414b9e896e02b1be6f3e P 5e9d825f803c40e089b64e319beccf6e [term 6 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:50.489634 28795 consensus_queue.cc:237] T 2fd06590221d414b9e896e02b1be6f3e P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:50.492241 26640 raft_consensus.cc:2396] T 0ddd395eec1146bc8a20292d81be4868 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e9d825f803c40e089b64e319beccf6e for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 01:54:50.496896 26379 leader_election.cc:288] T 23dcaac2d7d14adf820d826f05c1abc6 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 5 election: Requested vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:50.497390 26640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23dcaac2d7d14adf820d826f05c1abc6" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:50.497658 26640 raft_consensus.cc:3037] T 23dcaac2d7d14adf820d826f05c1abc6 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.498255 26379 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c9e4788d50d04d129e8d03125073c6f2"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:50.498463 26379 raft_consensus.cc:482] T c9e4788d50d04d129e8d03125073c6f2 P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:50.498577 26379 raft_consensus.cc:3037] T c9e4788d50d04d129e8d03125073c6f2 P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:50.501722 26379 raft_consensus.cc:504] T c9e4788d50d04d129e8d03125073c6f2 P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:50.502831 26508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e4788d50d04d129e8d03125073c6f2" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:50.503065 26508 raft_consensus.cc:3037] T c9e4788d50d04d129e8d03125073c6f2 P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:50.504375 26640 raft_consensus.cc:2454] T 23dcaac2d7d14adf820d826f05c1abc6 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 5.
I20240502 01:54:50.504853 26313 leader_election.cc:302] T 23dcaac2d7d14adf820d826f05c1abc6 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e9d825f803c40e089b64e319beccf6e, 83c50029f7a94cfca6d1ee68e0728aa2; no voters: 4bc1541b47df446297a369de05163012
I20240502 01:54:50.505331 28789 raft_consensus.cc:2781] T 23dcaac2d7d14adf820d826f05c1abc6 P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Leader election won for term 5
I20240502 01:54:50.506063 26508 raft_consensus.cc:2454] T c9e4788d50d04d129e8d03125073c6f2 P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 4.
I20240502 01:54:50.506647 26314 leader_election.cc:302] T c9e4788d50d04d129e8d03125073c6f2 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:50.509261 28789 raft_consensus.cc:686] T 23dcaac2d7d14adf820d826f05c1abc6 P 5e9d825f803c40e089b64e319beccf6e [term 5 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:50.509488 28940 raft_consensus.cc:2781] T c9e4788d50d04d129e8d03125073c6f2 P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Leader election won for term 4
I20240502 01:54:50.513660 26774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e4788d50d04d129e8d03125073c6f2" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:50.513909 26774 raft_consensus.cc:3032] T c9e4788d50d04d129e8d03125073c6f2 P 401942254c4c43a1ba02c4ae68d17320 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:54:50.513993 26774 raft_consensus.cc:729] T c9e4788d50d04d129e8d03125073c6f2 P 401942254c4c43a1ba02c4ae68d17320 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Running, Role: LEADER
I20240502 01:54:50.514287 26774 consensus_queue.cc:260] T c9e4788d50d04d129e8d03125073c6f2 P 401942254c4c43a1ba02c4ae68d17320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:50.514676 26774 raft_consensus.cc:3037] T c9e4788d50d04d129e8d03125073c6f2 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:50.509565 28789 consensus_queue.cc:237] T 23dcaac2d7d14adf820d826f05c1abc6 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:50.512746 26379 leader_election.cc:288] T c9e4788d50d04d129e8d03125073c6f2 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 4 election: Requested vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:50.517033 26379 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "37e818dc0b834eb185e27fcb3cb972e1"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:50.517218 26379 raft_consensus.cc:482] T 37e818dc0b834eb185e27fcb3cb972e1 P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:50.517371 26379 raft_consensus.cc:3037] T 37e818dc0b834eb185e27fcb3cb972e1 P 5e9d825f803c40e089b64e319beccf6e [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:50.517434 26774 raft_consensus.cc:2454] T c9e4788d50d04d129e8d03125073c6f2 P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 4.
I20240502 01:54:50.519456 26379 raft_consensus.cc:504] T 37e818dc0b834eb185e27fcb3cb972e1 P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:50.519963 26379 leader_election.cc:288] T 37e818dc0b834eb185e27fcb3cb972e1 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 4 election: Requested vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:50.520536 26508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e818dc0b834eb185e27fcb3cb972e1" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:50.520835 26508 raft_consensus.cc:3037] T 37e818dc0b834eb185e27fcb3cb972e1 P 4bc1541b47df446297a369de05163012 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:50.521229 26377 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c26358e3969143fcbf2f466241b360a7"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:50.521394 26377 raft_consensus.cc:482] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:50.521499 26377 raft_consensus.cc:3037] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Advancing to term 6
I20240502 01:54:50.522415 28940 raft_consensus.cc:686] T c9e4788d50d04d129e8d03125073c6f2 P 5e9d825f803c40e089b64e319beccf6e [term 4 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:50.522737 28940 consensus_queue.cc:237] T c9e4788d50d04d129e8d03125073c6f2 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:50.523387 26774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e818dc0b834eb185e27fcb3cb972e1" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:50.523571 26774 raft_consensus.cc:3032] T 37e818dc0b834eb185e27fcb3cb972e1 P 401942254c4c43a1ba02c4ae68d17320 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 01:54:50.523636 26774 raft_consensus.cc:729] T 37e818dc0b834eb185e27fcb3cb972e1 P 401942254c4c43a1ba02c4ae68d17320 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 401942254c4c43a1ba02c4ae68d17320, State: Running, Role: LEADER
I20240502 01:54:50.523939 26774 consensus_queue.cc:260] T 37e818dc0b834eb185e27fcb3cb972e1 P 401942254c4c43a1ba02c4ae68d17320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:50.524111 26377 raft_consensus.cc:504] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:50.524405 26774 raft_consensus.cc:3037] T 37e818dc0b834eb185e27fcb3cb972e1 P 401942254c4c43a1ba02c4ae68d17320 [term 3 FOLLOWER]: Advancing to term 4
I20240502 01:54:50.524768 26377 leader_election.cc:288] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 6 election: Requested vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:50.525346 26773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c26358e3969143fcbf2f466241b360a7" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:50.525540 26773 raft_consensus.cc:3037] T c26358e3969143fcbf2f466241b360a7 P 401942254c4c43a1ba02c4ae68d17320 [term 5 FOLLOWER]: Advancing to term 6
I20240502 01:54:50.525939 26378 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3d751cdd58eb4cef8039fd4c50855e99"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:50.526111 26378 raft_consensus.cc:482] T 3d751cdd58eb4cef8039fd4c50855e99 P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:50.526208 26378 raft_consensus.cc:3037] T 3d751cdd58eb4cef8039fd4c50855e99 P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.524222 26234 catalog_manager.cc:5484] T 0ddd395eec1146bc8a20292d81be4868 P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 5 to 6, leader changed from 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67) to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 6 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:50.527261 26234 catalog_manager.cc:5484] T 2fd06590221d414b9e896e02b1be6f3e P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 5 to 6, leader changed from 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67) to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 6 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:50.528123 26773 raft_consensus.cc:2454] T c26358e3969143fcbf2f466241b360a7 P 401942254c4c43a1ba02c4ae68d17320 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 6.
I20240502 01:54:50.528699 26378 raft_consensus.cc:504] T 3d751cdd58eb4cef8039fd4c50855e99 P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:50.529125 26314 leader_election.cc:302] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:50.529640 26508 raft_consensus.cc:2454] T 37e818dc0b834eb185e27fcb3cb972e1 P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 4.
I20240502 01:54:50.529706 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d751cdd58eb4cef8039fd4c50855e99" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:50.529280 26378 leader_election.cc:288] T 3d751cdd58eb4cef8039fd4c50855e99 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 5 election: Requested vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:50.529958 26643 raft_consensus.cc:3032] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 01:54:50.530047 26643 raft_consensus.cc:729] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:50.530352 26643 consensus_queue.cc:260] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:50.530755 26643 raft_consensus.cc:3037] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.530755 26773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d751cdd58eb4cef8039fd4c50855e99" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:50.530815 26378 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1ce4dd22bb314eb7855126e3fe1785f6"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:50.530933 26773 raft_consensus.cc:3037] T 3d751cdd58eb4cef8039fd4c50855e99 P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.530969 26378 raft_consensus.cc:482] T 1ce4dd22bb314eb7855126e3fe1785f6 P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:50.531081 26378 raft_consensus.cc:3037] T 1ce4dd22bb314eb7855126e3fe1785f6 P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.533514 26643 raft_consensus.cc:2454] T 3d751cdd58eb4cef8039fd4c50855e99 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 5.
I20240502 01:54:50.533609 26378 raft_consensus.cc:504] T 1ce4dd22bb314eb7855126e3fe1785f6 P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:50.533824 26773 raft_consensus.cc:2454] T 3d751cdd58eb4cef8039fd4c50855e99 P 401942254c4c43a1ba02c4ae68d17320 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 5.
I20240502 01:54:50.533962 26313 leader_election.cc:302] T 3d751cdd58eb4cef8039fd4c50855e99 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e9d825f803c40e089b64e319beccf6e, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:50.534314 28794 raft_consensus.cc:2781] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [term 6 FOLLOWER]: Leader election won for term 6
I20240502 01:54:50.528616 26640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c26358e3969143fcbf2f466241b360a7" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:50.534471 26640 raft_consensus.cc:3032] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 LEADER]: Stepping down as leader of term 5
I20240502 01:54:50.534586 26640 raft_consensus.cc:729] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:50.534595 28794 raft_consensus.cc:686] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [term 6 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:50.534972 28794 consensus_queue.cc:237] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:50.535871 28794 raft_consensus.cc:2781] T 3d751cdd58eb4cef8039fd4c50855e99 P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Leader election won for term 5
I20240502 01:54:50.535910 26640 consensus_queue.cc:260] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:50.536217 26640 raft_consensus.cc:3037] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Advancing to term 6
I20240502 01:54:50.536231 28794 raft_consensus.cc:686] T 3d751cdd58eb4cef8039fd4c50855e99 P 5e9d825f803c40e089b64e319beccf6e [term 5 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:50.536571 28794 consensus_queue.cc:237] T 3d751cdd58eb4cef8039fd4c50855e99 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:50.534305 26314 leader_election.cc:302] T 37e818dc0b834eb185e27fcb3cb972e1 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:50.537765 28794 raft_consensus.cc:2781] T 37e818dc0b834eb185e27fcb3cb972e1 P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Leader election won for term 4
I20240502 01:54:50.538683 28794 raft_consensus.cc:686] T 37e818dc0b834eb185e27fcb3cb972e1 P 5e9d825f803c40e089b64e319beccf6e [term 4 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:50.539005 26773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ce4dd22bb314eb7855126e3fe1785f6" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:50.539069 28794 consensus_queue.cc:237] T 37e818dc0b834eb185e27fcb3cb972e1 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:50.539294 26773 raft_consensus.cc:3037] T 1ce4dd22bb314eb7855126e3fe1785f6 P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.541148 26773 raft_consensus.cc:2454] T 1ce4dd22bb314eb7855126e3fe1785f6 P 401942254c4c43a1ba02c4ae68d17320 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 5.
I20240502 01:54:50.541472 26314 leader_election.cc:302] T 1ce4dd22bb314eb7855126e3fe1785f6 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:50.541697 28794 raft_consensus.cc:2781] T 1ce4dd22bb314eb7855126e3fe1785f6 P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Leader election won for term 5
I20240502 01:54:50.541823 26508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ce4dd22bb314eb7855126e3fe1785f6" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:50.541896 28794 raft_consensus.cc:686] T 1ce4dd22bb314eb7855126e3fe1785f6 P 5e9d825f803c40e089b64e319beccf6e [term 5 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:50.542052 26508 raft_consensus.cc:3037] T 1ce4dd22bb314eb7855126e3fe1785f6 P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.542270 28794 consensus_queue.cc:237] T 1ce4dd22bb314eb7855126e3fe1785f6 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:50.544863 26508 raft_consensus.cc:2396] T 1ce4dd22bb314eb7855126e3fe1785f6 P 4bc1541b47df446297a369de05163012 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e9d825f803c40e089b64e319beccf6e for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 01:54:50.538149 26378 leader_election.cc:288] T 1ce4dd22bb314eb7855126e3fe1785f6 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 5 election: Requested vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:50.546185 26640 raft_consensus.cc:2396] T c26358e3969143fcbf2f466241b360a7 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e9d825f803c40e089b64e319beccf6e for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 01:54:50.546165 26379 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b5ab243d9f1c4b568d38ed03027adb44"
dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
from {username='slave'} at 127.0.0.1:44276
I20240502 01:54:50.546502 26379 raft_consensus.cc:482] T b5ab243d9f1c4b568d38ed03027adb44 P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:50.546752 26774 raft_consensus.cc:2454] T 37e818dc0b834eb185e27fcb3cb972e1 P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 4.
I20240502 01:54:50.546973 26379 raft_consensus.cc:3037] T b5ab243d9f1c4b568d38ed03027adb44 P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.549428 26379 raft_consensus.cc:504] T b5ab243d9f1c4b568d38ed03027adb44 P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:50.550189 26379 leader_election.cc:288] T b5ab243d9f1c4b568d38ed03027adb44 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 5 election: Requested vote from peers 4bc1541b47df446297a369de05163012 (127.11.42.66:34003), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:50.551133 26774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ab243d9f1c4b568d38ed03027adb44" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:50.551513 26774 raft_consensus.cc:3037] T b5ab243d9f1c4b568d38ed03027adb44 P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.554271 26774 raft_consensus.cc:2454] T b5ab243d9f1c4b568d38ed03027adb44 P 401942254c4c43a1ba02c4ae68d17320 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e9d825f803c40e089b64e319beccf6e in term 5.
I20240502 01:54:50.554735 26508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ab243d9f1c4b568d38ed03027adb44" candidate_uuid: "5e9d825f803c40e089b64e319beccf6e" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4bc1541b47df446297a369de05163012"
I20240502 01:54:50.554986 26508 raft_consensus.cc:3037] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.556123 26314 leader_election.cc:302] T b5ab243d9f1c4b568d38ed03027adb44 P 5e9d825f803c40e089b64e319beccf6e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:50.556581 26234 catalog_manager.cc:5484] T 23dcaac2d7d14adf820d826f05c1abc6 P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 3 to 5, leader changed from 4bc1541b47df446297a369de05163012 (127.11.42.66) to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 5 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:50.557225 26234 catalog_manager.cc:5484] T c9e4788d50d04d129e8d03125073c6f2 P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 3 to 4, leader changed from 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68) to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 4 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:50.557794 26234 catalog_manager.cc:5484] T 3d751cdd58eb4cef8039fd4c50855e99 P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 4 to 5, leader changed from 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67) to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 5 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:50.558539 26508 raft_consensus.cc:2396] T b5ab243d9f1c4b568d38ed03027adb44 P 4bc1541b47df446297a369de05163012 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e9d825f803c40e089b64e319beccf6e for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 01:54:50.558285 26234 catalog_manager.cc:5484] T c26358e3969143fcbf2f466241b360a7 P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 5 to 6, leader changed from 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67) to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 6 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:50.561180 28789 raft_consensus.cc:2781] T b5ab243d9f1c4b568d38ed03027adb44 P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Leader election won for term 5
I20240502 01:54:50.561332 28789 raft_consensus.cc:686] T b5ab243d9f1c4b568d38ed03027adb44 P 5e9d825f803c40e089b64e319beccf6e [term 5 LEADER]: Becoming Leader. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:50.561589 28789 consensus_queue.cc:237] T b5ab243d9f1c4b568d38ed03027adb44 P 5e9d825f803c40e089b64e319beccf6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } }
I20240502 01:54:50.565954 26234 catalog_manager.cc:5484] T 37e818dc0b834eb185e27fcb3cb972e1 P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 3 to 4, leader changed from 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68) to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 4 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:50.566502 26234 catalog_manager.cc:5484] T 1ce4dd22bb314eb7855126e3fe1785f6 P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 3 to 5, leader changed from 4bc1541b47df446297a369de05163012 (127.11.42.66) to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 5 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:50.567801 26508 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cf09a605f32845a3aaad81d79063070b"
dest_uuid: "4bc1541b47df446297a369de05163012"
from {username='slave'} at 127.0.0.1:41462
I20240502 01:54:50.568015 26508 raft_consensus.cc:482] T cf09a605f32845a3aaad81d79063070b P 4bc1541b47df446297a369de05163012 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:50.568120 26508 raft_consensus.cc:3037] T cf09a605f32845a3aaad81d79063070b P 4bc1541b47df446297a369de05163012 [term 6 FOLLOWER]: Advancing to term 7
I20240502 01:54:50.570955 26508 raft_consensus.cc:504] T cf09a605f32845a3aaad81d79063070b P 4bc1541b47df446297a369de05163012 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:50.572069 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf09a605f32845a3aaad81d79063070b" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:50.572260 26379 raft_consensus.cc:3032] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [term 6 LEADER]: Stepping down as leader of term 6
I20240502 01:54:50.572367 26379 raft_consensus.cc:729] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:50.572590 26379 consensus_queue.cc:260] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:50.572521 26234 catalog_manager.cc:5484] T b5ab243d9f1c4b568d38ed03027adb44 P 5e9d825f803c40e089b64e319beccf6e reported cstate change: term changed from 3 to 5, leader changed from 4bc1541b47df446297a369de05163012 (127.11.42.66) to 5e9d825f803c40e089b64e319beccf6e (127.11.42.65). New cstate: current_term: 5 leader_uuid: "5e9d825f803c40e089b64e319beccf6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:50.572868 26379 raft_consensus.cc:3037] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [term 6 FOLLOWER]: Advancing to term 7
I20240502 01:54:50.575677 26379 raft_consensus.cc:2396] T cf09a605f32845a3aaad81d79063070b P 5e9d825f803c40e089b64e319beccf6e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bc1541b47df446297a369de05163012 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:50.576854 26508 leader_election.cc:288] T cf09a605f32845a3aaad81d79063070b P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 7 election: Requested vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:50.577348 26640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf09a605f32845a3aaad81d79063070b" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:50.577555 26640 raft_consensus.cc:3037] T cf09a605f32845a3aaad81d79063070b P 83c50029f7a94cfca6d1ee68e0728aa2 [term 6 FOLLOWER]: Advancing to term 7
I20240502 01:54:50.580420 26640 raft_consensus.cc:2396] T cf09a605f32845a3aaad81d79063070b P 83c50029f7a94cfca6d1ee68e0728aa2 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bc1541b47df446297a369de05163012 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 01:54:50.581030 26446 leader_election.cc:302] T cf09a605f32845a3aaad81d79063070b P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bc1541b47df446297a369de05163012; no voters: 5e9d825f803c40e089b64e319beccf6e, 83c50029f7a94cfca6d1ee68e0728aa2
I20240502 01:54:50.581526 28770 raft_consensus.cc:2726] T cf09a605f32845a3aaad81d79063070b P 4bc1541b47df446297a369de05163012 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240502 01:54:50.582387 26508 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8cf6da25f535446b9515444e093cae7f"
dest_uuid: "4bc1541b47df446297a369de05163012"
from {username='slave'} at 127.0.0.1:41462
I20240502 01:54:50.582581 26508 raft_consensus.cc:482] T 8cf6da25f535446b9515444e093cae7f P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:50.582706 26508 raft_consensus.cc:3037] T 8cf6da25f535446b9515444e093cae7f P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.585328 26508 raft_consensus.cc:504] T 8cf6da25f535446b9515444e093cae7f P 4bc1541b47df446297a369de05163012 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:50.586676 26774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cf6da25f535446b9515444e093cae7f" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:50.586892 26774 raft_consensus.cc:3037] T 8cf6da25f535446b9515444e093cae7f P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.587471 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cf6da25f535446b9515444e093cae7f" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:50.587687 26379 raft_consensus.cc:3032] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e [term 4 LEADER]: Stepping down as leader of term 4
I20240502 01:54:50.587787 26379 raft_consensus.cc:729] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:50.588053 26379 consensus_queue.cc:260] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:50.588383 26379 raft_consensus.cc:3037] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.589972 26774 raft_consensus.cc:2454] T 8cf6da25f535446b9515444e093cae7f P 401942254c4c43a1ba02c4ae68d17320 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 5.
I20240502 01:54:50.590457 26447 leader_election.cc:302] T 8cf6da25f535446b9515444e093cae7f P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 4bc1541b47df446297a369de05163012; no voters:
I20240502 01:54:50.590762 28770 raft_consensus.cc:2781] T 8cf6da25f535446b9515444e093cae7f P 4bc1541b47df446297a369de05163012 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 01:54:50.590905 28770 raft_consensus.cc:686] T 8cf6da25f535446b9515444e093cae7f P 4bc1541b47df446297a369de05163012 [term 5 LEADER]: Becoming Leader. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:50.591152 28770 consensus_queue.cc:237] T 8cf6da25f535446b9515444e093cae7f P 4bc1541b47df446297a369de05163012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:50.591394 26379 raft_consensus.cc:2396] T 8cf6da25f535446b9515444e093cae7f P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bc1541b47df446297a369de05163012 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 01:54:50.593216 26508 leader_election.cc:288] T 8cf6da25f535446b9515444e093cae7f P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 5 election: Requested vote from peers 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959), 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085)
I20240502 01:54:50.594430 26508 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "14a747ac14354fdab2a2993594d3398f"
dest_uuid: "4bc1541b47df446297a369de05163012"
from {username='slave'} at 127.0.0.1:41462
I20240502 01:54:50.594714 26508 raft_consensus.cc:482] T 14a747ac14354fdab2a2993594d3398f P 4bc1541b47df446297a369de05163012 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:50.594866 26508 raft_consensus.cc:3037] T 14a747ac14354fdab2a2993594d3398f P 4bc1541b47df446297a369de05163012 [term 5 FOLLOWER]: Advancing to term 6
I20240502 01:54:50.594616 26234 catalog_manager.cc:5484] T 8cf6da25f535446b9515444e093cae7f P 4bc1541b47df446297a369de05163012 reported cstate change: term changed from 4 to 5, leader changed from 5e9d825f803c40e089b64e319beccf6e (127.11.42.65) to 4bc1541b47df446297a369de05163012 (127.11.42.66). New cstate: current_term: 5 leader_uuid: "4bc1541b47df446297a369de05163012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:50.597649 26508 raft_consensus.cc:504] T 14a747ac14354fdab2a2993594d3398f P 4bc1541b47df446297a369de05163012 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:50.598850 26640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a747ac14354fdab2a2993594d3398f" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:50.599097 26640 raft_consensus.cc:3032] T 14a747ac14354fdab2a2993594d3398f P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 LEADER]: Stepping down as leader of term 5
I20240502 01:54:50.599174 26640 raft_consensus.cc:729] T 14a747ac14354fdab2a2993594d3398f P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:50.599110 26774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14a747ac14354fdab2a2993594d3398f" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:50.599337 26774 raft_consensus.cc:3037] T 14a747ac14354fdab2a2993594d3398f P 401942254c4c43a1ba02c4ae68d17320 [term 5 FOLLOWER]: Advancing to term 6
I20240502 01:54:50.599480 26640 consensus_queue.cc:260] T 14a747ac14354fdab2a2993594d3398f P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:50.599967 26640 raft_consensus.cc:3037] T 14a747ac14354fdab2a2993594d3398f P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Advancing to term 6
I20240502 01:54:50.602212 26774 raft_consensus.cc:2454] T 14a747ac14354fdab2a2993594d3398f P 401942254c4c43a1ba02c4ae68d17320 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 6.
I20240502 01:54:50.602617 26447 leader_election.cc:302] T 14a747ac14354fdab2a2993594d3398f P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 4bc1541b47df446297a369de05163012; no voters:
I20240502 01:54:50.602929 28162 raft_consensus.cc:2781] T 14a747ac14354fdab2a2993594d3398f P 4bc1541b47df446297a369de05163012 [term 6 FOLLOWER]: Leader election won for term 6
I20240502 01:54:50.603073 28162 raft_consensus.cc:686] T 14a747ac14354fdab2a2993594d3398f P 4bc1541b47df446297a369de05163012 [term 6 LEADER]: Becoming Leader. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:50.603343 28162 consensus_queue.cc:237] T 14a747ac14354fdab2a2993594d3398f P 4bc1541b47df446297a369de05163012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:50.603618 26640 raft_consensus.cc:2454] T 14a747ac14354fdab2a2993594d3398f P 83c50029f7a94cfca6d1ee68e0728aa2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 6.
I20240502 01:54:50.603938 26508 leader_election.cc:288] T 14a747ac14354fdab2a2993594d3398f P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 6 election: Requested vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:50.606276 26508 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "beff61f07ecf4ffab508da651a686c8e"
dest_uuid: "4bc1541b47df446297a369de05163012"
from {username='slave'} at 127.0.0.1:41462
I20240502 01:54:50.606462 26508 raft_consensus.cc:482] T beff61f07ecf4ffab508da651a686c8e P 4bc1541b47df446297a369de05163012 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:50.606582 26508 raft_consensus.cc:3037] T beff61f07ecf4ffab508da651a686c8e P 4bc1541b47df446297a369de05163012 [term 5 FOLLOWER]: Advancing to term 6
I20240502 01:54:50.606609 26234 catalog_manager.cc:5484] T 14a747ac14354fdab2a2993594d3398f P 4bc1541b47df446297a369de05163012 reported cstate change: term changed from 5 to 6, leader changed from 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67) to 4bc1541b47df446297a369de05163012 (127.11.42.66). New cstate: current_term: 6 leader_uuid: "4bc1541b47df446297a369de05163012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:50.609273 26508 raft_consensus.cc:504] T beff61f07ecf4ffab508da651a686c8e P 4bc1541b47df446297a369de05163012 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:50.610510 26508 leader_election.cc:288] T beff61f07ecf4ffab508da651a686c8e P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 6 election: Requested vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:50.611068 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beff61f07ecf4ffab508da651a686c8e" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:50.611294 26379 raft_consensus.cc:3037] T beff61f07ecf4ffab508da651a686c8e P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Advancing to term 6
I20240502 01:54:50.614095 26379 raft_consensus.cc:2454] T beff61f07ecf4ffab508da651a686c8e P 5e9d825f803c40e089b64e319beccf6e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 6.
I20240502 01:54:50.614737 26511 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9ead21d62e564a37a8c3d5a452621d13"
dest_uuid: "4bc1541b47df446297a369de05163012"
from {username='slave'} at 127.0.0.1:41462
I20240502 01:54:50.614925 26511 raft_consensus.cc:482] T 9ead21d62e564a37a8c3d5a452621d13 P 4bc1541b47df446297a369de05163012 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:50.615039 26511 raft_consensus.cc:3037] T 9ead21d62e564a37a8c3d5a452621d13 P 4bc1541b47df446297a369de05163012 [term 5 FOLLOWER]: Advancing to term 6
I20240502 01:54:50.617620 26640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beff61f07ecf4ffab508da651a686c8e" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:50.617851 26640 raft_consensus.cc:3037] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Advancing to term 6
I20240502 01:54:50.620599 26640 raft_consensus.cc:2396] T beff61f07ecf4ffab508da651a686c8e P 83c50029f7a94cfca6d1ee68e0728aa2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bc1541b47df446297a369de05163012 for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 01:54:50.620950 26446 leader_election.cc:302] T beff61f07ecf4ffab508da651a686c8e P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 5e9d825f803c40e089b64e319beccf6e; no voters:
I20240502 01:54:50.621533 28770 raft_consensus.cc:2781] T beff61f07ecf4ffab508da651a686c8e P 4bc1541b47df446297a369de05163012 [term 6 FOLLOWER]: Leader election won for term 6
I20240502 01:54:50.621663 28770 raft_consensus.cc:686] T beff61f07ecf4ffab508da651a686c8e P 4bc1541b47df446297a369de05163012 [term 6 LEADER]: Becoming Leader. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:50.621910 28770 consensus_queue.cc:237] T beff61f07ecf4ffab508da651a686c8e P 4bc1541b47df446297a369de05163012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:50.623756 26511 raft_consensus.cc:504] T 9ead21d62e564a37a8c3d5a452621d13 P 4bc1541b47df446297a369de05163012 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:50.624608 26511 leader_election.cc:288] T 9ead21d62e564a37a8c3d5a452621d13 P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 6 election: Requested vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151)
I20240502 01:54:50.625350 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ead21d62e564a37a8c3d5a452621d13" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:50.625581 26378 raft_consensus.cc:3032] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e [term 5 LEADER]: Stepping down as leader of term 5
I20240502 01:54:50.625671 26378 raft_consensus.cc:729] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:50.625857 26640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ead21d62e564a37a8c3d5a452621d13" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:50.626077 26640 raft_consensus.cc:3037] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Advancing to term 6
I20240502 01:54:50.625969 26378 consensus_queue.cc:260] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } }
I20240502 01:54:50.626266 26378 raft_consensus.cc:3037] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Advancing to term 6
I20240502 01:54:50.629441 26378 raft_consensus.cc:2396] T 9ead21d62e564a37a8c3d5a452621d13 P 5e9d825f803c40e089b64e319beccf6e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bc1541b47df446297a369de05163012 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 01:54:50.629777 26511 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1c47a74a6dc54d5eb0b47818d816b7fd"
dest_uuid: "4bc1541b47df446297a369de05163012"
from {username='slave'} at 127.0.0.1:41462
I20240502 01:54:50.629998 26511 raft_consensus.cc:482] T 1c47a74a6dc54d5eb0b47818d816b7fd P 4bc1541b47df446297a369de05163012 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:50.630167 26511 raft_consensus.cc:3037] T 1c47a74a6dc54d5eb0b47818d816b7fd P 4bc1541b47df446297a369de05163012 [term 5 FOLLOWER]: Advancing to term 6
I20240502 01:54:50.633121 26511 raft_consensus.cc:504] T 1c47a74a6dc54d5eb0b47818d816b7fd P 4bc1541b47df446297a369de05163012 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:50.635605 26640 raft_consensus.cc:2396] T 9ead21d62e564a37a8c3d5a452621d13 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bc1541b47df446297a369de05163012 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 01:54:50.636492 26643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c47a74a6dc54d5eb0b47818d816b7fd" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:50.636777 26643 raft_consensus.cc:3037] T 1c47a74a6dc54d5eb0b47818d816b7fd P 83c50029f7a94cfca6d1ee68e0728aa2 [term 5 FOLLOWER]: Advancing to term 6
I20240502 01:54:50.639536 26643 raft_consensus.cc:2454] T 1c47a74a6dc54d5eb0b47818d816b7fd P 83c50029f7a94cfca6d1ee68e0728aa2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 6.
I20240502 01:54:50.640033 26446 leader_election.cc:302] T 1c47a74a6dc54d5eb0b47818d816b7fd P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc1541b47df446297a369de05163012, 83c50029f7a94cfca6d1ee68e0728aa2; no voters:
I20240502 01:54:50.640288 28162 raft_consensus.cc:2781] T 1c47a74a6dc54d5eb0b47818d816b7fd P 4bc1541b47df446297a369de05163012 [term 6 FOLLOWER]: Leader election won for term 6
I20240502 01:54:50.647925 26446 leader_election.cc:302] T 9ead21d62e564a37a8c3d5a452621d13 P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bc1541b47df446297a369de05163012; no voters: 5e9d825f803c40e089b64e319beccf6e, 83c50029f7a94cfca6d1ee68e0728aa2
I20240502 01:54:50.656208 26511 leader_election.cc:288] T 1c47a74a6dc54d5eb0b47818d816b7fd P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 6 election: Requested vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:50.656373 26774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c47a74a6dc54d5eb0b47818d816b7fd" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:50.656620 26774 raft_consensus.cc:3037] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320 [term 5 FOLLOWER]: Advancing to term 6
I20240502 01:54:50.658977 26511 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "245ca06c39774fb5b0f522550a63ac5b"
dest_uuid: "4bc1541b47df446297a369de05163012"
from {username='slave'} at 127.0.0.1:41462
I20240502 01:54:50.659195 26511 raft_consensus.cc:482] T 245ca06c39774fb5b0f522550a63ac5b P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:50.659348 26511 raft_consensus.cc:3037] T 245ca06c39774fb5b0f522550a63ac5b P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.659922 26774 raft_consensus.cc:2396] T 1c47a74a6dc54d5eb0b47818d816b7fd P 401942254c4c43a1ba02c4ae68d17320 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bc1541b47df446297a369de05163012 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240502 01:54:50.662168 28953 raft_consensus.cc:2726] T 9ead21d62e564a37a8c3d5a452621d13 P 4bc1541b47df446297a369de05163012 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240502 01:54:50.662842 28162 raft_consensus.cc:686] T 1c47a74a6dc54d5eb0b47818d816b7fd P 4bc1541b47df446297a369de05163012 [term 6 LEADER]: Becoming Leader. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:50.663081 28162 consensus_queue.cc:237] T 1c47a74a6dc54d5eb0b47818d816b7fd P 4bc1541b47df446297a369de05163012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:50.665127 26234 catalog_manager.cc:5484] T beff61f07ecf4ffab508da651a686c8e P 4bc1541b47df446297a369de05163012 reported cstate change: term changed from 4 to 6, leader changed from 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67) to 4bc1541b47df446297a369de05163012 (127.11.42.66). New cstate: current_term: 6 leader_uuid: "4bc1541b47df446297a369de05163012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:50.666227 26511 raft_consensus.cc:504] T 245ca06c39774fb5b0f522550a63ac5b P 4bc1541b47df446297a369de05163012 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:50.666927 26511 leader_election.cc:288] T 245ca06c39774fb5b0f522550a63ac5b P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 5 election: Requested vote from peers 5e9d825f803c40e089b64e319beccf6e (127.11.42.65:34085), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:50.667196 26378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "245ca06c39774fb5b0f522550a63ac5b" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "5e9d825f803c40e089b64e319beccf6e"
I20240502 01:54:50.667449 26378 raft_consensus.cc:3032] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e [term 4 LEADER]: Stepping down as leader of term 4
I20240502 01:54:50.667537 26378 raft_consensus.cc:729] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5e9d825f803c40e089b64e319beccf6e, State: Running, Role: LEADER
I20240502 01:54:50.667812 26378 consensus_queue.cc:260] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:50.667975 26774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "245ca06c39774fb5b0f522550a63ac5b" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:50.668102 26378 raft_consensus.cc:3037] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.668198 26774 raft_consensus.cc:3037] T 245ca06c39774fb5b0f522550a63ac5b P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.668251 26511 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d9dd99b54b7c4d19a80c229bf99d7976"
dest_uuid: "4bc1541b47df446297a369de05163012"
from {username='slave'} at 127.0.0.1:41462
I20240502 01:54:50.668432 26511 raft_consensus.cc:482] T d9dd99b54b7c4d19a80c229bf99d7976 P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:50.668522 26511 raft_consensus.cc:3037] T d9dd99b54b7c4d19a80c229bf99d7976 P 4bc1541b47df446297a369de05163012 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.670881 26378 raft_consensus.cc:2396] T 245ca06c39774fb5b0f522550a63ac5b P 5e9d825f803c40e089b64e319beccf6e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bc1541b47df446297a369de05163012 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 01:54:50.670993 26774 raft_consensus.cc:2454] T 245ca06c39774fb5b0f522550a63ac5b P 401942254c4c43a1ba02c4ae68d17320 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 5.
I20240502 01:54:50.671710 26447 leader_election.cc:302] T 245ca06c39774fb5b0f522550a63ac5b P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 4bc1541b47df446297a369de05163012; no voters: 5e9d825f803c40e089b64e319beccf6e
I20240502 01:54:50.672155 28162 raft_consensus.cc:2781] T 245ca06c39774fb5b0f522550a63ac5b P 4bc1541b47df446297a369de05163012 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 01:54:50.672289 28162 raft_consensus.cc:686] T 245ca06c39774fb5b0f522550a63ac5b P 4bc1541b47df446297a369de05163012 [term 5 LEADER]: Becoming Leader. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:50.672546 28162 consensus_queue.cc:237] T 245ca06c39774fb5b0f522550a63ac5b P 4bc1541b47df446297a369de05163012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } }
I20240502 01:54:50.676949 26234 catalog_manager.cc:5484] T 1c47a74a6dc54d5eb0b47818d816b7fd P 4bc1541b47df446297a369de05163012 reported cstate change: term changed from 4 to 6, leader changed from 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68) to 4bc1541b47df446297a369de05163012 (127.11.42.66). New cstate: current_term: 6 leader_uuid: "4bc1541b47df446297a369de05163012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:50.678221 26234 catalog_manager.cc:5484] T 245ca06c39774fb5b0f522550a63ac5b P 4bc1541b47df446297a369de05163012 reported cstate change: term changed from 4 to 5, leader changed from 5e9d825f803c40e089b64e319beccf6e (127.11.42.65) to 4bc1541b47df446297a369de05163012 (127.11.42.66). New cstate: current_term: 5 leader_uuid: "4bc1541b47df446297a369de05163012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d825f803c40e089b64e319beccf6e" member_type: VOTER last_known_addr { host: "127.11.42.65" port: 34085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:50.671264 26511 raft_consensus.cc:504] T d9dd99b54b7c4d19a80c229bf99d7976 P 4bc1541b47df446297a369de05163012 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:50.681784 26511 leader_election.cc:288] T d9dd99b54b7c4d19a80c229bf99d7976 P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 5 election: Requested vote from peers 83c50029f7a94cfca6d1ee68e0728aa2 (127.11.42.67:43151), 401942254c4c43a1ba02c4ae68d17320 (127.11.42.68:37959)
I20240502 01:54:50.682307 26774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9dd99b54b7c4d19a80c229bf99d7976" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "401942254c4c43a1ba02c4ae68d17320"
I20240502 01:54:50.682500 26774 raft_consensus.cc:3037] T d9dd99b54b7c4d19a80c229bf99d7976 P 401942254c4c43a1ba02c4ae68d17320 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.683027 26640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9dd99b54b7c4d19a80c229bf99d7976" candidate_uuid: "4bc1541b47df446297a369de05163012" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "83c50029f7a94cfca6d1ee68e0728aa2"
I20240502 01:54:50.683296 26640 raft_consensus.cc:3032] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 01:54:50.683405 26640 raft_consensus.cc:729] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 83c50029f7a94cfca6d1ee68e0728aa2, State: Running, Role: LEADER
I20240502 01:54:50.683730 26640 consensus_queue.cc:260] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bc1541b47df446297a369de05163012" member_type: VOTER last_known_addr { host: "127.11.42.66" port: 34003 } } peers { permanent_uuid: "83c50029f7a94cfca6d1ee68e0728aa2" member_type: VOTER last_known_addr { host: "127.11.42.67" port: 43151 } } peers { permanent_uuid: "401942254c4c43a1ba02c4ae68d17320" member_type: VOTER last_known_addr { host: "127.11.42.68" port: 37959 } }
I20240502 01:54:50.684165 26640 raft_consensus.cc:3037] T d9dd99b54b7c4d19a80c229bf99d7976 P 83c50029f7a94cfca6d1ee68e0728aa2 [term 4 FOLLOWER]: Advancing to term 5
I20240502 01:54:50.685096 26774 raft_consensus.cc:2454] T d9dd99b54b7c4d19a80c229bf99d7976 P 401942254c4c43a1ba02c4ae68d17320 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc1541b47df446297a369de05163012 in term 5.
I20240502 01:54:50.685817 26447 leader_election.cc:302] T d9dd99b54b7c4d19a80c229bf99d7976 P 4bc1541b47df446297a369de05163012 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401942254c4c43a1ba02c4ae68d17320, 4bc1541b47df446297a369de05163012; no voters:
I20240502 01:54:50.686122 28770 raft_consensus.cc:2781] T d9dd99b54b7c4d19a80c229bf99d7976 P 4bc1541b47df446297a369de05163012 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 01:54:50.686276 28770 raft_consensus.cc:686] T d9dd99b54b7c4d19a80c229bf99d7976 P 4bc1541b47df446297a369de05163012 [term 5 LEADER]: Becoming Leader. State: Replica: 4bc1541b47df446297a369de05163012, State: Running, Role: LEADER
I20240502 01:54:50.686534 28770 consensus_queue.cc:237] T d9dd99b54b7c4d19a80c229bf99d7976 P 4bc1541b47df446297a369de05163012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: